Limit of Improper Integrals

First of all, we need to digest the question more clearly.

What is the limit of integrating x^2 from t to positive infinity with respect to x, as t approaches positive infinity?

This is a trick question because it seems that we are asking for one limit, but in fact it is a limit of limit, the quantity "integrating x^2 from t to positive infinity with respect to x" itself is a limit, defined as
limit u approaches infinity "integrating x^2 from t to u with respect to x".

Apparently, that does not exist, the sequence increase without bound.

With that, the rest is easy, you cannot find the limit of a sequence of non existent quantities. Therefore, the answer to the question is undefined - the limit does not exist.

Without using integral at all, one can also try a problem like this.

(lim y-> infinity (lim x -> infinity (y - x)))

It is not 0, it is similarly undefined. Note the use of bracket to ensure the limits are taken one by one.

The case of exp(-x) is trivial because it converges to 0 but a divergent integrand confuses me the most. If it ends up with an indeterminate form, can we further simplify it?

If I only answer your particular question (without the general discussion), you first evaluate the definite integral, then you take limit.

　lim(t→∞) ∫[t ~ +∞] x² dx
= lim(t→∞) [x³/3] | [t ~ +∞]

ends up in (∞ - ∞) indeterminate form.

2015-01-25 02:47:43 補充：
However, you can consider some other cases like:

　lim(t→∞) ∫[t ~ +∞] exp(-x) dx
= lim(t→∞) [-exp(-x)] | [t ~ +∞]
= lim(t→∞) [exp(-x)] | [+∞ ~ t]
= lim(t→∞) [-exp(-t)]
= 0