Actually, the sun emit a kind of short-wave radiation which will not be absorbed by us directly. However, the Earth's surface will absorb this kind of radiation and emit another kind of radiation--long wave radiation, this is the kind of radiation which is absorbed by us.
At places of high altitude, there is less carbon dioxide and dust(since the air is becoming thinner and thinner with increasing altitude) to absorb the long wave radiation, less heat is trapped. Moreover, at places of higher altitude, it is much further away from the true heat source--the earth's surface, and the air cannot absorb as much heat as that in lower altitude. It does not matter with the distance form the Sun.
As a result, it is becoming colder and colder as altitude of a place increases.