Light is electromagnetic wave (EM wave) in nature. It travels in space by the mutual interaction of electric and magnetic field components.
When light (EM wave) enters a glass prism, the electric field and magnetic field components internact with the glass molecules, held together by intermolecular force which is an electromagnetic force.
It has been shown, by the famous physicist James Clark Maxwell in the 19 Century, that the speed of light (EM wave) in a substance is determined by the permitivity (this is a term to describe the electric property of a substance) and permeability (this is a term describing the magnetic property of the substance) of the susbtance. Maxwells derivation showed that the higher the permitivity or mpermeability of a substance, the slower the EM wave travels in that substance. Since vacuum has the lowest permitivity and permeability, thus light travels fastest in vacum than in other substances.
For most substances, values of their permeability do not varies much within the optical frequency range. Thus, the speed of light in these substances only depends on the value of permitivity.
Values of permitivity for glass depend on wavelength. The longer the wavelength, the smaller the value of permitivity is. Since red light has the longest wavelength in the visible spectrum, the permitivity for red light is the smallest among other colour lights. As the smaller the permitivity, the higher the speed of EM wave is, thus the speed of red light is the highest in glass prism over other colour lights.
You may refer to the following web-page for more details:
http://en.wikipedia.org/wiki/Refractive_index