physics?

This is because of difference in pressure.

Pressure is the force per unit area. Hence, for the same applied force, a small area of contact leads a high pressure.

In your example, if you apply, say, a 10 N force onto your pen. Assume the point of contact is only 0.1 mm^2 (i.e. 10^-7 m^2), the pressure that the pen exerts on your skin is 10/10^-7 N/m^2, or 10^8 N/m^2

However, suppose the area of contact of your piece of rubber eraser is about 1 cm^2 (i.e. 10^-4 m^2), the pressure now becomes 10/10^-4 N/m^2 = 10^5 N/m^2, which is 100 times lower than that given by the pen.