Why can't I erase dirty marks made by an eraser?

Erasers vary in quality. Cheap ones always leave some residue, and it gets worse with age. But all erasers deteriorate as they get older, either beginning to disintegrate or becoming hard. They should be discarded at the first sign. Sounds like you are trying to use one which is well past its useful life-span. The marks it has left will have set like concrete, and cannot be removed without damaging the paper.

Oil and grease and sun light mess up an eraser.

Have you tried rubbing on plain paper until the dirty marks have been rubbed away?