Why is it that when I’m introverted and quiet, I draw more attention to myself?

Like when I worked fast food, I came to work, did my job and then clocked out. I did not socialize and I was the talk of the town! Everybody gossiped about me all the time. 
But there were plenty of other males that did not socialize and kept to themselves and people didn’t even know their names or care! I don’t understand why I drew in all the attention. Is it solely because i’m female?