Friend trying to get me to do weed?

Midway through last year I became friends with a guy I worked with. We talked during work and exchanged socials/number. I no longer work there but we keep in touch and talk every week.

I only see him as a friend and we hung out once in the city and shopped around in the city and played games of pool.

A few weeks ago we were messaging and he talked about his mate smoking too much weed. I said I had never tried it, that I would like to but I shouldn’t because I am diabetic.

For the next few weeks my friend kept asking me if I wanted to do weed and kept saying we should do it at night at his mates house who I have never met before.
I kept ignoring his question but he wouldn’t let up. 
He said I could go first because he knows I’m diabetic and he’d watch out for me.

This weekend he was going to a house party with his mates I also haven’t met before. I said no I was working and couldn’t make it. He kept repeatedly asking me to go over even though I said no. 

On the day of the party he was still telling me to go to the train station and he would meet me there to which I said no again.

I felt weird about his incessant invites even though I had said no. I’m considering if I should continue contact with him or not.

Would anybody else feel weird if this was them?