Men: Is he the one afraid of being hurt?

I've been hooking up / sort of dating my guy friend. we were friends for a year (he liked me for 6 months) then we started seeing eachother. We're EXTREMELY close, he says I'm the only one he can be himself around and the only women he truly respects. When things start to get a little more serious he pulls away. He keeps asking me to hang out but then something is always more of a priority. (And no I know for a fact there is no other woman) so one day I confront him about it and he says "you don't want to date me. You can't handle something casual and that's all I can offer" so I said "why do you think I can't handle it?" And his answer made NO sense... he said "Because what if you met a guy who you really liked and showed interest in you and you ended up really liking him and wanted to be with him. And technically if we're casual then you're single, you would go with him right?" I was stunned bc that has nothing to do with my question so I hesitated and said "well if I'm single... then yes?" And he said "right so then it could never work"

so basically he's saying I can't handle something casual with him bc I couldn't handle falling for another guy? lol is he subconsciously saying he's afraid of being hurt and ME leaving HIM?