10 year old son keeps adjusting himself?

Hi my son has been doing this since last year, every time he sits down he grabs his crotch and pulls it down like he is adjusting himself. I thought that was normal as my husband says he has to adjust himself from time to time. But my son does it a little more then normal. Sometimes while sitting in church he will grab his bible and put over himself and adjust or itch or something like that, not sure what he is doing. He doesn't look like he is playing with himself at all, just constantly grabbing himself like he is uncomfortable. So I asked what is going on and he said his underwear are to loose and he has to pull them down. So we got him some snug boxer briefs that fit good, yet he still does it. He grabs himself and then wiggles in his seat like he is so uncomfortable. Is this normal? Should I take him to see a doctor? He doesn't have a uti as he said it doesn't hurt to pee or anything and no fever. Like I said it's been a whole year and it's driving me crazy. I finally tell him to go to the bathroom if he itches or something but not to be drawing so much attention to himself. Is this a purberty thing, is it hair coming in that can cause this much grabbing of himself? What is it????