Is it right to ignore my uncle?

My uncle is married to my dad's older sister, and they have 2 sons. I used to hang out with them when I was younger, and many people said that I looked like my dad's sister, even now. When my mom found out that I hanged out with them whenever they came over to my grandmother's. She told me to stop hanging out with them. That was last year. This year, I got curious and asked her why, she said that it was because my uncle did something bad to one of the helpers at my grandmother's house. That helper went back to her home country. I asked her what happened, but she wouldn't tell me and just said that you will understand when you grow up. She said that my parents and my grandfather know, but not my grandmother. And I also realised my grandfather doesn't interact with his sons, but my grandmother does. My grandmother is he one who invites them, not my grandfather. And I remember when I was younger, my uncle said that if he could, he would adopt me. And which is kinda creepy and weird. Recently I've been hanging out with his sons again, and we are becoming friends. But it feels really awkward. I still don't know what happened. Should I keep on ignoring him and hang out with my cousins? Thanks.