Why are we forced into college right after high school?

I understand that we won't be so motivated to go to college if we wait after high school, but I want to be able to travel after school, not go back for another four years and not have any money to travel afterward. I Don't want to wait until I retire and am too old to do the things I can do now. It's like if you Don't go straight to college after you graduate high school then you're considered a failure, or people just think less of you. That or you have to disappoint your family. I feel like I need to mature more before deciding what I want to do for the rest of my life, too. That's just a big decision for someone my age, and it seems that people change their minds after college a lot, pretty much wasting that four years and the money they spent for college. Am I wrong to think like this?