There are a lot of reasons. Not the least of which being that statistics show either no significant benefit to larger calibers for most self defense situations (ignoring special purpose applications like windshield or barrier penetration, etc) or even a slight edge with the smaller calibers like the 9mm because of more reliable followup and controllability.
People can train more for less money, and train less to achieve similar performance (less time spent training to overcome aversion to recoil).
Plus, the 9s can be packaged smaller, lighter, and cheaper without giving much up in performance. For example, I would be very comfortable carrying a subcompact 9, but not a subcompact 45, because the recoil on a gun like that is brutal. Plus there is the capacity argument (more ammo in the same form factor).
In the end, I love my 44 Mags and my 45 ACPs and the 357 Sigs and the 40S&Ws... but I carry a 9mm +P or 380 ACP
Why do some people quit using 92 octane gasoline and end up using 87 octane? Because there was no spectacular gain in performance and just burning up money.
weak wrists, better control an recovery, finacially inadequate.
I have switched from bigger calibers for self defense, not hunting, because 9mm will do all I need it to do without much of a problem. I have better control and can sight faster with 9mm. I still have 10mm and 44 mag and 45 ACP for other reasons.
9mm, 40 S&W and 45ACP perform about the same, but 9mm is cheaper, can be built on a smaller frame, has less recoil and holds more rounds. Though smaller 9mm is faster so in the end it carries about the same energy as 45 ACP standard pressure rounds, certain 9mm +P load carry more energy than standard 45 loads. For me 9mm meets the needs of a concealed carrier, and I have carried a full size 45 before. I do like 185gr 45 +P loads, and there is a slightly larger gape in energy with that specific load compared to 124gr 9mm+P, but for something I carry around all day I'd pick 9mm first.
the reason is the same why some people move from a bigger house to a smaller one, or they trade in their truck for a car.
the reasons are many and and varies person to person. some people switch because 9 is cheaper to shoot. some people switch because 9mm's can be more compact than 40 or 45 without significant loss in capacity. some people switch because they found a 9mm they liked. some people switch because they want to.
just like everything else in life, bigger doesn't automatically mean its better.
sometimes bigger is detrimental to a specific task than if you want with smaller, or a new tool better suited for it.
for example, you want to start conceal carry. what you have now, a selection of full size 45's and 40's, are just too big for this purpose. if you got a more compact 45 or 40, you feel the magazine capacity is unsatisfactory. so you select a 9mm because it can be small, has satisfactory performance, and still have sufficient capacity.
i own the biggest hand cannon of them all, the 500 mag, but i prefer my 9mm's most of the time because bigger doesn't always mean better. my 9's are more compact, easier to handle, carry and shoot, more practical, more economical. the 500 is more of a fun gun or display piece (although i do shoot it regularly with reduced power ammo) its massive size and weight i tend to think of it more as a long gun than a handgun. and for purpose of defense its got a number of serious problems that a lesser, more reasonable caliber does not have. bigger doesn't mean better, in fact bigger is just bigger.
Department policy, person preference, physical requirements and scores of other possible reasons.
Cheaper ammo, longer time firing due to less force on hands.
You'll find that many people enter the gun world thinking they want to be "badd-a$$" and they go out and get a gun that they think would be suitable for an Internet commando. Then reality sets in. Carrying a gun is a pain... literally... a pain. Suddenly that big caliber gun isn't so cool. It's just a big heavy chunk of steel. They find themselves looking at sub-compact and pocket guns because in reality that's what most people can reasonably conceal. All the myths about caliber comparisons get put into perspective and they realize that IN THE REAL WORLD... when you shoot ANYTHING at someone, the person will run away. IN THE REAL WORLD, a 22 caliber searing hot hole in your body is plenty painful and lethal. IN THE REAL WORLD, the old saying of "the best carry gun is the one that you actually carry" rings true.
With all of that, people who really do carry guns will carry smaller, lighter guns and that means smaller lighter ammo.
Finally... take an educated look at the difference between a 9mm and the next one up... the 40. Compare ballistics, velocities, etc. Notice all the overlap? Yeah... to get something out of a 40 that you can't get out of a 9mm you've got to be shooting some boutique rounds. Now do the same thing with a 45. It's a bigger hole and a slower bullet. So... how much force is delivered to the target? Wow... amazing how that physics stuff works. Now take into account the modern self defense rounds that dump virtually ALL their energy into the target and ask... why would I want a 45? So I can carry fewer rounds?
Yes... the bigger guns CAN be made to outperform a 9mm, but the reality is that there simply isn't enough of a difference to matter in the real world. If someone is doing real unbiased comparisons they quickly discover that all the hype around the bigger rounds is just that... hype.
If you talk to guys about their carry gun and they're being Mr. Tough Guy chances are that they don't carry all the time.
As others say, there are lots of reasons. One I'd add is that the bullets available today are better. If you'd asked me twenty years ago, I'd have scoffed at a defensive gun that didn't start with a four. Yesterday I picked up a subcompact nine. It'll be easy to carry, work perfectly well, and (since it's a modular Sig P320) if I decide to change parts, I can turn it into a full-sized 45 ACP if I change my mind later.
They figure a lot of rounds is better. Too lazy to learn how to aim, they prefer to spray lead.
I found some of the answers quite good, but from MY perspective: I carry a .357 magnum Dan Wesson with the snubby barrel. It's bigger and harder to hide than the small and even medium frame autos like the 9mms and it doesn't hold a dozen plus rounds.
I also know that proper shooting technique in a combat situation requires a point and shoot technique which is learned through intense practice. (I took Jim Gregg's COPS course many years ago and KNOW where every shot will go!) In law enforcement, I was in 4 shoot outs. All at less than 5 yards (normal in a shoot out situation and MUCH different than the Hollywood 50 + yard "combat" shooting.!) and I was the one who walked away from each one. I never fired more than 2 shots. (Another combat shooting technique learned is to always fire 2 and be ready to fire the next pair if the first didn't end the situation.)
Both rounds would have been fatal in all cases and knocked my opponents off of their feet, where even if they were still alive, they would hit the ground hard enough to NOT shoot back! I've seen people take a dozen plus 9mm hits and still be standing. They were dead, but didn't know it yet and were able to take their shooter with them before they collapsed.
Unfortunately, many people do NOT know proper shooting techniques and don't practice enough. They are told that because they have lots of bullets, they can stop anything. Spray and pray does NOT work. I once saw a shootout where two people at less than 10' emptied their 9mms (each started with at least a dozen bullets) and neither hit the other!
Some tried the larger calibers, but the recoil made them look for something "easier" to shoot. Just remember that recoil and stopping power have a lot in common! As far as the "new bullet designs" impacting more stopping power, it is true up to a certain point. Remember that a bullet that fragments easily doesn't penetrate as far and a bullet with good penetration doesn't fragment as much. The same diameter and design bullet fired with a larger powder charge will still have more knock down power and because of the higher impact power will allow the fragments to penetrate much further . Also a larger diameter bullet fired even at slightly lower velocity can spread that impact over a wider area and doesn't need as much expansion in the bullet to produce the same impact pressures.
I love 9mm. by far my favorite caliber. Its easy to control, low recoil, and effective. there is no point in having a big bullet, if you dont hit your target. plus they are inexpensive to shoot.
Assuming this is about self defense. Some people think faster bullets stop/kill better. I use 300 grain .44 reverse wad cutters, loaded special. You want all the energy to stop inside the target.
Better is a big caliber for people who can't hit with any of them. From your question, I conclude you are one of those people.
Go big or go home. You can't miss fast enough to win a gunfight.
With any kind of luck, you will be able to prove this point firsthand.