Is Atlantic City a big city?

No, it's not.  It really has a beach-town kind of feel to it as opposed to a city vibe.  But it certainly qualifies as a city.  It's just not a big one, nor does it feel like a big one.

No.  I would not call a city of 40,000 people a big city.

Depends on your definition of big. About 40,000 call it home.
Smaller than NYC but a lot bigger than the City of Shelby Mt. population just over 3,000.