If pro and con are opposites, wouldn't the opposite of progress be congress?

Very droll. The prefix con-, which means “with” is not the opposite of pro-

The pros and cons is a shortened form of pro et contra, 'for and against',

I say regress.

Bored I take it

You got that joke from family guy
Joe says it nobody responds XD

No, but regress is.

English is stupid.
Congress means "a formal meeting or series of meetings for discussion between delegates, especially those from a political party, trade union, or from within a particular sphere of activity." and progress means "movement toward an improved or more developed state, or to a forward position" The two aren't related to each other, even as opposites so unfortunately not.

A group of baboons is called a congress.

https://m.youtube.com/watch?v=GPXkjtpGCFI

We've heard that joke a million times

~Aizen