You shouuldn't say or do anything to him because it's your spouse's job to deal with their family members. Anything you say would only make matters worse so your mate should defend you.
1. refuse to be in the same room with him,
2. demand that he apologizes and tell your fmaily they should insist he does
3. Invite the family to your house but not him.