Merchants accepting credit cards are the ultimate responsible party.
Each time when credit cards are used, accepting merchants are charged by credit card processors, which are usually banks, merchant fees (about 3% of the entire amount processed).
That means the merchants get $97 per $100 charged.
In that case, even consumers pay no fee to banks (such as annual fees, late fees, interests, etc.), banks are still be able to recover their expenses and bad debts from merchants.
There is a reason why stores selling high-value items don't want to accept credit cards or impose a surcharge.