What's it called to cancel a refugee's passport due to serious criminal record?

You don't mean 'cancel his passport' you mean 'revoke his citizenship'.  That can only happen either if there was fraud during the application process or the person committed an act of treason against the host country.   Either way, the criminal would also have to hold another citizenship, a country cannot deliberately make someone stateless.

deportation......                      

only the home nation can cancel a passport.  however, a criminal refugee can be deported

not sure but it means they wont let him in cause of his criminal record

That is called "tyranny".