When do we use "of" following an adjective?

Yes, this is much the same as:
"You are rude to speak filthy words."

This pattern goes as:
It is "an adjective" (indicating a person's nature or character, such as kind, nice, foolish, and clever) of a person to-inifinitive.

Here are some other examples:

He is the taller of the two brothers.

She is the fairest of them all.

Your answer is devoid of any relevance to the question.

Peter has some of my good looks.

It also happens in a few phrases like "short of breath" and "slow of tongue." I can't really think of many examples of that though.