What does it mean when a guy friend says your dress is beautiful? Is that a compliment to me or the dress per se?

It means he thinks your dress is beautiful for some reason. It's plain English.. unless English is your second language?

It's a compliment to you if you made the dress.

He likes your dress.

"Your dress is beautiful" = the dress is beautiful
"That dress is beautiful on you" = that has more to do with you, not just the dress.

He may not have meant it pointedly to exclude you from the compliment.
It's more common that a woman would make the distinction to be catty. Take it as a compliment, and don't overthink it.