For starters, you're doing it wrong. In the first sentence you said, "I'm not into furry porn..." A quirk about the human mind is that when you describe something (ie. yourself or the furry fandom) by describing what it is not (ie. you're not into furry porn), their mind puts those things together in a mental association. It doesn't matter that you used the word "not"; those individuals will associate you the furry with furry porn. Even if they didn't know furry porn existed, they would still immediately associate furry with porn because you put it into their minds.
Next, you (and your friends who are also furries) need to develop a stronger sense of pride to be a furry and courage to defend it. As a furry among non-furries, such as at work, you are a representative of the furry fandom so it's your responsibility to be the best representative you can be. When people at work were calling furries "sick" and "perverted", you should have taken the time to ask what gave them that impression and why they think all furries are like that. And then you should have described what furries and the fandom really are all about and corrected their misconceptions. Ask them if you seem sick and perverted because, after all, you're a furry, and remind them that the actions of a handful of people do not define or represent the entire furry fandom overall.
It's not that you have to march around town with a banner saying "I'm a furry", or just outright tell everybody you know and meet that you're a furry. That's as unnecessary as someone going around telling everybody they're into quilting or are fans of some TV show. But as a furry who knows what the furry fandom is all about you know there is plenty to be very proud of about the furry community/fandom and are in a good position to describe what it is while countering any myths, misconceptions, and stereotypes your family, friends, and coworkers may have.
I'll include a link to a video, "Presenting Furry Fandom to the Public", a 6-part series of videos describing what to say and how best to say it when describing what the furry fandom is all about to family, friends, coworkers, strangers, and even the media. Some main lessons from the video are (1) never try to describe what we are by listing the things we are not (like I described above), (2) keep things positive, and (3) don't assume people have heard anything negative or know anything about the furry fandom. I also have my own personal philosophy: always show enthusiasm and pride to be a furry; never show embarrassment or shame or people will only perceive it as something you *should* be embarrassed and ashamed about.
https://www.youtube.com/watch?v=Z2DSc8GHl5g&list=PLebqXGpkDOdLIhQcPxdd7HOLF3oKhFvh4
參考: furry since 1999, active in the furry community since 2008, and tremendously proud to be a furry