While you may take it as a joke, it's still an insensitive comment. However, I do understand that your intention may not have been to harm a particular group. I do recall reading it and you mentioned how someone took it offensively, which I would understand coming from a marginalized racial group myself.

If someone, feels offended understand that it may not have anything to do with not being able to take a joke and everything to do with a long history of oppression. Food for thought.