Of course it is, thats the whole point isnt it? You are saying he is something less than he is.
Have you never said to someone "....is a big girl" and someone has replied, "thats insulting to girls?" The comeback is meant as a joke, but there is truth in it as well.