racial jokes are simply incredibly sensitivity-based, and because everyone has a different sensitivity level, its impossible to really see how something'd effect someone else.
its just very, very tricky.
some people will think your brother and his boss are horrible people and should be shot. others'll think they're just bein stupid and others that they're just having fun.
everyone is right in their own mind, and nothing anyone says can tell them different.
my friends and i make racial jokes all the time, but only because we know one another, and know we can get away with the jokes. we're not hurting anyone's feelings. in the same fasion, we make weight jokes, height jokes, religious jokes, personality jokes, drunk jokes... whatever. they're individual experiences or characteristics we all share, and can discuss or laugh about toegether.
what your boss and brother did wrong was not correctly perceive their environment. if its pissing you off, it was cruel of them to continue. in that instance, they're pretty much upsetting you on purpose, which is wrong in my book.