I don't really know, but I'll hazard a guess:
Taking off your hat is usually a sign of respect (i.e. taking off your hat indoors or during the national anthem). That respect, at its core, is self-deference. Saying that you will take off your hat for nothing would seem to imply that will not defer to anyone.
My
