I think I'd at least set the record straight by informing the boss's boss that I was wearing a kilt and not a woman's skirt -- maybe have a photo handy.
He's still likely to give same answer about wearing it, but at least it will be based on accurate information.