If someone tells me that I shouldn't wear a kilt I say "OK I'll take it off!". That usually shuts them up

Most folks just assume that you're wearing it for a "reason"; the games or a special event or the like. Some will ask if your Scottish. They are mostly just interested. Boneheads will give you a hard time and there's not much you can do about them except not waste your time.
And yes, Chicks Dig It!
Bob