No sweat. To me, having Scottish ancestors, there is a bit of symbolism there, as well. I don't see that as a reason to keep me from wearing a kilt to the bar, the grocery store, or a Halloween party, though. To each their own.
I've worn my kilt to a bar/pub on many occasion, usually after a Scottish society function. I've also worn it in stores, grocery and other wise.

T.