This is simple: Would you wear that shirt with pants?

Then you can wear it with a kilt.