I'd disagree with the yellow being more casual than a white. A white shirt without a tie is very casual. If you are going with too much colour, it's more of an attention getter than just hanging out type of ware. I've got an informal dinner party to go to this friday, and am planning to ware my Herratage of Scotland kilt with a simple black sporran, white shirt and black hose. I'll probably bring a grey sweater with me in case the party moves outside.

The best advice from here has been wear what you would if you didn't have the kilt.. the kilt is just like a pair of trousers.