My observation is that people really want to categorize your kilt-wearing and sound like they know what they're talking about, so they bring up the tartan's clan. When you're just wearing a tie, or a vest, the question simply isn't there to ask because ties and vests are much more commonly worn.

It's the same sort of reason people ask us if we're pipers or whatever - they have to be able to put you into some sort of category and they talk you up using the knowledge they have (though scant it may be) in an effort to do so.

Then again, I may be a bit jaded about people...