It is so difficult to generalise. On the whole I agree with Matt.

Having said that, I have never attended a Highland Games or Gathering in the USA and, when I look at the many photos of such events that are scattered across the Internet, I am amazed that people can dress as they appear to - and get away with it! One sees truly horrifying sights, almost enough to make one's stomach churn

So, whilst a lot depends upon the nature of your 'casual' kilt, I do get the impression that just about anything goes over on your side of the pond and I guess you'll be fine.

I am not so sure that you'd escape so lightly over here, especially in Scotland, where casual kilts are hardly ever seen. In a few years time things could be very different, however, due to the increasing popularity of Utilikilts and their kind. The lightweight, synthetic fabric, casual tartan kilts so popular in the hotter parts of the USA, might take a while longer to be accepted - but I hope I will proved to be wrong.