I think a lot of women feel threatened that they will no longer be the center of attention if the man they are with creates a stir whilst kilted.

Either that, or they feel they will be judged somehow if they are not as "dressy" as a man in a kilt is perceived to be. People think kilts are formal dressy things, and a woman may feel slouchy compared to a dapper looking gent in a kilt.