It depends on how heavy the kilt pin is. I've got a heavy one which I wouldnt want to leave on a nice kilt for too long between uses. Also /how/ you pin your kilt can affect it. I've found that the pins with the really sharp points tend to be more damaging because you end up splitting the threads in the weave, whereas the ones which are rounded a bit more push the threads aside, and the 'holes' will seal back up when the pin is removed.

That being said, I'm going to be making quite a few kilt pins from the tines on a half rack of antlers which I picked up for making knife handles, that way I'll have a decent pin for most of my kilts.