Interesting problem.
I would suspect factions have developed with your group, that have nothing to do with your kilt. Im not sure how you would change the by laws to prohibit kilts in a nudist group. Seems like the ACLU would have a field day with that one.
I wouldnt be too quick to leave. You started the group. Im sure you must have followers within the group.

Craig