I use both, and I think which one I choose is determined by the subject.

Singular subject: the kilt.
I wear the kilt.
He wears the kilt.

Plural subject: kilts.
They wear kilts.
We wear kilts.

I would also say "I have three kilts," not " I have three of the kilt."