I've stopped using the little buckle assembly now when I make that sort of kilt belt - I use velcro, as it's kinder on the fabric of the kilt, and you don't need a long strap inside either.

Have the soft bit facing out towards the kilt roughly where the adjustments will take place, and stitch the prickly bit on the inside towards the end of a shortish (say 6" - 10") strap so that it'll engage with the soft bit and accommodate your waist fitting.