My Irish fisherman's knit sweater has a reasonably looser weave than a military-style wooly pully, and I would be worried about having the belt buckle catch the sweater, even for a belt with no sharp edges. I would opt for wearing the sweater without a belt for this reason alone. Additionally, my mother knitted it for me and it is not replaceable. Just my opinion on the subject, mind you.
May you have warm words on a cool evening, a full moon on a dark night, and a smooth road all the way to your door. - Irish Blessing
Bookmarks