The first few times I wore my wool kilt (HoE mill), it itched. As I've worn it, it itches progressively less. Either I'm getting used to it, or the wool is "breaking in". But it feels different to the hand, too - feels smoother and softer. So my take would be to keep wearing it since you said you're not allergic, and eventually it'll feel a bit better.