The fact of the matter is that berets, balmorals, tam o' shanters, hummel bonnets, caubeens, and similar European soft hats, are all descended from a common form of knitted cap universally popular in Europe in the late middle ages and renaissance periods:



So, I say go ahead and wear a beret! Pom-poms weren't added to Scots bonnets till the latter 1700s, and many continued without them for decades after - looking quite "beret-ish"...!