Early one I wore one of my flat caps with my kilts. Looking back on it now I do think it looks a bit off. Wear what you like, but I am not a fan of the look for myself.

Having said that my flat cap is the only hat I wear besides a touque (no, not a wool cap....a touque!) after sports and training.