I don't know of any rule, but I'd caution the wearer to be conscious of his color choices and how they mix. For example, red and white diced hose worn with green garters would look awfully Christmassy, and the some hose worn with blue garters would look very flag-like, for both the Stars and Stripes and the Union Jack. Some wearers might want that kind of "theme" look, and if done with some forethought, it could be very impressive. Heraldic colors could be mixed in this fashion, and those who look for such things would have a nice topic of converstion when they spotted someone so accoutered.