I looked at the title of this again, and perhaps we've veered off on a vector.

Yes, it does matter. Neatness and appropriate scale vis-a-vis the collar and your own size will make it, and you, look more... or less... well suited (or kilted) for the milieu in which you find yourself. For second-rate settings, it's a second-rate issue, but for the office or fine dining I think it's absolutely essential.