Well as I often say, I'm no fashion expert, but in my pre-kilt days I was a pretty traditional dresser when it came to suits or formal wear. I was always told by my elders that the rule of thumb was that button-down collar shirts, keeping with their sporting origin, go with sport-coats or blazers, worn without a tie.
A traditional suit and tie should be worn with anything but a button-down.
I will say that I often see people wearing button-downs with suits here in Texas, and in the South in general I guess , but I don't care for it.