"Button-Down" collars were originally designed as casual shirts- the buttons secured the free ends of the collar so they would not flap about while one was playing polo.

Although I've never played polo, I typically don't wear them with tie and jacket/ suit. I usually opt for a spread collar as I like a fuller half-windsor knot. I do wear open collar button-downs during the summer with khakis and loafers, which is a sort of unofficial male uniform here in the South.

All that being said, not wearing button-downs with a tie is often a custom "more honored in the breach than the observance" and you'll find many men dressed this way. Iin the same way that you'll see men wearing casual loafers with suits, instead of a more appropriate laced shoe.

Cordially,

David