I think for me it breaks down into how hot it is, and if I'm wearing a tie. As a general rule for daywear, if its warm and the day does not call for a tie, I'll wear a colored shirt, (usally matching the hose) no vest.
If it's a little cooler, at night, or more formal, I'd go with a white shirt, vest and colored tie.