If you are going casual why not just wear the shirt open at the neck? Any kind of jacket is a bit more formal and the picture of the man wearing a jacket and waistcoat looks terrible without a tie.