I'd forego the tie (and the vest). Whether you wear a sweater or jacket is up to you (and I'd take the weather and venue into consideration). Your other accoutrements should then follow suit.

'Holiday casual' does suggest the kitschy holiday-themed sweaters, etc. and lots of red and green.