I wear sweater/jacket combinations a lot, expecially with kilts. I think it looks wonderful! Go for it.