I agree with the odd looks if you look odd. I remember going to Spain in March one year, it was -5 or so here (a really brisk end of February), and 50F there. So, I was walking around in a tank top and jeans. Add to the fact that I was 2-3" taller and a good 100# heavier than anyone we met (and a nearly 2' ponytail), I got lots of odd looks. One elderly woman in Toledo told my mother that she needed to get me some clothes before I froze to death. The second time I went (some years later), I wore a utilikilt for a good part of the trip. Oddly enough, it seemed to fit with the rest of me, so I actually caught less attention than the first time around.

Pretty funny stuff.

So, incongruity (whether with the surroundings, or with yourself) will be noticed. But, with the right non-belligerent attitude, everything works out fine.