I'm with Blu.

When I'm bumming around the house, I'm either in my robe (in the morning) or I wear loose shorts. If I'm leaving the house, then I change.

(Oh, and I'm not one of the 24/7 guys; I wear kilts maybe about twice a week or so.)