If I wore kilts around the house, I'd have to guess I'd take off the sporran, much as I take my wallet and watch off when I get home from work.

I'd never wear a kilt out of the house without it, though.