Being "covered" indoors (now that I'm way far away from the Army) is a personal choice. Yes, there are "some" guidelines - places of worship, organizations with dress codes (schools come to mind), or when dealing with the military or para-military organizations. But just look at how many men sit down in a restaurant and eat an entire meal without removing their headgear. (Personally I don't understand that - but that's me.)

So "right" or "wrong" (most places in the US) is more a matter of upbringing/personal preference and local custom than anything else.