This depends on a lot of factors, such as what type of tone you're going for, what type of amp it is, etc. I can tell you with regards to my experience.
I mostly use older amps and/or simpler designs that don't have master volumes or gain controls. So if I want a dirtier sound, I'll crank the amp up to get the overdrive. Depending on the wattage, the volume will vary.
On my Gibson GA-5 Skylark, which is only about 10 watts, I can dime it, which breaks it up very nicely, but it's still not earth-shattering by any means. On my homemade Marshall 18-watt clone, I can do the same thing, and it still doesn't peel the paint off the walls.
I mean, you're not gonna want to stick your ears a few feet from the speaker or anything, but you don't need to leave the room.
For cleaner tones, obviously I have to have the volume lower. But I still usually like to move a decent amount of air. Sometimes for certain clean parts, low volumes seem to work ok.