For me, it's only when it's too loud. At lower levels I can make sensible decisions for all of the working day. As I got older, loud got more unpleasant and now I find I am sticking in ear plugs more and more, so I can go back to my modest level position, and be effective. Walking next door and having to stick them in is a real nuisance, but pretty vital. I still do the odd stint mixing for picture in tiny little spaces where the small speakers are wound up quite high, and there is continual chat from the comms - which you need to mentally filter - and as long as you keep everything reasonable, level wise, you can do a shift quite effectively. For me - the usual problem is with bass. Modest monitoring level and deep bass is tricky - so at the start I get that balance roughly done, then listen on my headphones (oddly DT100's) but I know what they sound like and they are sealed, so I get bass and kick drum balance pretty well pegged, then I take them off and see what kick and snare sound like on the small things at lower volume. Often the speakers are Genelecs, and appear to be a bit weak, bass wise - but I can hear what 'proper' mix level is, and just remember that on the output, there is plenty of bottom end - I just cannot hear it because of the speakers and the lower overall level.
What I cannot do is make sensible decisions when it is really loud.