I wonder if you've actually listened to see which speaker sounds the best? When --I originally read that in the guitar recording bible I found on here--can't remember the author--he mentioned that at some point, you've got to take your ear plugs out, and actually stick your noggin in the fuckin cabinet. He said it would instantly jump out at you which speaker sounds the best.
When I read that, I thought it was bullshit. But then I tried it, and instantly I was like "yeah, this one does sound better than the other." Just something to think about. It seemed your speaker choice was arbitrary.
He also mentioned that to test it with your head at full volume you make an instant decision, as if you are that close to the cab at the volume where you have total speaker involvement--i.e. everything vibrating harder than... you know

--you can stand about a minute without damaging your hearing.