Right, I see what you mean. What I was trying to say though is that because the world is "stereo" (Although you don't hear it as such), your brain is used to translating things on your bad side.
For instance, If a sound comes from that side, your other ear will still hear it, and it will be a...