Bulls Hit said:
....If I record my project out to a casette tape, I can up the levels on the tape deck until the peaks are around +3db and the tape sounds good with no distortion.
Yep - that's the essential difference. Tape you can push as hard as you like and it will just saturate more as opposed to digital which will just sound more and more harsh and 'clippy'.
You are right in attempting to raise the individual track levels prior to mix down. You are also correct in attempting to use a compressor to do the trick. What it sounds like you are not doing is adding a peak limiter after the compressor at the individual track or bus level.
Any setting you dial in on a compressor is not (normally) going to be set to have an attack time of "0"... You always (normally) let some of the transient through to retain the proper dynamic balance of the individual notes. So, the body of the notes get compressed, the compressor then applies however much make-up gain and all the while your transient peaks are being ignored by the compressor and have been made louder by the make-up gain.
If you slap a peak limiter on after the compressor set to only catch the most extremely high signal and then let it go really quickly, you will be able to raise the level without peaking off your head.
Apply that logic at a track/group level, then at a bus level and finally at the master bus stage. Do that, make sure nothing is hitting the red anywhere in the chain and it should sound both loud and good.
Once you have that exported to a single stereo .WAV file you can bodge it up even louder in an attempt at home 'mastering' using judicious amounts of the same process.
Ciao,
Q.