There's a little bit of noise in just about every step of the recording and reproducing chain. A lot of it sounds like a hiss. As a general matter, you want to make sure your signal is in the vicinity of what's nominal at each stage of the chain, and not way under nominal, where it gets all tangled up with the noise. As a more particular matter, it sort of depends on where the hiss is getting into the signal chain: air ducts near the microphone? The mic's electronics? The mic cable? The mic preamp? The mixer? An outboard effect box? An A/D converter? A tape deck? The cable connecting any of these? Recording signal too low (either analog or digital)? A D/A converter? A tape deck again? More cables ... mixer ... more processors ... amps ... speakers ... snakes?
Ultimately, if you're going to have loud parts and quiet parts in the same recording, you're probably going to wind up with some audible noise in the quiet parts. Frankly, I don't think it will kill anybody. It may even be part of what makes the quiet parts sound quiet, the way you want them to. Others may disagree.