The real truth is that rounding errors do affect all of our signals, not just the quietest ones. The following is not exactly how it works, but close enough if you squint.
Assume our signal swings between -1 and 1. Assume our chose bit depth allows us 5 decimal places of precision - about like 16bit. The largest signal we can represent short of all the way on is going to be 0.99999. Turn that down 20db, and we have 0.09999. Turn it back up 20db, and you've got 0.99990. That's a rounding error. You really have decreased the the dynamic range of the signal by 20db. You might not really hear that noise on louder signals because it's small compared to everything else going on, but it's there, and at -80db, it's starting to be noticeable.
But a 32 bit floating point mix engine gives us 19 decimal places! In order to "reduce the resolution" back to the "16 bits" we had above, we have to turn it down by 260db and then turn it back up. In some of my noise work, I do stupid shit like that, but in the real world you'll never notice it.
To the OP - the gain plugin (or whatever gain control is available in your DAW) does exactly the same thing as the fader, so adjust it where it makes sense to you. There are good reasons to shoot for keeping your faders around 0, but loss of resolution is not one of them.