The audigy cards have terrible, terrible self noise.
It has to do with the architecture of the input bus- the line and mic inputs are both open when you unmute on or the other, so you get a fair amount of noise.
In all honesty, ditch the audigy unless you are using it exclusively for digital in.
If you have to use the audigy, then, as the others mantioned, turn off the 20dB boost and use the line inputs w/ the signal chains mentioned above.
I would suggest that your next purchase be that of a halfway decent soundcard/AD converter. Sine you've already got a mixer, you may as well jsut find one of the 2-channel USB/FW jobbies, like the tascam US-122 or a M-Audio Audiophile.
The A/D/A converters on the Audigy aren't exactly the greatest.
However, Audigys have their place. Mostly for gaming, but if oyu've got het front panel breakout you've got a couple of extra digital IO spots, which sometimes come in handy.
Just remember that once you've got a digital signal, it's a lot more hardy than an analouge signal.