Have you got a healthy gain on the mixer? The VU meters should be well lit. I am also assuming that you have turned the recording volume up on audacity, its at the top of the page where the little microphone symbol is.
The rca inputs are on the same sector as monitoring back from the USB device, so you will not be able to record from these (they do this incase you are using a separate USB device), plug the mp3 player into stereo line inputs.
Also, if using a windows machine you also have to set the volume on the windows mixer, control panel > advanced >options > properties > (select mixer device to input) > (tick controls) > click OK, then adjust volume.
Alan.