Since when did the MR16 have RCA sockets, mine are all 1/4" phono?
First:-I presume you are trying to apply effects to your raw recording, in which case the Aux Out are the better choice as you can locate to individual tracks (including 15/16 if you have a final mix down) Second:- When you assign the output of the Effects Processor to the recording tracks, make sure you have pressed the Rec Select keys for those tracks and keep the faders for those tracks to 0 unless you are listening via a separate speaker system. (this is why you use the Aux Out sockets and not the Stereo Out sockets) The Stereo Out and the Headphones Out come from the same signal source i.e. the stereo Master Buss. The Aux Out can be connected to any individual recording track.
One trick I have used is to Record on the 1/2. Set Aux Out to 1/2 and, using a patch lead, feed the signals back through Inputs A/B into tracks 3/4. I then switch on the Input Eq for Inputs A/B and test the effects available in the EQ Library. These are pretty limited but can be useful at times. So whether you use the EQ Library or you patch the raw signal through an effects processor, the process is exactly the same.
I think the Aux Out sockets are only available on the MR16 but they are very useful. You can apply effects as described. You can connect Aux Out to another mixer and blend in new signals through that remote mixer or you can just patch the signals back to re-record. An example of this may be to copy 1/2 to 3/4 and then, using the clip board, copy the 3/4 signal with a small time slip back. Mix 3/4 with 1/2 and you get a "wall of sound" effect.
