Get a cable with a 1/8" stereo jack on one end and RCA plugs on the other. Connect the RCAs to your Xenyx's RCA tape outputs. Plug the other end (1/8" stereo jack) into the Line Input on the Macbook.
Set the Xenyx's main volume level to 0 db.
Plug your vocal mic into channel 1 on the Xenyx. Set your fader level (bottom knob) at 0 db. Bring up the gain knob until you're showing a signal that's peaking around 0 on the meters. Pan channel 1 all the way to the left.
Plug your guitar or POD or whatever into channel 2 on the Xenyx. Set your fader level (bottom knob) at 0 db. Bring up the gain knob until you're showing a signal that's peaking around 0 on the meters. Pan channel 2 all the way to the right.
When you play your guitar, only the right meter should be moving. When you sing into the mic, only the left should be moving.
Open Garageband on your computer. Create a new real instrument track. In the Track Info to the right (should show up when you create the track), set it to record from "Channel 1 (Mono)". Turn "Monitor" to "On" and move the volume slider all the way to the right.
Create another new real instrument track. In the track info for this, set it to record from "Channel 2 (Mono)". Turn "Monitor" to "On" and move the volume slider all the way to the right.
Arm both Garageband tracks to record, and hit record. Play your guitar and sing. The guitar should be on one track while the vocal is on the other.
You can continue to add tracks this way, where whatever is panned to the Left on your mixer will correspond with Channel 1 (Mono) in Garageband, and likewise for things panned Right corresponding with Channel 2 (Mono).
To monitor what you've already recorded while you're doing new tracks, use the speakers or headphone output on the Macbook. Alternately, you could get a second cable with a 1/8" Stereo plug on one end and RCA plugs on the other. You would plug this stereo plug into your headphone output on the Macbook, and connect the RCA plugs to the Tape In RCA jacks on the Xenyx. Connect your headphones or monitors to the Xenyx (headphones to headphone jack, monitors to CTR Out jacks). Press the "Tape/CD to CTR" button. DO NOT press the "Tape/CD to Main Mix" button or you will create a feedback loop. Use the "Headphone/CTR Room" volume knob to adjust the monitoring level. Leave the Main Mix volume level be.
If this doesn't work I don't know what to tell you.