I own an M-50, which is the same vintage as the 512/520 and shares the same architecture. Here's a run down on how to get a sound out of it. I apologise if you already know all of this, but as the other member said, these desks have a weird setup, you actually have to sacrifice two of your eight subgroups, and use the monitor section, to send a stereo mix to your stereo master output. Why didn't they add a dedicated L/R buss and assignment switch????
Start this process by setting all faders to nil, and panning everything to centre, and disabling all buss assignment switches, and turning all buss monitors to OFF (centre the bats), and deselecting everything in the switchrack.
Take a paired RCA cable, connect the ST MSTR OUT L/R rca output to your mixdown deck/stereo amp. This is your audio output. If you want to remove some circuitry from your output path, use Buss 1/2 as your main stereo mix and patch staight out of that to your mixdown deck/amp. EVERYTHING will have to be bussed to 1 & 2. Panning will then be determined solely by the channel panning; you are not using the monitor section, its volumes or panning. Doing it this way loses some input channels; through the monitor section you can actually use 2track A & B as extra input channels (without pan or eq) that are sent to Stereo master A by selecting them in the monitor switchrack IN ADDITION to anything else you are monitoring there. That's one of the keys to this desk, you can monitor multiple sources simultaneously via the switchrack. Later TASCAM's, like the 2524, allow only one switch on this rack to be selected at once.
Back to getting a sound: connect an RCA cable to one of the OSC OUTs. Connect the other end to the TAPE IN of channel 1. Set the CH1 input select switch to RMX, set the outer concentric pot to about 2 o'clock (roughly 7). I can't remember if you need to activate the test tone, play around with that to find out. Set it to 1k.
Depress the buss assignment switches for buss 1 & 2. Pan the channel centre. Pull the fader up to 7. Go over to the buss faders, bring busses 1 & 2 up to 7. Go to the monitor section, switch mon 1 & 2 to BUSS, pan 1 hard L, 2 hard R, volume for each to 7. On the monitor switchrack, select MON only, make sure the mono switch is not on. Bring up the STEREO MASTER A to 7. You should hear test tone in the centre of the stereo image. Go back to channel 1, and play with the panning, it will move according to this pot.
Now pan CH 1 hard left; on Buss 1 & 2 monitors, pan them both centre. Test tone will be centred - you've just set up a mono mix by panning your buss monitors centre. No matter how you pan CH1 it will be centred, but the volume should increase as you pan towards centre on CH1.
Some further experiments to try: get about four CD players 9or tape decks) , send two of them into 2TRK A and B, send another two to the tape ins of
CHs 1 to 4. Monitor the lot at once, then figure out how to monitor the ones you want to by selecting and deselecting various buss monitors and switches in the switchrack. Play with the panning at channel and buss monitor stages. You can even patch directly into the buss faders (PGM IN 1-8), this signal will come up on the buss fader then go to the monitor section.
Someone just handed me a beer, so I've lost my train of thought. If you have more questions, I can probably answer them, I might even be able to mail you a scan of a block diagram from the M-50 which would be very close to your setup. Everything here applies to the M-5p, but I'm certain these same concepta apply to the M512 and M520. (I just went to buy a 512, but it was pretty rough and the guy was insulted by my low offer).
Good luck.