...
For the new user, a lot is to be said about getting familiar with the settings and signal routing options & how to work them. Have you consulted the manual? A manual can be downloaded from the Tascam website.
There's the unknown of how your settings and workflow are situated before any valid troubleshooting or guesstimated references about problems could be made.
Where are you located? Is there anyone you know with Portastudio experience who could demo the unit and verify it?
I'll just float these ideas at random:
When the channels 1-4 are switched to "Tape", the signal comes off tape, flows thru the mixer channel and then the master fader, with Line Out switch set to "L/R" the signal should be heard in the phones and off the Line-Outs on RCA connecors.
When in recording or overdubbing mode, the offtape signal should be heard in the headphones and Monitor-Out lines on RCA jacks, when the Effect2/Cue knobs are turned up, the Monitor section has to be set to Effect2/Cue (Cue); Effect2/Tape Cue (Tape Cue); then controlled with the Level pot which controls the headphone level and Monitor Out level.
Good luck./DA