How are you testing it? Playing back an existing tape, or recording a new one? I'm not sure if the A77 has a way to play back from the record head, which would be handy to diagnose problems like this. If you can make it do a direct monitor, does it have the same problem just passing the signal through without any record or playback involved?
If it is a record/playback issue, it could be a number of things. You can try to establish where it's a recording issue by playing a test tape and seeing whether that has the same problem (demag the deck first to avoid damaging the test tape!)
I think you also had a Fostex R8 or something, didn't you? If you don't have a test tape it might be worth putting a recording from that through the machine at a pinch... just to get a rough idea.
Things which may be causing this:
1. Bad capacitors in the audio path
2. Electronics way out of calibration (bias?)
3. Heads out of alignment