First of all, when using a 4-track cassette unit, there is no "side b." You use the full width of the tape in one direction. This is what gives you more than two tracks.
Secondly, it's possible that one of the internal belts is slipping, indicating the belt is worn and needs to be replaced...