I suspect that somewhere the signal is overcooked. Input of the mic to the preamp?/output of the preamp to the Fostex?/input of the Fostex?... How much gain others are using doesn't really help you because the real issue would be how much gain do you need on your particular equipment. When I use an external pre, the ones on the VF80 are turned all the way down. At that point I could get distortion by having my input from the mic too hot, or my output from the pre too hot. Make sure you're metering both the input and output on your pre. (Plus the meters on the VF80).
Now if the tracks actually have NO distortion individually, then perhaps with their cumulative levels you are peaking. When you play back on the VF80, the individual track LCD's show what was recorded on the hard drive -- regardless of where you set your fader on playback... When playing back, remember that the peaking LED's aren't in the loop. What are your L/R meters showing? They should be hitting at or just below the top dotted line.
In general, peaking doesn't sound like distortion... If the Fostex was peaking, you'd know (its sounds like crunching metal)
So... that leads me to suspect the problem may be in the preamp levels before it gets into the Fostex. Also, are you dialing in too much tube? Perhaps cumulatively it's too much...
Hope this all makes sense. You'll track it down.