Well...
This sounds rather strange. I have a DMP3 as well, and it is certainly a hot thing...
I'm running it into my sound card, which is at nominal -10 dBv level, and on my setup, I can hardly approach any levels on the VU meters before it severely overloads the inputs.
If I take my SM58, put the gain on hi (out position) and the knob at around 3 o'clock, I get around 0 dB on the VU meters at normal speech levels close-miced (like you normally use a SM58 on stage). If I raise my voice, the VU meters peak, and the clip light comes on from time to time, but only momentarily.
You should get similar results with your SM57 -- which I think is what you say also?
However, if I run that into my sound card, it overloads drastically. I need to back down to the 12 o'clock position, at which the VU levels read around -10 dB. I now get peak levels just below 0 dBFS (i.e. I'm close to clipping on the input converters in my sound card), but the reading of e.g. wavelabs VU meters now also hoover around -10 dB.
This is what I would expect also. The DMP3 is calibrated to output +12 dBv (RMS, I think) when showing 0 on the VU meter (for a 1 kHz sine tone) -- which is very hot! It means the peak level will be at +15 dBV, and then the overload margin is still up to 7 dB before maxing out at the specified +22 dBv (18 Volt!). This is balanced output, and since I'm running it unbalanced, I loose 6 dB. But still, to get it down to the max input level of my soundcard, I need to back down further to avoid overloading.
Note that there is a difference in the way a VU meter and a peak meter works. A VU meter averages the music, and the reading you get is more related to the loudness of the music. A peak meter (here: the LED that flashes) shows you when even a single peak of the waveform is getting closer to clipping. You can have one take sound at e.g. -8 dB VU without peaking and another sounding at -10 dB VU but peaking from time to time, and they may both be roughly equally loud sounding.
So, peak levels often read more than 10 dB hotter than the VU reading.
Furthermore, the DMP3 VU meters are output meters, and the clip LED reflects input level (after the preamp). So, you should make sure the LED only lits up momentarily (which it does 4 dB before "soft" clipping in the intermediate gain stage), and use the VU meters to check the overall levels.
If your recording device is spec'ed at +4 dBu, then you should still have plenty of headroom, but I guess you would need to have the gain knob set at 3 o'clock to make it hot enough.
Here's some reading to do on a long and sleepless night
http://www.digido.com/portal/pmodule_id=11/pmdmode=fullscreen/pageadder_page_id=36/
http://www.digido.com/portal/pmodule_id=11/pmdmode=fullscreen/pageadder_page_id=59
http://www.soundonsound.com/sos/1996_articles/may96/metering.html
So much for the long winded theory...
I assume you have tried this with only one mic at a time also?
And it is a problem with both of your mics?
Are you running balanced on unbalanced from the DMP3 to your Fostex? I assume you're using a line input. If there is a gain stage/knob to twist, make sure it is set at line level (which it normally is if set around 2 o'clock on many devices).
Hope this helps explain more than it confused,
-- Per.