I heavy never used the TASCAM USB 122, but from what I could find online, it does not have built in pres. What your running into is most likely the distortion caused by putting a mic level (-30 to -60dbm) signal into a line level input (+4dbm). Where a pre-amp will benifit you is by properly converting the mic level signal to line level. I am also guessing that since you are not using a pre-amp, you have to have your gain up super high to get a decent signal. Once again it is because the TASCAM USB 122 is expecting a line level signal and getting a mic level signal (creating even more noise for a improper gain structure).
If you are getting excessive noise from the Tascam, you have something set up wrong or another issue. Even the most basic interface nowadays gives a reasonable quality.
I have XLR from the mic directly to the Mic in. As someone else mentioned, I have to turn the gain all the way up in order to get a decent volume. If phantom power is on, the volume is higher, but so is the line noise.
Something going on with the mic or cable, if the mic needs phantom it won't work with it off, if the mic does not need phamtom it won't make any difference if it's on or off.
As asked above, what is the mic, is the cable a proper mic cable with 2 cores and a shield (not a guitar cable with a XLR on the end).