Probably has nothing to do with the phantom power, you're likely introducing too much gain and the noise that's already in your signal chain is being amplified right up front. I can't imagine why you'd need to "dime" the input level. You might consider an external mic preamp with phantom power.
