And as a manufacturer of preamps with transformers on the input,
I would also recommend turning the phantom off before removing the mic, and on after connecting.
The current pulses generated by connecting with power on are of short duration, but can be of surprisingly high amplitude depending on the various circuit configurations, and these pulses are what can magnetize transformer cores. Mic and mic-pre alike.
The input capacitors of transformerless preamps have the full 48 volts with respect to ground if the phantom is powered up first, plugging a mic in will discharge this power thru the mic and the cable capacitance, this can be a strong spike.
In transformer coupled preamps, the current is limited by the 6K8 resistors, but the 7ma of dc smacked repeatedly into a core can be an issue over time. That's why Studer recommends demagging the cores of the input transformers in the older boards.
I'd rather not magnetize anything, so removing the phantom is best before changing connections.