Is it possible to remove the speaker plug on the backside and only use the XLR output when I record something, or is it going to break down if I do so?
The amp must have a load. If it doesn't, you'll run the risk of damaging the amp. However, I bet the line out works without having the amp on. Just flip the power switch on without turning the standby on. Connect the XLR to your interface and record.