Yeah, maybe you need a hotter iron/bigger tip. You basically have to have the metal you're soldering to hot enough to melt solder in order for it to stick. And since the ground goes to the body of the plug, there's a lot of metal mass to heat up so it takes a long time. Make sure you are setting the plug on something insulated when you are doing this, a metal vise or pliers will suck the heat out of the metal in the plug, making it even harder.