Cold temperatures aren't likely to damage speakers, however cranking the volume before they have the chance to thaw out can do damage. If you consider cracking the cone, or separating the cone from the surround to be damage (I tend to think of this as damage.)