You also have the "heat" issue with amp clipping, it forces the coil to move in an unnatural manner causing heat... not to mention that the actual cone bends in all kinds of fucked up contortions which causes the paper to go soft [and / or the surround] netting you shit for sound.
At the speaker level, distortion created in the program material [i.e. distorted guitar/vocals/etc.] is OK... distorting the actual speakers either by overdriving the power amp or the speakers themselves is a bad thing [even with distorted guitar sounds, you really want to minimize crushing the speakers unless you enjoy changing drivers on a regular basis... once the speakers in a guitar cabinet "go soft"... you can't buy a tone from them, you have to have the drivers reconed or replaced].
Peace.