I don't see how incorrect polarity of one speaker vs another would damage an amp but you never know...
As for the music itself, it doesn't matter if you hook up all the positive terminals of the amp to the negative terminals of the speaker and all the negative terminals of the amp to the positive terminals of the speaker (as Haloeb said).
A musical waveform contains a larger number of sine waves all superimposed on top of one another. The wave starts at zero, rises, then falls below zero, and rises to zero again. So the only difference between the 'correct' (+ve to +ve, -ve to -ve) connection and the other way around is that one wave will rise first whereas the other will fall first. Now if one speaker is connected correctly and the other is opposite, the two waves are 180 degrees out of phase and you can tell a slight difference.
Experiment for yourself if you have an audio editor:
- If you take a waveform in an audio editor and invert it you have the same situation as if you used the 'incorrect' (+ve to -ve, -ve to +ve) connection for all speakers. It won't sound any different.
- Take the same waveform but invert only one channel. Now you have one channel 180 degrees out of phase with respect to the other. Can you tell the difference? Sometimes yes, sometimes no.