Rule #1, if you hear distortion then turn down the volume immediately!
Running speakers from an underpowered/inadequate amplifier tends to be much more problematic than running speakers with more amp power. When an amp clips, that is massive distortion that can quickly ruin drivers.
I do believe the Crown XLS line has a "soft clipping" feature which should make speaker damage less likely, and I think they may even have "soft clipping" lights to let you know when that happens.
In general, a higher power amp tends to be more about dynamic headroom. There is a good chance that most amps will be running at just a few watts at any given "normal" volume and set of speakers. But, you want that power in reserve when the music or movie calls for it.