What confuses me is that these amplifier distortion ratings are typically given at 8 ohms, and most speakers operate over a range of impedances. For example, my B&W speakers hit a low of 4.2 ohms (probably in the higher frequency area) with a nominal impedance of 8 ohms.
warnerwh, I've read that speakers normally distort at low frequencies, and such distortion is described as being less unpleasant. I think I've experienced this myself with a pair of Tannoy HPD-315A Lancaster speakers. With the tweeter turned down, there is a considerable amount of low frequency distortion audible, but this distortion is not audible with the tweeter in the flat setting.
For amplifiers, they are supposed to distort mostly in the high frequency area, which apparently sounds more disagreeable. 40 to 90 watt stereo amplifiers that I've used, having distortion ratings less than 0.5% (8 ohms), all sound the same.