At very low listening levels, this would be true. However, a speaker will have increased non-linear behaviour as it's piston movement exceeds the linear limits of it's magnetic gap and/or suspension components. As you reduce the low frequency content going to a speaker, you reduce it's amount of movement, resulting in lowered distortion(s) that occur as a result of these nonlinearities. And at a point, especially with high amplitude bass, the non-linearity can become extremely audible on most small speakers at moderate to high SPLs. Even in the absence of non-linear distortion effects (not a realistic situation), in extreme cases, a speaker can even cause a phase modulation effect, if the cone moves enough to have inter-phase cancellation/summation effects of it's highest frequencies as they emit from different points in space periodically as a result of the low frequency high amplitude components forcing the the small diameter speaker to move substantial amounts when producing moderate to high SPLs. This effect is commonly referred to as Doppler distortion with speakers, although technically Doppler distortion is defined differently. But the reference remains common.
-Chris
