Agreed. I think the character of distortion affects its audibility. So do a lot of other people.
In the English language, yes, frequency response deviations are distortion, amplitude distortion to be specific. In the audio world, however, distortion typically refers to artifacts in the output signal that aren't supposed to be there, like harmonics of a fundamental, noise, or some sort of waveform distortion, like clipping.