I don't think it's a "mismatch". The non-linearity thing I was talking about was with the tweeter. The tweeter is a metal dome. Unless these are made correctly, they can ring at high volumes, which is probably what you were hearing.
Another cause of a "harsh" sound is sometimes an amplifier near clipping.