Sure. Higher IM distortion (direct result of increased cone excursion), thermal compression, and the driver's thermal and physical excursion limits are the sort of factors that come into play as you lay on the power. Speakers have their limits, after all, and they'll sound worse when operated close to those limits.