See and I disagree with that characterization.
First, I disagree that a term like diminishing returns must restrict output. Second, I disagree with any characterization that wanting high output is a luxury. Why? I think exotic wood veneers, high gloss paint finishes, that is luxury. Beryllium tweeters with gold accents, that is luxury. Performance isn't luxury. To me this is akin to suggesting a corvette is a luxury car because its fast. A Ferrari is the right comparison to Champagne wishes and caviar dreams. The Corvette is the poor mans Ferrari, offering similar or better performance, minus the luxury, leading to diminishing returns. A Honda Accord is not the point of diminishing returns, it's not fast.
To say that speakers aren't played that loud so why, says who? How loud a speaker plays depends on the individual and the size of their room. In my room and listening distance (which is not that large), no standard speakers can come even close to hitting THX reference levels. In fact, the vast majority fall about 6-12dB short, which for me, is far too much. Those that can, such as Klipsch, make too many compromises in sound quality. For me, these are not a point of diminishing returns. They are still well down on the performance meter.
But diminishing returns is a subjective concept here. It is all about one values. If you listen at background levels with low dynamic range content, then dynamic range of a system is unimportant and diminishing returns is hit at a very different level. That is not me.