One of the goals of a competitive benchmark loudspeaker listening test is to provide feedback to the engineers to optimize the sound quality of the speaker. So if the test " turns out badly for the house brand" the feedback from the listeners is used to fix the loudspeaker, and it is retested.
Certainly, one would hope that that is how the information would be used. But it would be good to fix whatever problems a speaker might have without it having to lose to other speakers; one hopes for the best it can be (for its price).
As I pointed out earlier, the comprehensive anechoic loudspeaker measurements don't lie. There are 25+ years of scientific research that have shown that the speakers with the best set of technical measurements tend to score the highest ratings in controlled listening tests. This was true when I worked at the National Research Council (a disinterested 3rd party) 20 years ago, and it is still true today.
I would like to see comprehensive measurements on a number of old speakers that I have heard and liked, as well as the competition at the time. I do know that the single on-axis frequency response isn't enough, but that isn't what you are saying. Rather ironically, some of the speakers I have liked have had less good specifications than some of the competition that I liked less well. For example, when the Ohm Walsh 2 speaker came out, I liked it very much for a speaker at its price point (which was about $750 in the early 1980's, if my memory is right). But as I recall, their frequency response specification was with a worse tolerance than usual (+/-4dB), and even so, they did not go out to 20kHz (not that there is much musically at such a high frequency, even if one is able to hear that high). The Ohm website lists the frequency response as 42 - 17,000 Hz, but without a tolerance specified (which, again, if my memory is correct, was originally specified +/-4dB). But I have not seen comprehensive tests of this speaker, nor have I seen anything for off-axis response (which, for frequencies below the crossover point for the tweeter, should be identical to on-axis response, until one gets behind or nearly behind them, given the Walsh driver and the blocking of the sound to the rear).
As for your statements:
The most preferred speaker had the flattest on-axis response with the smoothest off-axis curves. The least preferred speaker had among the worst frequency response in terms of flatness and smoothness.
I am not disputing those claims, as it is more than just on-axis response that is important. And I do expect that overall preferences could be correlated with appropriate measurements (I don't think speakers are magic devices), though I do not pretend to know precisely what measurements matter or how much they matter.
When I think about several of the speakers I rather like, there is often something unusual about the dispersion pattern, as Ohm Walsh speakers and Magnepan and Apogee ribbon/planar speakers.
But not all of the speakers I have liked very much are so unusual; the U.S. version of the Aurum Cantus Leisure 2SE is my favorite bookshelf speaker (obviously, my favorite must be selected from among those I have heard, so this is not a claim of it being better than all others). I have not seen comprehensive tests of any of these speakers, so I cannot say how well they correspond with the idea that the best speakers have "the flattest on-axis response with the smoothest off-axis curves." It is unfortunate that such information is simply not available for the vast majority of speakers.
I would also be interested in participating in double blind listening sessions with some of my favorite speakers compared with others, as it would be interesting to find out if non-audio considerations have influenced my preferences. Certainly, I do not expect to be immune to things that influence human perceptions, but I do know that it isn't as simple as being a sucker for exotic speakers, as otherwise I would like Martin Logan speakers much more than I do (I have never heard a Martin Logan speaker that I liked very much, even though I wanted to like them and had heard that they were supposed to be superb).
But I would really like to see proper measurements of the various speakers, to know what it is that is different about them.
