THE FORMAT
Having eight pairs of loudspeakers to be thoroughly auditioned and reviewed in one day by 9 people is no simple matter...especially if the results are to be meaningful. While we have run blinded A/B testing of speakers in the past, it usually involved a 'shootout' format. This year will be a little different.
Typical shootouts involve some sort of protocols which include conjured scoring, ranking, or rating system with scoring done by a few individuals. Scores are usually given to various aspects of loudspeaker performance, e.g. bass extension, soundstage expansion, midrange clarity, etc. While perhaps interesting in its own right, this kind of reporting can be very problematic because of listener biases where the results are, unless done by strict scientific methodology, statistically insignificant and possibly quite misleading. Example: 6 people evaluate speakers A, B, C, and D for "treble clarity" on a scale of 1-5. The scores are 3.1, 3.2, 3.4, and 3.2. Does that really tell us anything of value? No. But you can bet some people will be thinking speaker C is 'superior'...though this has no basis in fact nor offers a repeatable, provable status. There are so many elements to scientically accurate speaker evaluation, that the myriad of variables are usually beyond measurable control.
So we'll be taking a slightly different approach in the 2010 GNW GTG. We will not be picking 'winners' or 'losers'. The reviewers will be offering up their impressions of these loudspeakers in descriptive ways, as you would in your own audition at a dealer, for example. Of course we will try to hone in on discreet aspects of the speakers' performance. But our focus will be on comparing these 8 sample pairs for how they differ...not which ones we prefer. In the process of setting up the format, we decided to add a session that may even pinpoint some reviewers' biases. But lest I give too much away, here is the basic setup of our busy schedule for the GTG on September 18:
Pre-Audition:
Thanks to the good will and equipment of Dennis Murphy, we will be measuring all speaker pairs in room, both open into the room and concealed by a large speaker cloth screen. These measurements will likely be published, or at the least summarized, in the results thread.
Session 1:
Using Chesky Records' Guide to Critical Listening, the reviewers will descibe their impressions of all 8 speaker pairs, presented blind in A, B, C, D, E, F, G, H order. Reviewers will report on perceived speaker transparency, visceral impact, dynamics, etc. on a standardized report form.
Session 2:
Using cuts from rock, pop, and classical selections, in random order (e.g. Dw/C, w/A, w/F, etc.), all speaker pairs will be A/B compared to all other pairs. This will also be a blinded auditioning period.
Session 3:
The screen will be removed, and all speakers will again be reviewed, in yet another random order, and reported on while hearing reviewer provided audition CD's. Prices will also be announced on all speakers. Reporting will focus again on transparency, visceral impact, dynamics, etc. We will also ask the reviewers for a 'value' description for each pair at this time.
The resulting information (some quite specific) and any data compiled should prove interesting. At the very least, the summarized reports, and any participant auditioning thoughts and posts ought to be of interest and use to the thousands of hobbyists in the i-net world that can't easily audition these fine speakers.
We're looking to have a lot of fun at the GTG. But hopefully we can offer something useful to our friends in the audio world, as well.
