And I am sure clinical researchers and statiticians would scoff at this test...
I am one of those clinical researchers and I have no problem at all with this test. Yes, instead of varying one component at a time, they compared two different systems where nearly everything was different except the speakers. But because they concluded that it was difficult for the listeners to consistently hear differences between the two systems, we don't need tests of individual componets from those systems. So their conclusions were fair and their methods were good enough. I don't see a weakness at all. If the conclusions were different, then we would need a lot more testing to isolate what might be the source of those differences.
I do have one tiny quibble with people who use the term DBT for this. This was not a double blind test. The listeners were blinded but the testers were not. It was a blind test, sometimes called a single blind test. Call it a SBT not a DBT. In this case, there is nothing wrong with that. There is no reason to believe that the testers were intentionally or unintentionally affecting the results recorded by the listeners, or that a genuine DBT would yeild a different outcome. The kinds of tests that really do need DBT conditions are large clinical trials of an experimental drug compared to a placebo, where both docs and patients do not know who is getting what. So let's please remember the difference
.
My favorite quote about the problem of audiophiles who refuse to believe blind listening test results comes from Floyd E. Toole, who is the pioneer in doing these tests.
“Knowledge of the products that are being evaluated is generally understood to be a powerful source of psychological bias. In scientific tests of many kinds, and even in wine tasting, considerable effort is expended to ensure the anonymity of the devices or substances being subjectively evaluated. In audio, though, things are more relaxed, and otherwise serious people persist in the belief that they can ignore such factors as price, size, brand, etc. In some of the “great debate” issues, like amplifiers, wires, and the like, there are assertions that disguising the product identity prevents listeners from hearing differences that are in the range of extremely small to inaudible. That debate shows no signs of slowing down.”
