An Overview of Testing Methodologies

[gene]

<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><A href="http://www.audioholics.com/news/editorials/TestingMethodologies.php"><IMG style="WIDTH: 125px; HEIGHT: 69px" alt=[test] hspace=10 src="http://www.audioholics.com/news/thumbs/test_th.jpg" align=left border=0></A>Double-blind, single-blind, ABX…all these terms and more are bantered about on the forums and elsewhere when discussing someone’s subjective experience with a piece of audio equipment.<SPAN style="mso-spacerun: yes"></SPAN>This editorial is meant as a primer for the testing methodologies used to validly measure someone’s subjective experiences.<SPAN style="mso-spacerun: yes"> </SPAN>This is not a definitive guide nor is it meant to be.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Its meant for the layman wanting to understand the different testing methodologies through rather h<SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">umorous anecdotal examples conjured up by our own </SPAN><?xml:namespace prefix = st1 /><st1ersonName><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Tom Andry</SPAN></st1ersonName><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">.<SPAN style="mso-spacerun: yes"></SPAN></SPAN></P></SPAN>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">[Read the Editorial]</SPAN></P>

 

[nibhaz]

Great Article!

A truly entertaining article on a rather dry, but very important subject when pursing the truth in audio.

 

[MBauer]

Excellent article

Good insights and very entertaining, especially given a subject that can be quite dry, congrats to Tom

 

[Sheep]

Its funny how this article came out right when Jaxvon's B&W review starting getting all...."heated"


Sheep

 

[Tom Andry]

Its funny how this article came out right when Jaxvon's B&W review starting getting all...."heated"


Sheep

Remember: Correlation does not equal __________.

I started this article a few weeks ago when my discussion about double-blind methodology became heated.

 

[Sheep]

Remember: Correlation does not equal __________.

I started this article a few weeks ago when my one discussion about double-blind methodology became heated.

ahh, everything is becoming heated. People to often get their panties all swisted in a knot about nothing, oh wait, excuse me someone just said all channels driven tests are important... hey you, ya you, those tests *voices fade out in the distance*


Sheep

 

[Tom Andry]

ahh, everything is becoming heated. People to often get their panties all swisted in a knot about nothing, oh wait, excuse me someone just said all channels driven tests are important... hey you, ya you, those tests *voices fade out in the distance*


Sheep

At least they didn't say cables make a difference...that would have caused a riot!

 

[Sheep]

At least they didn't say cables make a difference...that would have caused a riot!

Tom, I bring your attention to this....

To jaxvon:

From my own experience with the b&w 705's I have to tell you and others here that the speakers are NOT bright! They are tonally almost perfect. They might be erring just a small bit towards bright, but that is something you can use! Not a serious flaw. What you heard on this demo meant many other things. The combo was not good, or the components and/or the cables. Judging by how you described the audioquest cables they must have been the gibraltar which are pretty good. But there are still better (or perfect I would dare say) cables for such speakers. The thing is that these speakers tell the truth about what you connect them with. And if you found the result you heard terrible I guarantee you that with a Pioneer receiver and dvd player that are not really meant for serious music reproduction the result could be even more terrible! I tried these speakers home with my system which is Musical Fidelity A300 amp, AH! Njoe tjoeb cd400 reference and wired all around with Harmonic Technology wires. Which is very important. At that time I was trying also different cables. The HT cables work best as a whole. Speaker cable, interlink, power cables. Each HT wire I was adding to the system, the sound improved becoming more refined, natural, transparent, open with excellent soundstage. I tried also Audioquest cables which are quite good but couldn't match the result of the HT's. Actually no other reasonably priced hiend cable can match them to my opinion. Everything else I tried with the 705's was missing something. Some cables were giving terrible results as well. Or sounded more grainy and less natural. And I tried HT cables with more speakers, not just the 705's! And the result was everytime similar. Musical purity with minimised distortion and grain.
Also my cd player that has the siemens E288cc tubes althought is quite detailed, it has smooth high's. The A300 amp has a perfect tonal balance with low distortion and grain. I'm not saying I have the best components, I'm sure there are better out there. But what I'm saying is that the synergy counts more than the parts. If you make a bad combination with the 705's you will hear the results. Terrible!
It seems that you do like some things about b&w speakers just as I do, judging also by your mixed feelings in that audition. Don't forget also that bad recordings cannot hide from these speakers. They will sound what they are, and worse with wrong components and cables. I just say that either you have to find another speaker that you think suits your taste, but try it in your system! Because an audition at the store will sound quite different with different room and system. Or give the b&w 705's a different combination. And because of personal experience I tell you: don't waste your time with time with sub-standard cables. They make a BIG impact to the sound. For better or worse. You can't go wrong with HT wires. With many others, including some fancy brands, you can!
But you try things out yourself and trust your ears! But try to biwire speakers like the 705's. Or if you don't, never use the metal jumbers! They add to this 'terrible sound' experience! Better use a decent jumber cable, preferably from the same brand (even better: the same wire) as the speaker cable.
Also the 705's like the cdm-1nt's sound better (more refined) with the tweeter grill off! It seems that this cover creates distortions and sibilance.
If you must have other speakers and want this kind of openess, detail and soundstage, then you could also try AVI's neutron-IV and Pro-9, or Epos speakers, or Audiophysics to name a few. The AVI speakers seem to be the most monitor like sound, very neutral but they are also very detailed and honest so matching needs good care, just as with the b&w's. And each speaker has it's own strengths.
I hope my opinion helps some people that are at the look for such speakers.
Cheers

No riot yet.......yet

Sheep
EDIT: if you were joking, bah!

 

[Nomo]

I kept waiting for the punch line. Tom are you getting serious on us?
Just kidding. Nice article.

 

[Tom Andry]

I kept waiting for the punch line. Tom are you getting serious on us?
Just kidding. Nice article.

yeah, I should have signed it OPRA!

 

[REWJR]

Subjective and Objective speaker testing measurement

The God father of the double blind testing was DR. Floyd toole (NRC research fame now with Harmon International ) who wrote the definitive white paper on this subject.

Dr. Floyd Toole is why! Who's he, you ask?

(excerpt from Stereo Review SEPT. 1998 pg. 112 by Corey Greenberg "The High End" article - actually this whole discussion is based on Corey's article)
"The ripples of Dr. Floyd Toole's groundbreaking work on lab-controlled listening test at Ottawa's National Research Council in the 1980s can be found today in the rapid dominance of such NRC-inspired Canadian speaker lines as Energy, Paradigm, and PSB."
(end of excerpt)

Dr. Toole joined Harman International's new state-of-the-art speaker laboratory in Northridge, California back in 1991, and he's since brought aboard many of his ex-NRC assistants such as Sean Olive and Allan Devantier. These imported Canadian's best loudspeaker minds were given the job to jump-start JBL.

LAB-CONTROLLED LISTENING TEST
In launching the new Harman Consumer Group Acoustical Engineering Lab, Dr. Toole has built an NRC-inspired "home away from home" where the Harmanized Canucks can continue to practice their special blend of subjective- and objective-based speaker design. That's the genius of the NRC and now Harman's Listening Lab**: by using not just measurements but also rigorously controlled listening tests with both audiophiles and civilian listeners, engineers can better correlate measured performance with subjective sound quality and push their designs in directions that listeners repeatedly prefer.

SUCESSFUL?
(excerpt from Corey's Stereo Review article, listed above)
"To call Harman's Listening Lab the best speaker-evaluation setup I've ever heard is selling it short. ...

http://caa-aca.ca/PEIWEBpage/PEI_Toole.htm

 

[mtrycrafts]

The God father of the double blind testing was DR. Floyd toole (NRC research fame now with Harmon International ) who wrote the definitive white paper on this subject. ]

He wrote more than a white paper on the subject and more. He has a library of JAES, peer reviewed papers published including the one on DBT testing of speakers.

Remember, some white papers by others are full of pseudoscience.

 

[REWJR]

I am aware of his work as he is a personal friend of the family since the NRC days and I always manage to share time with him and his HK Canadian right hand men Sean and Allen at CEDIA...where I usually slip into one of his lectures ...

 

[mtrycrafts]

I am aware of his work as he is a personal friend of the family since the NRC days and I always manage to share time with him and his HK Canadian right hand men Sean and Allen at CEDIA...where I usually slip into one of his lectures ...

Thanks for your response and the link. Nice.

 

[acurtas]

Very well written.

I knew a lot of that, but never saw it all laid out that clearly. Next time I want to demo a new amp to my friends, I'll make them read that beforehand (and before I blindfold them twice )

A

 

[pbarach1]

An excellent introduction to testing methodology

You basically discussed several weeks' worth of the basic principles that are covered (less clearly most of the time!) in a basic course on experimental methodology. I would quibble, though, with the idea that SINGLE blind tests are as valid as double-blind tests in evaluating audio equipment. Single blind tests introduce a source of unintentional bias, namely the experimenter/researcher's. He or she may unknowingly skew the listener's responses by means of exploiting primacy or recency effects (i.e., the piece of equipment you listened to first or last may tend to sound better), by nonverbal cues that neither the listener or the researcher is aware of, etc. A double-blind (or ABX) method will eliminate these suggestibility effects.

Also, the editorial was careful to point out that researchers always list the LIMITATIONS of their findings. This is all too often not done in the comaprisons of audio equipment. For example, if people can't reliably hear the differences between a $100 amp and a $10K amp, has anyone checked to make sure that the listeners have good enough hearing to pick up a difference? This factor, along with listener fatigue and the choice of source material, is a source of bias that audio writers have never controlled for. At least they ought to be mentioning it as a limiting factor in their conclusions.

On the other hand, if listeners DO hear a difference, was the equipment set up properly to make sure that the only differentiating factor between A and B was the amp and not the cabling or volume levels? Responsible audio writers are much better at getting everything technically right technically to eliminate this kind of bias, while ignoring factors in their listeners that may make the results suspect.

 

[Tom Andry]

I would quibble, though, with the idea that SINGLE blind tests are as valid as double-blind tests in evaluating audio equipment. Single blind tests introduce a source of unintentional bias, namely the experimenter/researcher's. He or she may unknowingly skew the listener's responses by means of exploiting primacy or recency effects (i.e., the piece of equipment you listened to first or last may tend to sound better), by nonverbal cues that neither the listener or the researcher is aware of, etc. A double-blind (or ABX) method will eliminate these suggestibility effects.

There are plenty of ways to control for everything you mentioned. A good test - single or double blind, or whatever - elegantly designed will control for most everything that can be anticipated. The problem I have with a lot of the research about expectancy effects is that much of it is contradictory. Unless things have changed significantly in the last few years, the general consensus is that expectancy bias is something you should think about and mention in the limitations section of your paper but you can't lose too much sleep over it. Sure, you try to control for it but you are usually limited in how much you can do. Since I had my Research Methods professor proof this paper for me, I'm guessing that he agrees.

People that harp on double blind as THE test are generally just being adversarial. No one does double blinds for everything (outside of the pharmaceutical field). It just isn't practical (in most educational and some medical research, it isn't even ethical). Double blind tests are good at ferreting out small or subtle effects. Effects that people might imagine or that are easily swayed by many of the sources of bias described in the article. Most the people that claim to hear differences in audio say that they are massive. Throwing a sheet over your equipment and having someone switch them without you seeing is generally enough to refute huge effects. In this case, a single blind test would be more than enough.

So, while you are technically right that double blind testing controls bias very well, I contend that other testing methodologies can be just as effective in showing difference, controlling bias, and, most importantly, doesn't take a grant from the federal government to afford.

 

[mtrycrafts]

No one does double blinds for everything (outside of the pharmaceutical field). .

Not sure if this excludes the DBT based research in audio at the Canadian NRC or at Harman, or Paradigm, Mirage, or PSB, to name a few.

Computer controlled speaker turntables, score keeping, etc.