Setting up a Speaker Shootout or Component Comparison the Right Way

A

admin

Audioholics Robot
Staff member
In preparation for the forthcoming speaker shootout, we thought it'd be a good idea to explore the correct way to set one up. One of the most popular articles we ever write on Audioholics is speaker or cable shootouts. Shootouts are really just a comparison of two or more products. This sounds like a fairly straightforward process where you place two competing products in the same room and take a listen/look. But the reality is that it is much more complicated.


Discuss "Setting up a Speaker Shootout or Component Comparison the Right Way" here. Read the article.
 
highfigh

highfigh

Audioholic Overlord
Is the test designed to compare each speaker in its recommended position? I know some sound best against a wall, fairly far away and somewhere in-between. Is the room treated?
 
Last edited:
Djizasse

Djizasse

Senior Audioholic
Nice, but I was really hoping for a more in depth guide. Where's the Part II? :D
 
L

Loren42

Audioholic
Testing

Everything you do is worthless unless you use a double blind testing format.

By that I mean both the test subject that is judging the comparison and the person conducting the test are absolutely unaware of which item is being tested when. Results are tallied by an independent researcher.

Then you need to repeat the test enough times to be sure any results are statistically relevant.

The problem with the above method is that it will rapidly show, for instance, that speaker cable is meaningless. The blind test that used Monster Cable and coat hangers is a great example of this. Once you go on to prove that coat hanger sounds the same as expensive speaker wire your advertising revenue will drop to zero.

So my bet is that you won't do a proper test.
 
gene

gene

Audioholics Master Chief
Administrator
--------------------------------------------------------------------------------

Everything you do is worthless unless you use a double blind testing format.

By that I mean both the test subject that is judging the comparison and the person conducting the test are absolutely unaware of which item is being tested when. Results are tallied by an independent researcher.

Then you need to repeat the test enough times to be sure any results are statistically relevant.

The problem with the above method is that it will rapidly show, for instance, that speaker cable is meaningless. The blind test that used Monster Cable and coat hangers is a great example of this. Once you go on to prove that coat hanger sounds the same as expensive speaker wire your advertising revenue will drop to zero.

So my bet is that you won't do a proper test.
Um I strongly disagree that it has to be DBT. If the listeners are unaware of the speakers they are listening to they can still make relatively excellent unbiased judgements. I often do quick comparisons of speaker pairs when doing reviews without any DBT or ABT testing and its easy to hear clear differences between speakers as long as they are properly level matched and positions (ie. L1L1 R1R2).

Cables are another story b/c the sonic differences are so subtle and often non existant, something we always stress. Speakers often sound drastically different which makes it much easier to setup a fairly accurate test without doing a true DBT.
 
majorloser

majorloser

Moderator
"In preparation for the forthcoming speaker shootout...

Okay, I'll bite. What ya' got planned? :rolleyes:
 
R-Carpenter

R-Carpenter

Audioholic
Speakers are more difficult to compare due to on and off axis response (crossover, driver differences and combination) and room interactions. You really need to place compared speakers in the same position for a DBT. Inexpensive SPL meter and some pink noise will make level matching easy enough. Large screen with acoustic fabric will make it truly double blind.
Sometimes a A-B speaker switch box cam make even easier job.
As far as Blind or not blind, I disagree. Nice looking speakers usually admired first for the looks and automatically regarded as good sounding.
Perhaps a professional reviewer has a less biased view but your average customer will buy with their eyes.
 
3db

3db

Audioholic Overlord
Speakers are more difficult to compare due to on and off axis response (crossover, driver differences and combination) and room interactions. You really need to place compared speakers in the same position for a DBT. Inexpensive SPL meter and some pink noise will make level matching easy enough. Large screen with acoustic fabric will make it truly double blind.
Sometimes a A-B speaker switch box cam make even easier job.
As far as Blind or not blind, I disagree. Nice looking speakers usually admired first for the looks and automatically regarded as good sounding.
Perhaps a professional reviewer has a less biased view but your average customer will buy with their eyes.
I tend to agree with that assessment. Most people will buy an aesthetically prettier speaker whose sound is marginaly worse than the ugly duckling beside whose sound is marginaly better.
 
B

BoB/335

Junior Audioholic
I tend to agree with that assessment. Most people will buy an aesthetically prettier speaker whose sound is marginaly worse than the ugly duckling beside whose sound is marginaly better.
I agree! The Song Towers I listened to side by side with M80's were much better looking so that's what I'm going with.
 
Hipnotic4

Hipnotic4

Full Audioholic
When testing speakers, I always found it was better to test with not just familiar material, but new material as well...the new material helps my ears pay more attention to what sounds or should sound "right"...

Looking foward to the shootout though!
 
ozmedia

ozmedia

Audioholic
I agree! The Song Towers I listened to side by side with M80's were much better looking so that's what I'm going with.
having sold high end speakers (for many manufacturer's) for over 20 years now, I always use the exact same comment.

"Bottom line, do you look at your music or listen to it?" (though either way I'd still sell them RBH, great looks and phenomenal sound, best bang for the buck) :D
 
lsiberian

lsiberian

Audioholic Overlord
Everything you do is worthless unless you use a double blind testing format.

By that I mean both the test subject that is judging the comparison and the person conducting the test are absolutely unaware of which item is being tested when. Results are tallied by an independent researcher.

Then you need to repeat the test enough times to be sure any results are statistically relevant.

The problem with the above method is that it will rapidly show, for instance, that speaker cable is meaningless. The blind test that used Monster Cable and coat hangers is a great example of this. Once you go on to prove that coat hanger sounds the same as expensive speaker wire your advertising revenue will drop to zero.

So my bet is that you won't do a proper test.
This is also a poor way to test speakers. We all have different listening preferences and many speakers take adjustment periods. I think FR is one of the best pure ways to test a speaker system. But if the speaker is ugly it shouldn't be considered.
 
highfigh

highfigh

Audioholic Overlord
This is also a poor way to test speakers. We all have different listening preferences and many speakers take adjustment periods. I think FR is one of the best pure ways to test a speaker system. But if the speaker is ugly it shouldn't be considered.
What if the ugly one sounds best?

IMO, the listener is the only one who needs to be unaware of which speaker is playing, and when. As long as the person conducting the test doesn't communicate with the listener in any way, the test is fair. For that matter, the tester can be in another room when they switch the speaker connections.

Sounds like we all need to go on a road trip.
 
no. 5

no. 5

Audioholic Field Marshall
Perhaps a professional reviewer has a less biased...
In an experiment performed at Harman International that concerned that very subject, Harman employees were asked to rate the subjective sound quality of Harman products and a loudspeaker from a competitor; in a blind comparison the competitor's product came received the lowest score of the four products, in a sighted comparison, the competitor came in third.
 
L

Loren42

Audioholic
What if the ugly one sounds best?

IMO, the listener is the only one who needs to be unaware of which speaker is playing, and when. As long as the person conducting the test doesn't communicate with the listener in any way, the test is fair. For that matter, the tester can be in another room when they switch the speaker connections.

Sounds like we all need to go on a road trip.
A single blind test is not a bad test, but a double blind test reduces the potential for bias even more. That being said, I would be happy with a blind test from the listener compared to having the listener fully cognitive of which speaker was being played at any given time.

There is bountiful and well grounded data on how tests should be conducted and why.
 
highfigh

highfigh

Audioholic Overlord
A single blind test is not a bad test, but a double blind test reduces the potential for bias even more. That being said, I would be happy with a blind test from the listener compared to having the listener fully cognitive of which speaker was being played at any given time.

There is bountiful and well grounded data on how tests should be conducted and why.
I understand the concept but if the person conducting the test can't or won't communicate with the listener, it would be basically the same result. However, for the test to be absolutely spotlessly valid, double blind is the preferred method.

I wouldn't mind if the tests were performed in the dark.
 
gene

gene

Audioholics Master Chief
Administrator
having sold high end speakers (for many manufacturer's) for over 20 years now, I always use the exact same comment.

"Bottom line, do you look at your music or listen to it?" (though either way I'd still sell them RBH, great looks and phenomenal sound, best bang for the buck)
That's funny b/c usually when I conduct face offs with RBH speakers in the mix, they are typically the ugliest but best sounding speakers :D
 
mtrycrafts

mtrycrafts

Audioholic Slumlord
Speakers are more difficult to compare due to on and off axis response (crossover, driver differences and combination) and room interactions. You really need to place compared speakers in the same position for a DBT.
In research labs like the one up in Canada, NRC and at Harman, speakers on a turntable to assure same location placement and swap speakers in less than 2 seconds, silently, so that acoustic memory is not lost.

Inexpensive SPL meter and some pink noise will make level matching easy enough.
Can an spl meter do .1 dB level matching capability? Don't think so. Need to use a volt meter.


..Large screen with acoustic fabric will make it truly double blind.
Only if the person swapping the speakers is not ever visible to the listener.


As far as Blind or not blind, I disagree. Nice looking speakers usually admired first for the looks and automatically regarded as good sounding.

Agreed. :D Toole has aptly demonstrated this. :D
Perhaps a professional reviewer has a less biased view but your average customer will buy with their eyes.
Don't think even they have immunity from it. So, all depends on what one is after from such a comparison.
 
T

theranman

Audiophyte
I've conducted a few comparo's myself, and to be honest, the whole single-blind, double-blind stuff is totally unnecessary. When a test is set up properly, you'll find that speakers differ SO MUCH that you'll laugh at the notion of DBT's. Sure, you'll pat yourself on the back for going thru the monumental extra effort to do a DBT, but the results likely won't change. Sure, once in a while you find yourself comparing two very similar sounding speakers, but in my experience, most sound drastically different thruout the midrange. Oh...one more thing. I use an 80hz high pass filter thruout the tests because newbies are so easily swayed by louder and fake bass. :D

My method:

1) Most important: Buy an inexpensive A/V switcher with remote from Ebay. The only cost $25 or so, and let you sit back in the listening chair and instantly switch back and forth. Instantly! None of that disconnecting, setting up, taking down of speaker BS. Your auditory memory isn't that good.

2) Radio shack meter for sound level matching. Yes, it's good enough.

3) Two pair of speaker stands if you're testing bookshelves.

4) Set them up right next to each other AB.....AB style.

5) Two amps with adjustable gain so that you can level match the two.

6) Single source...preferable CD player.

Go to town and be amazed how quickly and easily you can tell differences.
Also fun to do is to run the response thru a laptop + microphone analyser program. Then you'll see the typical monster dip in the midrange that I hate. :D

Ever since I did a few of these comparo's, I decided that I vastly preferred the sound of flat response pro studio monitors...and active ones at that. Different strokes, eh? Also, I don't think it's unreasonable to pit expensive monitors against cheap ones. You'll often find that their respective values don't relate to their price tags as far as sonics is concerned.

Well, that's MY 2¢
Take it with a truckload of salt. ;)
 

newsletter
  • RBHsound.com
  • BlueJeansCable.com
  • SVS Sound Subwoofers
  • Experience the Martin Logan Montis
Top