Setting up a Speaker Shootout or Component Comparison the Right Way

[gene]

I've conducted a few comparo's myself, and to be honest, the whole single-blind, double-blind stuff is totally unnecessary. When a test is set up properly, you'll find that speakers differ SO MUCH that you'll laugh at the notion of DBT's. Sure, you'll pat yourself on the back for going thru the monumental extra effort to do a DBT, but the results likely won't change. Sure, once in a while you find yourself comparing two very similar sounding speakers, but in my experience, most sound drastically different thruout the midrange. Oh...one more thing. I use an 80hz high pass filter thruout the tests because newbies are so easily swayed by louder and fake bass.

My method:

1) Most important: Buy an inexpensive A/V switcher with remote from Ebay. The only cost $25 or so, and let you sit back in the listening chair and instantly switch back and forth. Instantly! None of that disconnecting, setting up, taking down of speaker BS. Your auditory memory isn't that good.

2) Radio shack meter for sound level matching. Yes, it's good enough.

3) Two pair of speaker stands if you're testing bookshelves.

4) Set them up right next to each other AB.....AB style.

5) Two amps with adjustable gain so that you can level match the two.

6) Single source...preferable CD player.

Go to town and be amazed how quickly and easily you can tell differences.
Also fun to do is to run the response thru a laptop + microphone analyser program. Then you'll see the typical monster dip in the midrange that I hate.

Ever since I did a few of these comparo's, I decided that I vastly preferred the sound of flat response pro studio monitors...and active ones at that. Different strokes, eh? Also, I don't think it's unreasonable to pit expensive monitors against cheap ones. You'll often find that their respective values don't relate to their price tags as far as sonics is concerned.

Well, that's MY 2¢
Take it with a truckload of salt.

This is an extremely refreshing post that is right on the money!

 

[R-Carpenter]

In research labs like the one up in Canada, NRC and at Harman, speakers on a turntable to assure same location placement and swap speakers in less than 2 seconds, silently, so that acoustic memory is not lost.[/QUOTE]


1)That's a good idea but it bring up a question of how different size speakers will interact and affect each others sound in free space. Baffle step compensation is designed differently by different manufacturers.
I'd go with turntable. It's not hard to manufacture. Basically it's a large Lazy Susan with carpeted base.

Can an spl meter do .1 dB level matching capability? Don't think so. Need to use a volt meter.[/QUOTE]


2)Ehh, yes if we want to be perfectly precise. How do we account for differences in off axis FR between different speakers? It will affect SPL matching unless the reviewer is in anechoic chamber.
Personally, I'd go with pink noise within 0.5db. Differences in FR between the speakers are very obvious in comparison to amps, CD players and other electronics.




Don't think even they have immunity from it. So, all depends on what one is after from such a comparison.[/QUOTE]

3)I was being a bit sarcastic about it.



Also, I would advise using a different amp for a review. Impedance matching speaker selector such as the one sold by Parts Express would be a better choice.

On another note, I think a reviewer should use real MLS measurements and not a Micky Mouse analyzer.

 

[avaserfi]

This is an extremely refreshing post that is right on the money!

I made this post in another thread, but considering the disucssion here I feel it is worthy of being copied:

Certainly, auditioning is fun, but it has been shown time and time again that sighted listening tests are biased by a variety of factors. I suggest reading the following literature to illustrate the exact effects of bias in a sighted listening test.

Toole, Floyd E. Subjective Evaluation: Identifying and Controlling the Variables. J. Audio Engineering Soc., Conference Paper 8-013. May 1990.

Toole, Floyd E.; Olive, Sean E. Hearing is Believing vs. Believing is Hearing: Blind vs. Sighted Listening Tests, and Other Interesting Things. J. Audio Engineering Soc., Conference Paper 3894. November 1994.

 

[R-Carpenter]

Oh...one more thing. I use an 80hz high pass filter thruout the tests because newbies are so easily swayed by louder and fake bass.


Go to town and be amazed how quickly and easily you can tell differences.
Also fun to do is to run the response thru a laptop + microphone analyser program. Then you'll see the typical monster dip in the midrange that I hate.



Yes, but aren't you the one who's going to do a review? And what do you mean by fake bass? Isn't lower region of FR of a speaker a part of it's character? I am just asking a question, what's the purpose of this 80hz filter?

I am sorry but what do you mean by “typical” deep in the midrange? Do all speakers suffer from it?
Are you MLSing speakers or just reading room cancellation?

 

[majorloser]

I made this post in another thread, but considering the disucssion here I feel it is worthy of being copied:

They might be the best sounding speakers in the world.
But if their ugly, they won't be going in my home.

So yeah, there is a bias in sighted tests. But we are talking about comparisons, not strictly testing for performance.

Besides, if you do it right there will be a non-voting person controlling the a/b switch and not telling you which pair your listening to. The people listening just need to sit back, relax, close their eyes and listen. Once your ears pick out a particular pair you like, it will become very obvious when that pair is playing. It worked out quite well at my house last year when Gene ran the show. We came up with the top two bookshelfs being the Status Acoustic Decimos and a pair of AV123 x-ls speakers. Definitely not in the same ballpark with appearance and price range. And for what it's worth, the worst performers were the SVS bookshelves.

Unfortunately the Decimos sounded so good I had to buy a pair. And at this point, I don't want to find anything that sounds better. My wallet couldn't stand it and wife would kill me.

So what's the lesson to learn from this?

DON'T KEEP EXPENSIVE GEAR THAT YOU DON'T OWN IN YOUR HOME FOR VERY LONG.
YOU MIGHT END UP WANTING TO BUY IT.

 

[mtrycrafts]

But what about the differences in sensitivity and FR between speakers, is it possible to get multiple speakers within 0.1dB of each other?

You cannot and don't think the labs EQ the speakers so the levels are matched, but you do need to level match and perhaps, with speakers, you do need to use a very sensitive spl meter that is capable of such level.
Come to think of it, the same voltage may not be enough as when the speakers are the constant and the components are swapped; then, it is imperative to use a volt meter as the speaker will output the same level with the same input voltage.

 

[mtrycrafts]

I've conducted a few comparo's myself, and to be honest, the whole single-blind, double-blind stuff is totally unnecessary. When a test is set up properly, you'll find that speakers differ SO MUCH that you'll laugh at the notion of DBT's. ...

Go to town and be amazed how quickly and easily you can tell differences.
....

The issue is not that there are still sonic difference between speakers but where does one's bias will take them; the one that visually impresses and affects the sensory inputs or, in fact the one that one really prefers due only to the sound and nothing else.

If DBT was not necessary, even with speakers, research labs would not use them. After all, that is what they used to find out what people of all backgrounds prefer, flat frequency response, wide dispersion, etc. And demonstrated that without it, the other sensory inputs do a number on preferences.

 

[mtrycrafts]

They might be the best sounding speakers in the world.
But if their ugly, they won't be going in my home.

So yeah, there is a bias in sighted tests. But we are talking about comparisons, not strictly testing for performance.

Besides, if you do it right there will be a non-voting person controlling the a/b switch and not telling you which pair your listening to. .

Yes, but, if you can see the speakers I am sure you will have a good probability of knowing which one is on

The visual impact should play a role, of course, but in my estimation after your sonic experimentation and see which has more weight for you.

Even with that switcher person not telling you, it still matters and can affect the bias responses. That is why DBT is used when it matters.
In this case though...

 

[mtrycrafts]

1)That's a good idea but it bring up a question of how different size speakers will interact and affect each others sound in free space. Baffle step compensation is designed differently by different manufacturers.
I'd go with turntable. It's not hard to manufacture. Basically it's a large Lazy Susan with carpeted base.

Apparently the turntable method at the labs are enough not to have interference between the speakers when they are 180 behind in the off mode.

2)Ehh, yes if we want to be perfectly precise. How do we account for differences in off axis FR between different speakers? It will affect SPL matching unless the reviewer is in anechoic chamber.
Personally, I'd go with pink noise within 0.5db. Differences in FR between the speakers are very obvious in comparison to amps, CD players and other electronics.

The point of level matching is not to make it harder to differentiate, it is to eliminate a volume difference that the listener may misconstrue as being better. You don't EQ the hole frequency band, perhaps at one frequency, like 1kHz. And the other parameters are really what will tell one speaker apart from the other and stand it out, that is the whole point, which is the better speaker.

3)I was being a bit sarcastic about it.

missed it

On another note, I think a reviewer should use real MLS measurements and not a Micky Mouse analyzer.

What's wrong with Mickey Mouse?