Revealing Flaws in the Loudspeaker Demo & Double Blind Test

A

admin

Audioholics Robot
Staff member
What is the point of a listening test? For the listener, the intent is easy to identify. They are (often) thinking of buying a product. But when a short demo isn't enough, what options do you have? The obvious solution is third party validation. Manufacturers will often try to sell you the line that their speakers are as good as speakers many times their price. They'll even run their own tests (often Single or Double Blind) to prove that the participants couldn't hear a difference between their and another speaker. But what are they really saying? We break it down for you and show how these claims prey on people's lack of statistical understanding.


Discuss "Revealing Flaws in the Loudspeaker Demo & Double Blind Test" here. Read the article.
 
MinusTheBear

MinusTheBear

Audioholic Ninja
Great article. Another BIG flaw/drawback of controlled tests or listening to speakers in showrooms is the amount of listening time is limited and might not be accurate of what you might experience with a speaker over long listening sessions. Some listening tests will give you an infinite amount of time, but if the panel is large and they are trying to do it in one day, the listener will not have enough time. You have to be careful for speakers that sound good on first listen but overtime are fatiguing. Some speakers are voiced specifically like this

If you are buying retail or over the internet the best thing is a 30 day audition in your own home where you can listen to a speaker extensively and over a long period of time and not just for 10 mins or up to a few hours - controlled or not.
 
cwall99

cwall99

Full Audioholic
While Tom's spot on in his analysis, there are a number of factors that argue in favor of using some form a single-blind test when auditioning new speakers.

Let's say I go into a store that maybe carries 2 or 3 lines of speakers that have speakers in my price range. Just for kicks, let's just say these two brands are B&W and Klipsch (OK, these are two brands at a store that's local to me).

Now, first, I'm going to hope that the store has a way to compensate for the efficiency of the Klipsch (because, if they don't, it'll quickly be obvious which speaker is Klipsch and which is the B&W: the Klipsch'll be a lot louder).

Let's presume they have this, then what I'd like to be able to do is switch back and forth between speakers without know which one is playing. I don't want to know which brand is playing because all my expectations about that brand will color my judgement. So, a single-blind test, if set up correctly, will prevent me from knowing which speaker is playing, and, therefore, I like to think I'll actually decide based on how the speaker sounds rather than on my expectations about how I think the speaker ought to sound.

Now, when you switch to other things, say, like $15,000 speaker wires, you better take the time to set up a DBT for me. That's a lot of cash to drop on something that, IMHO, will have, at the very best, only a minimal impact on the sound quality.

Um, for the record, I don't have $15,000 speaker wires. More like $15 speaker wires.
 
AcuDefTechGuy

AcuDefTechGuy

Audioholic Jedi
Level-match the speakers with a SPL meter.

That's the key.

Yeah, DBT in drug trials is not the same as speakers.

FYI, these days, we don't compare drugs to PLACEBO anymore since it is unethical. We compare "new" drugs to drugs that are proven. Basically we want to know if the "new" drug can be as effective or more effective and has the same or less adverse effects (and costs less :D).

And in a DTB, the Pharmacist knows exactly which drug is which since we have to prepare them. The person administering the drug (Nurse) and person receiving the drug (patient) do not know which is which.
 
Last edited:
G

goldstarsteve

Audiophyte
Double blind testing is not for speakers

Double blind testing in the format known as ABX is a fine tool but not for speakers. As you put it in your article;

"The fact is that Double Blind Testing (DBT) really has very little place in the audio world - regardless of the claims and beliefs of manufacturers and some forum members. Double Blind Testing is designed to eek out small differences between very similar stimuli."

That is true for speakers which are very different from each other in sound. Everyone agrees that different speakers sound, well different, so a single blind test to compare speakers is spot on.

It is not so true for amplifiers, dacs, or for the real vodoo merchants wires. As these components are similer to one another you need an ABX test (see if user can hear a difference when he doesn't know which component he is listening to and allow him to switch between them instantly as audio memory is so short). If the user can hear a difference, any difference, then you can go ahead now and single blind test. If you dont ABX first then you may just be single blind testing some guesses....
 
T

the_phew

Enthusiast
On the internet direct vs. traditional speaker retailer debate:

It's true that larger companies that sell through dealer networks can benefit from economies of scale, and that an ID company doesn't necessarily have lower costs. However, I've noticed that the large companies seem to be more susceptible to commodity price fluctuations, currency exchange rates, gas prices, etc.

Take Paradigm for example. They have always been considered a good "value" brand among the major speaker manufacturers. However, the retail price of many speakers they sell has increased nearly 50% in the past five years or so (I'm told due to the strength of the Canadian dollar vs. the USD). It's hard to stomach knowing that they are charging such a huge premium over the initial price, for the same product. I'm not aware of any ID speaker companies that have raised their prices so much in as short a time.
 
gene

gene

Audioholics Master Chief
Administrator
There is nothing wrong with doing SBTs for audio tests. We do them often when comparing products. The problem is when manufacturers do them with their own staff and mistakenly call them DBTs. Some of the very same manufacturers won't submit products to 3rd party shootouts even if they follow a blind protocol.

Familiarity bias is a sure way their staff will pick their own speakers as the winner. Using the argument "similarly good" is a brilliant marketing strategy to ensure you never lose your own blind test even against more expensive and better engineered products.
 
Last edited:
AcuDefTechGuy

AcuDefTechGuy

Audioholic Jedi
I don't ask for much.

The 3 things I mainly want to know from any speaker review or shootout are

1) honest speaker measurements [on & off-axis response, anechoic or pseudo],


2) whether the speaker sounds extremely clear and detailed, instead of sounding compressed, DSP'ed, or equalized,

and

3) that the review is done by 3rd party, not by Axiom or any manufacturer. :D
 
Last edited:
lsiberian

lsiberian

Audioholic Overlord
Definitely a soap box for Tom.;)

Why double blind speakers at all? If you care at all about looks then you want the speaker to impress and instill confidence. It's not hard to be unbiased enough to make reasonable choices. I come in with my cash and I want the best speaker for my money. I listen to them all painstakingly until I have a winner.
 
gene

gene

Audioholics Master Chief
Administrator
Definitely a soap box for Tom.

Why double blind speakers at all? If you care at all about looks then you want the speaker to impress and instill confidence. It's not hard to be unbiased enough to make reasonable choices. I come in with my cash and I want the best speaker for my money. I listen to them all painstakingly until I have a winner.
To me it implies the manufacturer considers their speaker to be uglier than the competition if the listener doest pick their speaker as the winner in a sighted test. That doesnt instill much confidence in ones product IMO.
 
3db

3db

Audioholic Slumlord
I can understand why a DBT test done on speakers is not really helpful, espically when looks and the Spousal Acceptance Factor (SAF) comes into play. I can see the reason for a SBT espicially if one has narrowed down his choices to several models and really is having a hard time to pick out the difference (assuming room placement and looks are on a equal footing) .

Thanks for the article. :)
 
krabapple

krabapple

Banned
Here's a thing: DBTs aren't inherently flawed. They do require good design and execution. With that, they can be used both for testing 'difference' AND for testing 'preference'. The protocols will be different (e.g, ABX versus ABC/HR), but the all-important 'double blindedness' remains. The 'answers' they give are 'likelihoods' rather than absolutes.

Sighted 'testing' of sound *is* inherently flawed. There are plenty of books out now about how the mind plays 'tricks' on us, and how we over-estimate the reliability of our own perception and judgement. The list of psychological biases and confounders we are subject to, is long. Even being a trained listener doesn't get you a free pass, it just improves your odds. At bottom the question is how can we know whether a subjective report of difference (absent any objective measurements) is likely to be true beyond a reasonable doubt? Recognition of this is why blind testing exists.

If nothing else, it could be interesting to read a counterpoint to this article from someone who has conducted multiple DBTs of loudspeakers. I hear there's a guy at Harmon who might fit the bill...
 
AcuDefTechGuy

AcuDefTechGuy

Audioholic Jedi
Here's a thing: DBTs aren't inherently flawed. They do require good design and execution. With that, they can be used both for testing 'difference' AND for testing 'preference'. The protocols will be different (e.g, ABX versus ABC/HR), but the all-important 'double blindedness' remains. The 'answers' they give are 'likelihoods' rather than absolutes.

Sighted 'testing' of sound *is* inherently flawed. There are plenty of books out now about how the mind plays 'tricks' on us, and how we over-estimate the reliability of our own perception and judgement. The list of psychological biases and confounders we are subject to, is long. Even being a trained listener doesn't get you a free pass, it just improves your odds. At bottom the question is how can we know whether a subjective report of difference (absent any objective measurements) is likely to be true beyond a reasonable doubt? Recognition of this is why blind testing exists.

If nothing else, it could be interesting to read a counterpoint to this article from someone who has conducted multiple DBTs of loudspeakers. I hear there's a guy at Harmon who might fit the bill...
I agree.

But you know, everyone is an expert. :D

Everyone is a psychologist. Everyone is a doctor. Everyone is an engineer. :D

Everyone knows what's best.:D

And they don't need any stinking proofs. :eek:
 
P

PENG

Audioholic Slumlord
I agree.

But you know, everyone is an expert. :D

Everyone is a psychologist. Everyone is a doctor. Everyone is an engineer. :D

Everyone knows what's best.:D

And they don't need any stinking proofs. :eek:
No wonder in the US and Canada and some other countries, Dr, Engr, Psyc are licensed!!: Oops.. Pharmacists too..just to make sure participants in DBTs are not influenced by drugs.:D
 
gene

gene

Audioholics Master Chief
Administrator
The bottom line is you have to spend a good amount of time with a speaker to ultimately determine if you like how it sounds. A bright/bassy speaker will initially sound better than a neutral speaker in short sighted or blind tests. Try living with a bright speaker over a course of a few months. It's painful. I know this b/c years ago I was attracted to such a speaker and bought them. A year later I was the happiest camper to replace them with a more neutral and initially duller sounding speaker.
 
gtpsuper24

gtpsuper24

Full Audioholic
I couldn't wait to get rid of my bright, midrange ringing speakers after a year or so. Its amazing after you get a dull or truely neutral speaker how long you find yourself just sitting and listening to them, compared to listening to a song or two only to just turn them off. I was naive to think that bright means detailed, boy was I wrong:D
 
Adam

Adam

Audioholic Jedi
I think that my NHTs are neutral, but they always sounded bright to me until I got a receiver with auto calibration that helps take care of some of the room effects (that technology is perfect for me, as I probably won't ever do room treatments). I absolutely hated listening to most music on them until I got my Pioneer with MCACC. Now, it's enjoyable again. :)
 
cpp

cpp

Audioholic Ninja
The bottom line
the real bottom line well regardless of how the speaker sounds in a store, or in a stores treated sound room nothing will determine the real bottom line until that speaker is in your room, using your electronics then you either did good, or you screwed up. That's the bottom line
 
DenPureSound

DenPureSound

Senior Audioholic
The bottom line is you have to spend a good amount of time with a speaker to ultimately determine if you like how it sounds. A bright/bassy speaker will initially sound better than a neutral speaker in short sighted or blind tests. Try living with a bright speaker over a course of a few months. It's painful. I know this b/c years ago I was attracted to such a speaker and bought them. A year later I was the happiest camper to replace them with a more neutral and initially duller sounding speaker.
Gene, you left out the most important piece of the puzzle, was this for not biasing anyone towards the ones you bought or ?

So what speaker did you buy in the end w/ a more neutral and initially duller sound?
 
DenPureSound

DenPureSound

Senior Audioholic
No wonder in the US and Canada and some other countries, Dr, Engr, Psyc are licensed!!: Oops.. Pharmacists too..just to make sure participants in DBTs are not influenced by drugs.:D
Just a fact, not ALL Engineers are licensed in the U.S.A. :)
 

Latest posts

newsletter

  • RBHsound.com
  • BlueJeansCable.com
  • SVS Sound Subwoofers
  • Experience the Martin Logan Montis
Top