How to Skew a Blind Listening Test

A

admin

Audioholics Robot
Staff member
Setting up a blind test is easy right? Or is it? While most people will take the most well known precautions, they often forget or don't know about some of the others than can seriously affect the outcome of the test. Here are some lesser known things to remember when you set up your listening test.


Discuss "How to Skew a Blind Listening Test" here. Read the article.
 
STRONGBADF1

STRONGBADF1

Audioholic Spartan
Good article, Tom. I would also like to see someone do research to see if certain sound signatures do better in quick tests. I would think speakers with more bass energy would fair better in quick tests and that might be tiring to some after living with it.
 
F

fredk

Audioholic General
Excellent article. Even if you don't do proper blind listening when selecting speakers, I think its helpful to understand how you can become biased
 
J

jostenmeat

Audioholic Spartan
I saw someone recently quote Dennis Erskine, on a "skewed", and arguably "malicious" test. Not blind, but it still has a lot of relevance to the second paragraph, Seeing is Believing: "Obviously, the most harmful thing for a participant to have in a listening test is preconceptions."

Dennis Erskine said:
There are two camps and always will be ... real or imagined. I tested this early on with a recording that (1) was recorded specifically for multi-channel; and, (2) both the DD and DTS version were from the same master, not remixed, levels not changed, etc.

I set the surround processor to display DTS when DD was playing and it displayed DD when DTS was playing. After the listening session 100% of those professing DTS was superior, still agreed DTS was superior. 100% of those who thought DD was superior still were in the DD camp at the end of the session.

Oh well.
:p
 

Brewed

Audiophyte
Is it not ludicrous to think that any "blind" test of speakers is tainted?

Surely only one pair of the test candidates can set up properly from an imaging point of view. And in this case, it would surely be Paradigm that dialed in their position and flopped the others where they may lay?

Tom, you know this. I've read your stuff.

Any speaker, regardless of price, has an optimal location in any given room. If you get a nicely centered vocal image but the sound is too bright, you've bought the wrong speaker for your taste. And vice versa.

And did I mention the sympathetic vibrations of the speakers not in use during a "blind" test?

My two cents......
 
gene

gene

Audioholics Master Chief
Administrator
I also firmly believe by placing multiple pairs of speakers side by side, you can cause excessive diffraction from adjacent baffles which could hamper the better speakers ability to image. I personally don't like to compare more than two pairs of speakers at a time for this reason and also b/c positioning relative to the listener becomes too drastically different.

In addition, some speakers require more/less toe-in to perform optimally. When placing speakers side by side, you typically nullify this by placing all of the speakers with no toe-in at all.

I think a combination of blind and sighted tests (blind first, sighted after) over a course of a few days is prudent in determining which speaker you really like better, assuming the pairs in question are close enough in sound quality. Music selection is also key as I've seen too many tests done with highly compressed rock music, thus diminishing the finer qualities of the better performing speaker.

One must also take in account to listening level. How linear is the speaker at low and high volume? How consistant does it sound over the SPL range you normally listen too?

If you really want to get scientific in determining which speaker is "better" you need a very large panel of listeners to get a good/accurate statistical sampling. As Tom points out, these listeners should be neutral people not working for the company whose products are under test.

Great article Tom!
 
gene

gene

Audioholics Master Chief
Administrator
A switcher could also be used which could degrade the performance of certain speakers (http://forums.audioholics.com/forums...ad.php?t=58393)

And not all speaker's ideal location are the same. Some might be better off closer to the rear wall, closer together, etc...
good point. Years ago when I went to Sound Advice to audition speakers, they had them fed thru a switcher. At the time they always seemed to promote B&W over any other brand they carried. When I listened between a pair of B&Ws and Kefs, I preferred the former. Then I asked to remove the switcher and feed the speakers directly to two identical amps, level matched. The listening results were much different then. The salesman was also shocked and claimed the switcher must be defective. After that experience, I was careful to examine all variables when auditioning under someone elses test conditions.
 
STRONGBADF1

STRONGBADF1

Audioholic Spartan
I also firmly believe by placing multiple pairs of speakers side by side, you can cause excessive diffraction from adjacent baffles which could hamper the better speakers ability to image. I personally don't like to compare more than two pairs of speakers at a time for this reason and also b/c positioning relative to the listener becomes too drastically different.

In addition, some speakers require more/less toe-in to perform optimally. When placing speakers side by side, you typically nullify this by placing all of the speakers with no toe-in at all.

I think a combination of blind and sighted tests (blind first, sighted after) over a course of a few days is prudent in determining which speaker you really like better, assuming the pairs in question are close enough in sound quality. Music selection is also key as I've seen too many tests done with highly compressed rock music, thus diminishing the finer qualities of the better performing speaker.

One must also take in account to listening level. How linear is the speaker at low and high volume? How consistant does it sound over the SPL range you normally listen too?

If you really want to get scientific in determining which speaker is "better" you need a very large panel of listeners to get a good/accurate statistical sampling. As Tom points out, these listeners should be neutral people not working for the company whose products are under test.

Great article Tom!
I believe the NRC (or maybe it was Harmon) uses a "turntable" to quickly rotate the speakers in and out. Still doesn't help with optimum positioning (unless its huge) but it's got to help some.
 
gene

gene

Audioholics Master Chief
Administrator
I believe the NRC (or maybe it was Harmon) uses a "turntable" to quickly rotate the speakers in and out. Still doesn't help with optimum positioning (unless its huge) but it's got to help some.
Yes I've seen that but I wonder what the switch out time was during that process.
 
STRONGBADF1

STRONGBADF1

Audioholic Spartan
Yes I've seen that but I wonder what the switch out time was during that process.
and does it make a noise of it's own? how would that affect the listening test???

...too much bother...

It seems to me that speakers with a very flat FR, room treatment and EQ is the answer to getting the sound signature the way you want it. (if your wife will let you.:))
 
Tom Andry

Tom Andry

Speaker of the House
I would also like to see someone do research to see if certain sound signatures do better in quick tests. I would think speakers with more bass energy would fair better in quick tests and that might be tiring to some after living with it.
I agree, but I also think those with more top end energy would do better. I think a laid back speaker that still has good extension is at a disadvantage in a quick test. To me, this is sister to the "louder is better" bias. In a quick test I'm guessing that, overall, brighter is better.
 
Tom Andry

Tom Andry

Speaker of the House
In addition, some speakers require more/less toe-in to perform optimally. When placing speakers side by side, you typically nullify this by placing all of the speakers with no toe-in at all.
That's true. In our test, I adjusted the toe-in for each speaker based on my previous review. That also dictated where I placed them in the room so that the more toed-in speakers wouldn't interfere with the toed-out.
 
STRONGBADF1

STRONGBADF1

Audioholic Spartan
I agree, but I also think those with more top end energy would do better. I think a laid back speaker that still has good extension is at a disadvantage in a quick test. To me, this is sister to the "louder is better" bias. In a quick test I'm guessing that, overall, brighter is better.

So, "sizzle and boom" beats "chocolaty" in a quick listening test...:confused:...:D
 
GO-NAD!

GO-NAD!

Audioholic Spartan
The article and the following commentary just hilight how difficult it is for any person to find the "right" speaker for them. Just some of the problems:

1) Most people don't have the time, or the inclination, to audition many speaker brands and models. I guess the more you're willing to spend, the more listening you should do.

2) Few stores have good acoustics for a good audition and pretty much none will have similar acoustics to your own room.

3) "Customer Reviews", as found on various websites are invariably positive - so what credence can you give them?

4) There aren't enough professional and blind comparisons amongst different speaker models. I know they're time-consuming and can cost a pretty penny to conduct, but I'd lend more weight to those than to any customer, including me!

5) When people ask for advice on buying speakers, I really don't like the standard response, "whatever sounds good to you". Why? Because short auditory memory, poor switching/level matching in-store and poor/different store acoustics still make the audition a crap shoot. That's why people are looking for educated and unbiased opinions.

That's also why I prefer the "shootout" over the individual speaker review. They're done in the same venue, at the same time, so at least there's some consistency in the listening conditions.

In addition, I'd like to see some standardization in the specifications, as listed by manufacturers, for speakers. There is so much room for fudging, that one has to take them with a grain of salt. Impedance ranges beyond the "nominal 8 ohms" would be helpful in determining the amplification requirements. FR should be measured under a set standard of conditions and at various angles. I know that proof is in the listening pudding, but with so many variables, we need a comprehensive list of objective, reliable measurements to aid in making decisions.

I like the concept of the ID manufacturers, but unless they are willing to really put their money where their mouth is and pay shipping both ways I'd be hesitant to bite. One or two may do that, but most don't.

That's why people look for guidance beyond "whatever sounds good to you". I fully understand the confusion amongst those who don't own quality speakers, want to get better, but are hesitant to walk through the minefield that is the speaker marketplace. Any other ideas out there for improving the testing methodology?
 
lsiberian

lsiberian

Audioholic Overlord
It think by far the best method for determining a speaker's quality is by using a FR graph plot from your listening position. Good luck with that though. :)
 
C

cynan

Enthusiast
I fully understand the confusion amongst those who don't own quality speakers, want to get better, but are hesitant to walk through the minefield that is the speaker marketplace. Any other ideas out there for improving the testing methodology?
An excellent summary of most of the major challenges of speaker shopping, GO-NAD (hee-hee). And yes, the "whatever sounds good to you" advice can be a bit simplistic. I think this advice can be problematic for someone that truly desires to invest in a worthy speaker upgrade, but has never really auditioned something in the range of "quality" in which they are looking. The upshot is that they may very well not be educated to know what really sounds the best to them. As a result, they may hear something that, on initial audition, seems pleasing, but which may, in reality, only be relatively more pleasing to other speakers they are auditioning at the time, with the source material they are auditioning with, etc. Furthermore, as most auditions are short, the problem of choosing a brighter sounding speaker (often appealing initially) that may become fatiguing after extended listening is a potential pitfall. Then there is the issue of the acoustics of the auditioning room vs the acoustics of the room where the new speakers will actually live...

Conversely, I think much of these pitfalls can be avoided with a bit of education and research and I do not think that acquiring that first pair (or set, in the case of surround sound) of decent speakers (upgrading from an HT-in-box or box store specials) has to be as harrowing an experience as your minefield metaphor depicts. Though it would be vastly preferable to be able to audition every speaker of interest in home for an extended period, I don't think this is a requirement in the execution of a successful upgrade from mid-fi to entry level hi-fi speakers. The main reason for this is market segmentation. Even among ID brands, there appears to be a consistent and discernible difference in quality in speakers which cost twice the price of others, more or less regardless of brand (which is particularly apparent for speaker pairs priced under $1000 due to the law of diminishing returns - a price point which is likely to be most popular with those looking to upgrade to something decent for the first time). This phenomenon has been plainly evident in speaker shootouts on this website.

With a bit of forethought into what one desires in a speaker, and research, one can have a fair degree of success in selecting speaker upgrades in the <$1000/pair range. A basic understanding of a few aspects of speaker design (IE, how soft dome tweeters generally sound smoother though less articulate than metal domes and whether someone's room is well damped to prevent undue reflections...), together with the weighing of feedback from various related online forums (staying away from forums on speaker manufacturer websites or "official"/owners threads on particular sites) and hopefully more than one professional review will more often than not produce favorable results. Any in-home auditions after that point is gravy (ie after narrowing down between 2 or 3 ID companies).

Are you guaranteed to get your absolute favorite speaker in the world in your price range? No - but you'll likely end up with something close. If not, you can always try again in a couple of years. (An advantage of going with something from a more popular company is the higher likelihood of decent resale value).

I think it would be much more challenging trying to upgrade from an already decent set of speakers to something more high end, due to reduced availability for auditioning, but mostly to the law of diminishing returns and the much more subtle (or even mystical ;) ) improvement of, say, $4000 speakers over a $2000 pair - though I admittedly have very little experience with speakers in this price range.

Granted, this point of view may be biased. Over the course of this summer (and most of the spring) I sought out my first decent set of speakers (<$1000), upgrading from a mishmash of cheap Polk Audio bookshelves (box store specials) and bottom of the line 8-year old Paradigms. Long story short, I received the speakers two weeks ago and have been very pleased so far - which, in the final analysis, is really all that matters.
 
P

PENG

Audioholic Slumlord
5) When people ask for advice on buying speakers, I really don't like the standard response, "whatever sounds good to you".
I am not sure if that is a standard response. If it is, then for sure I would give a non standard response. I typically would suggest one to first listen to as many live concerts as possible, especially classical and jazz concerts that do not use too much electronic gear. Instead of preferring, warm, bright, dark, analytical, organic, dynamic, lay back kind of sound I would just stick with "live" like kind of sound. That's why I prefer the sound of B&W 800 series speakers, but since I couldn't even afford the 803D, I settled for the lesser but still live like Energy speakers.

Back to the topic of blind tests, I think it is a non issue because unlike amplifiers, no matter how well you conduct the tests most people can speakers apart.
 
mtrycrafts

mtrycrafts

Seriously, I have no life.
Yes I've seen that but I wonder what the switch out time was during that process.
Those turntables are very quiet and very fast, not a manual switching rig. Their experiments showed that speaker positions, like speakers side by side, was a dead give away, and skewed the results.

Also, Harman and NRC under Toole, used a grading sheet of different areas. Swapped enough so the grades can be statistically analyzed and be meaningful.
 
mtrycrafts

mtrycrafts

Seriously, I have no life.
Back to the topic of blind tests, I think it is a non issue because unlike amplifiers, no matter how well you conduct the tests most people can speakers apart.
Yes, but at least bias is minimized and sound quality is maximized.
 

Latest posts

newsletter

  • RBHsound.com
  • BlueJeansCable.com
  • SVS Sound Subwoofers
  • Experience the Martin Logan Montis
Top