My attempt to compare an Outlaw 7500 and a Parasound A51

[Irvrobinson]

I wonder why respected names in various fields of acoustics and research facilities use blind listening testing to arrive at thresholds of detection, JND levels, etc, and publish peer reviewed journal papers on their research. To make a blanket statement that listening tests are flawed, blind or sighted to detect differences is total nonsense not based in facts.

Ah yes, the other DBT worshipper, mtycrafts. Just for starters, you need to improve your reading skills, or at least your comprehension skills when someone says your idol is flawed. What I said very specifically was that using listening tests, sighted or blind, to detect subtle differences between audio system components is flawed, because of the nature of hearing and auditory memory, and the fact that one is measuring human perception, not an observable phenomenon. A DBT for determining if your pimples go away is a different testing problem. The key word in my concern is subtle, and I advocated focusing on measurements for determining relative merit. I've participated in blind listening tests, I've talked to others who participated in blind listening tests, and the low quality of the answers, how many were just guesses versus definitive were always so low, that I simply disregard them, and that isn't total nonsense.

By the way, DBT comparison tests are not used for the detecting the thresholds of hearing.

 

[PENG]

Ah yes, the other DBT worshipper, mtycrafts. Just for starters, you need to improve your reading skills, or at least your comprehension skills when someone says your idol is flawed. What I said very specifically was that using listening tests, sighted or blind, to detect subtle differences between audio system components is flawed, because of the nature of hearing and auditory memory, and the fact that one is measuring human perception, not an observable phenomenon. A DBT for determining if your pimples go away is a different testing problem. The key word in my concern is subtle, and I advocated focusing on measurements for determining relative merit. I've participated in blind listening tests, I've talked to others who participated in blind listening tests, and the low quality of the answers, how many were just guesses versus definitive were always so low, that I simply disregard them, and that isn't total nonsense.

By the way, DBT comparison tests are not used for the detecting the thresholds of hearing.

And this, after I might have unintentionally contributed to clarifying your point in my post (#72), except I had the intended word "not" mistakenly missed in the second last sentence but can no longer edit it.

 

[Swerd]

I have followed this thread only for grins, primarily because it hasn't interested me enough to respond, until now.

Comparing the differences in sounds that might come from two different powerful and apparently well designed amplifiers is as interesting to me as comparing different brands of paint by listening to them dry. When audible differences become vanishingly small, the mind of the listener struggles to find something to detect. Blame the blind test designer, not blind perception tests in general.

… What I said very specifically was that using listening tests, sighted or blind, to detect subtle differences between audio system components is flawed, because of the nature of hearing and auditory memory, and the fact that one is measuring human perception, not an observable phenomenon.

That is the fault of human hearing and auditory perception, not the fault of blind testing. Blind testing attempts to account for those limitations.

… I've participated in blind listening tests, I've talked to others who participated in blind listening tests, and the low quality of the answers, how many were just guesses versus definitive were always so low, that I simply disregard them, and that isn't total nonsense.

Your conclusions may be incorrect. Blind listening tests reveal the flaw in human auditory perception, it isn't the flaw itself. The more obvious conclusion, at least to me, is that so many people guessed because they weren't sure they could hear any differences. Just like listening to different types of paint drying.

By the way, DBT comparison tests are not used for the detecting the thresholds of hearing.

Properly designed listening tests that include appropriate positive and negative listening test controls, and that include adequate statistical analysis, are essential to any attempt to determine what people can and cannot hear. This is true for any test that involves human perception. This isn't limited to medical testing, the food and wine business widely acknowledges the value of blind taste evaluations. Why is blind testing only questioned by some in audio?

 

[Irvrobinson]

I have followed this thread only for grins, primarily because it hasn't interested me enough to respond, until now.

Comparing the differences in sounds that might come from two different powerful and apparently well designed amplifiers is as interesting to me as comparing different brands of paint by listening to them dry. When audible differences become vanishingly small, the mind of the listener struggles to find something to detect. Blame the blind test designer, not blind perception tests in general.

That is the fault of human hearing and auditory perception, not the fault of blind testing. Blind testing attempts to account for those limitations.

Your conclusions may be incorrect. Blind listening tests reveal the flaw in human auditory perception, it isn't the flaw itself. The more obvious conclusion, at least to me, is that so many people guessed because they weren't sure they could hear any differences. Just like listening to different types of paint drying.

Properly designed listening tests that include appropriate positive and negative listening test controls, and that include adequate statistical analysis, are essential to any attempt to determine what people can and cannot hear. This is true for any test that involves human perception. This isn't limited to medical testing, the food and wine business widely acknowledges the value of blind taste evaluations. Why is blind testing only questioned by some in audio?

Swerd, you have also misinterpreted what I posted. I am not in any way demeaning the general concept of double-blind testing, I am questioning the veracity of the results when used in listening tests where the differences between A and B are subtle. There seems to be too much guessing and non-repeatable behavior involved, which in my opinion lowers the quality of the results.

My post was in the context of electronics, not speakers. Like so many people on this forum I have read Sean Olive's Audio Musings web site for years, and he is a proponent of DBTs for audio testing, and has posted many interesting studies and results. But Olive's tests involve speakers, he never mentions electronics. As I'm sure you'll agree, the differences between most speakers, except perhaps for different models of the same product line, are not so subtle as with electronics, and you don't have to guess to hear differences or express preferences. My contention is merely that for electronics, which at best normally have very subtle differences, DBTs aren't going to reliably tell you anything except that they are identical. I'm not advocating sighted tests as an alternative, I'm advocating just looking at measurements as an alternative.

 

[fmw]

I'm advocating just looking at measurements as an alternative.

Do you agree that all differences in measurements are not audible? Does it matter whether an amplifier has .01 % THD or .1%?

 

[Irvrobinson]

Do you agree that all differences in measurements are not audible? Does it matter whether an amplifier has .01 % THD or .1%?

I agree that just because differences are measurable does not mean they are audible. In the case of electronics I am merely saying that if you must differentiate between products, like amplifiers, use measurements, that DBTs are unlikely to yield high quality results.

It is an interesting question whether distortion differences of .01% and .1% are audible. As I discussed in a previous post, 0.1% is equivalent to -60db, or 30db of distortion in 90db of fundamental. I don't know if that's audible, but it strikes me as an interesting question, especially in how it might affect long-term satisfaction with a component.

 

[Swerd]

Swerd, you have also misinterpreted what I posted. I am not in any way demeaning the general concept of double-blind testing, I am questioning the veracity of the results when used in listening tests where the differences between A and B are subtle. There seems to be too much guessing and non-repeatable behavior involved, which in my opinion lowers the quality of the results.

I did misinterpret what you said. Thanks for the clarification. I think we actually agree with each other.

Related to all this is understanding that any test or measurement has a "lower limit of detection". It is up to the test designer to determine what this lower limit of detection is. It may very well be that blind tests of potential differences in audio amplifiers are beneath that lower limit. But to allow useful conclusions, a test has to define what the lower limit is.

The tester has to run a series of listening tests where there are known and defined differences in compared sounds. Imagine comparing a short musical passage to the same passage with various amounts of pink noise added to it. Listeners would hear music + 0%, 5%, 10%, 15%, 20%, etc. (I'm just spit balling here, I haven't done this.) The tester could then determine what level of added pink noise is readily heard by listeners, and this would be the lower limit of detection. Remember that it may very well be that half the listeners comparing music vs. music + 0% pink noise might report hearing differences. So the tester has to learn what level of added pink noise can be heard by significantly more than half the listeners. Only after doing this, can a tester conclude that differences in sound from two amplifiers is or isn't detectable. That's what I mean by a properly designed listening test.

 

[fmw]

I agree that just because differences are measurable does not mean they are audible. In the case of electronics I am merely saying that if you must differentiate between products, like amplifiers, use measurements, that DBTs are unlikely to yield high quality results.

Ok. We will disagree on that. If the differences are not audible, then the measurements aren't important in my view.

I suppose one could test two amps and find them to be audibly the same. Then you might redo the test using speakers that bring one of the amps to its knees while not affecting the other. The measurements might well differentiate or even predict that. That would be a case in which a DBT would show an audible difference with one set of speakers and not with another. I don't have any problem with that. And I do agree that measurements might predict that.

But for me it would simply be two different tests with two different results. And it wouldn't change the concept that the amps sound the same when working within their design parameters.

 

[Irvrobinson]

Ok. We will disagree on that. If the differences are not audible, then the measurements aren't important in my view.

But you have just hand-waved your way through the crux of the problem. You are making the assumption that certain differences are not audible because DBTs cannot reveal those differences, and my view is that there may be audible differences, and that we don't know how to test them, or at least we don't go through the trouble to do it. So in the end the difference between our views is that I think paying attention to specifications is more important then you apparently do, because I'm not sure what's audible and what isn't, and you are more confident that you do know.

 

[RichB]

But you have just hand-waved your way through the crux of the problem. You are making the assumption that certain differences are not audible because DBTs cannot reveal those differences, and my view is that there may be audible differences, and that we don't know how to test them, or at least we don't go through the trouble to do it. So in the end the difference between our views is that I think paying attention to specifications is more important then you apparently do, because I'm not sure what's audible and what isn't, and you are more confident that you do know.

... and perhaps there are ways to improve amplifier measurements.
Simulating speaker loads and that better simulate real speaker load and include amplitude variations in addition distortion.

- Rich

 

[fmw]

If audible differences don't exist in a listening test, then they don't exist. Listening is listening. There is no other way to determine if audible differences exist other than listening. Measurements are not ears and brains. They might be able to predict audible differences but they can't test for them. Only listening can do that.

 

[RichB]

If audible differences don't exist in a listening test, then they don't exist. Listening is listening. There is no other way to determine if audible differences exist other than listening. Measurements are not ears and brains. They might be able to predict audible differences but they can't test for them. Only listening can do that.

If audible differences are not detected in the listening test, they were not audible to the listeners.
If they were audible, it is bias, the test is invalid, or the equipment is broken

Got it

- Rich

 

[Irvrobinson]

If audible differences don't exist in a listening test, then they don't exist. Listening is listening. There is no other way to determine if audible differences exist other than listening. Measurements are not ears and brains. They might be able to predict audible differences but they can't test for them. Only listening can do that.

The problem is that listening tests have too many variables to be trustworthy. I think that unless you are the one running the tests, choosing the system, choosing and setting up the room, choosing the material, and choosing the participants, a listening test tells you too little. I think listening tests done by someone else are a worse predictor of satisfaction than measurements. If you tell me that an amplifier with distortion of -70db and an SNR of -80db is "good enough" because so-and-so's DBT listening tests said it was indistinguishable from the amp with -80db of distortion and an SNR of -100db, I'm still buying the amp with the better specs even if it costs a lot more. A single set of listening tests isn't trustworthy for long-term satisfaction.

 

[Hobbit]

The problem is that listening tests have too many variables to be trustworthy. I think that unless you are the one running the tests, choosing the system, choosing and setting up the room, choosing the material, and choosing the participants, a listening test tells you too little. I think listening tests done by someone else are a worse predictor of satisfaction than measurements. If you tell me that an amplifier with distortion of -70db and an SNR of -80db is "good enough" because so-and-so's DBT listening tests said it was indistinguishable from the amp with -80db of distortion and an SNR of -100db, I'm still buying the amp with the better specs even if it costs a lot more. A single set of listening tests isn't trustworthy for long-term satisfaction.

Hi Irv, While I agree with what you saying and would do the same thing as you in the case you suggest, it also our hobby after all too, from my experience more often the case is that at a given product level the specs will be quite close. If they're two marque brands we're considering with similar specs, I'd probably buy the one I can get the best deal on (sale!), assuming a significant difference in price, even if it was the one with the slightly worse specs. If the prices are comparable, I might just make an emotional choice towards a certain brand I like or trust more, better specs or not.

 

[Irvrobinson]

Hi Irv, While I agree with what you saying and would do the same thing as you in the case you suggest, it also our hobby after all too, from my experience more often the case is that at a given product level the specs will be quite close. If they're two marque brands we're considering with similar specs, I'd probably buy the one I can get the best deal on (sale!), assuming a significant difference in price, even if it was the one with the slightly worse specs. If the prices are comparable, I might just make an emotional choice towards a certain brand I like or trust more, better specs or not.

I agree, when the specs are similarly good I get into subjective factors, like build & parts quality, the reputation of the manufacturer, appearance, and of course price. You're right, it is still a hobby. When I bought the Levinson amps I used to own I admit that a significant factor in my purchase was that I was enamored with the engineering. They were utterly over-engineered, and I just liked that. My current ATI amp measures better I'm sure, and it's over 4db more powerful per channel into 8 ohm loads, but I don't get such a kick from taking the lid off and peeking inside.

 

[fmw]

The problem is that listening tests have too many variables to be trustworthy. I think that unless you are the one running the tests, choosing the system, choosing and setting up the room, choosing the material, and choosing the participants, a listening test tells you too little. I think listening tests done by someone else are a worse predictor of satisfaction than measurements. If you tell me that an amplifier with distortion of -70db and an SNR of -80db is "good enough" because so-and-so's DBT listening tests said it was indistinguishable from the amp with -80db of distortion and an SNR of -100db, I'm still buying the amp with the better specs even if it costs a lot more. A single set of listening tests isn't trustworthy for long-term satisfaction.

We're going around in circles. We're back to we don't know if things are audible or not because we can't trust bias controlled testing. Audible differences might exist or might not but we can't know for sure because there is no reliable test. So I'll buy some specifications just to be on the safe side. I'll never know whether or not it mattered, however, because I won't do a listening test because they aren't reliable. I'll just be comforted by the measurements. That, of course, is your privilege. It is not intuitive for me so I'll spend my time concerning myself with the things that I know have audible differences.

 

[fmw]

I guess Rich B is so incensed at what I say that he is driving me off the site with the negative reputation points. Since the site allows someone to do that, I'll wish you all farewell, good luck and good listening. I'm on to other things.

 

[Irvrobinson]

We're going around in circles. We're back to we don't know if things are audible or not because we can't trust bias controlled testing. Audible differences might exist or might not but we can't know for sure because there is no reliable test. So I'll buy some specifications just to be on the safe side. I'll never know whether or not it mattered, however, because I won't do a listening test because they aren't reliable. I'll just be comforted by the measurements. That, of course, is your privilege. It is not intuitive for me so I'll spend my time concerning myself with the things that I know have audible differences.

You've nailed exactly how I feel about it. We're going around in circles only because we're both stubbornly convinced we are correct.

 

[RichB]

I guess Rich B is so incensed at what I say that he is driving me off the site with the negative reputation points. Since the site allows someone to do that, I'll wish you all farewell, good luck and good listening. I'm on to other things.

FMW, I am not incensed at all and I do know what negative points are.
I centainly did not give you any knowingly.

- Rich

 

[PENG]

I did misinterpret what you said. Thanks for the clarification. I think we actually agree with each other.

Related to all this is understanding that any test or measurement has a "lower limit of detection". It is up to the test designer to determine what this lower limit of detection is. It may very well be that blind tests of potential differences in audio amplifiers are beneath that lower limit. But to allow useful conclusions, a test has to define what the lower limit is.

The tester has to run a series of listening tests where there are known and defined differences in compared sounds. Imagine comparing a short musical passage to the same passage with various amounts of pink noise added to it. Listeners would hear music + 0%, 5%, 10%, 15%, 20%, etc. (I'm just spit balling here, I haven't done this.) The tester could then determine what level of added pink noise is readily heard by listeners, and this would be the lower limit of detection. Remember that it may very well be that half the listeners comparing music vs. music + 0% pink noise might report hearing differences. So the tester has to learn what level of added pink noise can be heard by significantly more than half the listeners. Only after doing this, can a tester conclude that differences in sound from two amplifiers is or isn't detectable. That's what I mean by a properly designed listening test.

That makes sense to me and by the way Bob Carver, Julian Hirsch and others found out 0.15% crossover distortion was detectable if a single tone is used, with more and for complex musical material, the number increased, such as 6% for complex musical material. So much for the class A thing..