2010 Audioholics $1k Floorstanding Loudspeaker Faceoff

[gene]

Is Axiom set up to rotate the speakers into position, or are they side by side as yours were? That may be noticeable too that the speakers have changed positions.

No they were side by side like I did them. As far as I know, only Harman has the rotating platform for speaker swapping. The Axiom setup was very good and I will be doing a short editorial on it early this coming week.

 

[gene]

Also, the blind testing has been done and it was shown that you can't hear the difference. Alan Loft of Axiom was there for the tests.

I disagree. The problem with two-tailed tests where the objective is to prove the null hypothesis (all amps sound the same) is you wind up with a large amount of type 2 errors which can falsely conclude there wasn't an audible difference when in fact there likely was. There can be big differences in how an amp sounds and how it interacts with the complex load impedance of a loudspeaker even if both amps under comparison are operating in their linear region.

 

[AJinFLA]

The problem with two-tailed tests where the objective is to prove the null hypothesis (all amps sound the same)

I have never heard of such an objective, claim or test (of all amps ).
Can you provide a link?

There can be big differences in how an amp sounds and how it interacts with the complex load impedance of a loudspeaker even if both amps under comparison are operating in their linear region.

There are always extreme examples/poor designs, tube amplifiers with high output impedance, solid state with crossover distortion, etc, etc. that will indeed create an audible TF.
If you are referring to typical mass market "Pioneer receiver" solid state type amplification, I'd love to see your supporting evidence of "big differences in how an amp sounds" into real (non-extreme) loudspeakers under linear operation.
TIA.

cheers,

AJ

 

[lbolts20]

Thanks for doing this.

I was in the market for a set of towers and was considering the Axiom M60's. I wish they would have faired better in this test so I could just pull the trigger and buy them. I heard the Klipsch about a year ago side by side with the Paradigm Monitor 9's and didn't like either, more so the Klipsch.

The EMP's sound like they have good mids which is what I'm looking for, but these are ruled out since they don't make a larger center.

My search continues.

 

[MinusTheBear]

I have never heard of such an objective, claim or test (of all amps ).
Can you provide a link?

I am pretty sure Gene is referring to the statistics/math behind such testing which test a Null hypothesis Ho against an alternative H1 as well as the signficance (p -value) for type 1 and type 2 errors. These numbers are generated by excel from the data entered.

I have only read such statistics based papers done by Dr. Sean Olive and Dr. Floyd Toole but have never seen such testing on amps. That would be something to read. It would also be a lot of work.

 

[AJinFLA]

I am pretty sure Gene is referring to the statistics/math behind such testing which test a Null hypothesis Ho against an alternative H1 as well as the signficance (p -value) for type 1 and type 2 errors.

That's not what I'm referring to.
The tests of amplifiers that I have seen, were to determine whether audible differences between the DUT's could be detected by the listener(s), to statistical significance, under the tests conditions. That was the scope.
And yes, the results, for the DUT's, in those tests, by those particular listeners, under those conditions, were null.
I have not seen one where the objective was to prove the null hypothesis...or that a null, would indicate that "all amplifiers sound the same". It would simply indicate that no differences were detected, to statistical significance, within the scope of this particular test.
If there is a (valid) test that yields a statistically significant positive result for detection, I'd like to see it (link). TIA.

cheers,

AJ

 

[fredk]

AJ, that was my understanding as well. They basically chose a series of amps, some quite expensive and some not to see if people could hear a difference. As you wrote, they found no statistically significant difference in peoples ability to hear a difference.

 

[fredk]

Thanks for doing this.

I was in the market for a set of towers and was considering the Axiom M60's. I wish they would have faired better in this test so I could just pull the trigger and buy them. I heard the Klipsch about a year ago side by side with the Paradigm Monitor 9's and didn't like either, more so the Klipsch.

The EMP's sound like they have good mids which is what I'm looking for, but these are ruled out since they don't make a larger center.

My search continues.

It is interesting what people carry away from an article. Looking at the measured scores, I took it that three of the four speakers were very close to each other in quality of sound.

The individual comments were contrasts of how two sets of speakers sounded in relation to each other. I have since corresponded with an individual that owns both the M60 and a pair of RBH (NOT the EMPs in this face-off). He agreed with the comments, but pointed out exactly what I wrote above. He actually loved both speakers and feels that both are linear and neutral in their presentation. With respect to the bass output of the two, he said it was not that the Axioms were that bad, but that the RBH were really that good.

No doubt you can find better speakers, but it may end up costing you quite a bit more.

 

[PENG]

There can be big differences in how an amp sounds and how it interacts with the complex load impedance of a loudspeaker even if both amps under comparison are operating in their linear region.

I can see that if we compare amps that have significant output impedance. I would sure hope even mid range receivers with SS class A/B amps do not suffer from high output impedance like some tube amps do.

 

[gene]

Regarding Double Blind Test Issues

I normally don't link to competitor sites but this article was pretty well done discussing the issues in misinterpreting blind test results and how they can lead you to false conclusions that no audible differences were detected:

http://www.stereophile.com/features/141/

 

[AJinFLA]

I normally don't link to competitor sites but this article was pretty well done discussing the issues in misinterpreting blind test results and how they can lead you to false conclusions that no audible differences were detected:

http://www.stereophile.com/features/141/

Yep, all well known issues that one must be aware of when testing/drawing conclusions. Do note however, the utter frivolity of claiming not-so-subtle differences between electronic component X and Y, in casual "subjective" uncontrolled, sighted listening, when the same claimer is forced to randomly guess, when listening to X and Y blind.

What should also be noted, is how listeners can "hear" big differences, sighted, when told that B and C has been swapped, when in fact, nothing in the electro-acoustic system has changed whatsoever.

Luckily your tests were with Loudspeakers, where audible differences are not as subtle as to be "hidden" by blinding .

cheers,

AJ

 

[Alex2507]

I normally don't link to competitor sites but this article was pretty well done discussing the issues in misinterpreting blind test results and how they can lead you to false conclusions that no audible differences were detected:

http://www.stereophile.com/features/141/

13 pages? C'mon, look at my avatar. There's no way for me to get through that. Are there Cliff Notes available? I was under the impression that the $1,000 Denon, Yamaha, Onkyo and Marantz rec'rs would sound if not exactly the same then pretty close to it. Am I to understand that that is a misconception?

 

[gene]

13 pages? C'mon, look at my avatar. There's no way for me to get through that. Are there Cliff Notes available? I was under the impression that the $1,000 Denon, Yamaha, Onkyo and Marantz rec'rs would sound if not exactly the same then pretty close to it. Am I to understand that that is a misconception?

Yea agreed that article was lengthy even by my standards.

When comparing receivers in the same price class assuming they are all using similar amp topologies (discrete a/b) and are roughly the same weight than I think its safe to assume you are splitting hairs audibly and should instead be shopping features at that point. If however you are under the assumption that a $200 Pioneer receiver will be sonically indistinguisable to the $4k Axiom Class D amp as long as both amps aren't driven into clipping, than I'd say you are grossly mistaken.

 

[AJinFLA]

If however you are under the assumption that a $200 Pioneer receiver will be sonically indistinguisable to the $4k Axiom Class D amp as long as both amps aren't driven into clipping, than I'd say you are grossly mistaken.

Not familiar with the Axiom design, but if it has enough Class D high frequency eq/phase shift driving a real speaker, why wouldn't it sound different from the Pioneer? Does it?
Can the Axiom be distinguished from the Pioneer under controlled listening? Link?
If so, do the measurements readily explain it?
Not quite sure what you're getting at here Gene. Or what any of this might have to do with blind vs sighted listening.
One is the de facto standard of the scientific community and the other the standard of the audiophile community...who have magazines and jewelry to sell .

cheers,

AJ

 

[gene]

Not familiar with the Axiom design, but if it has enough Class D high frequency eq/phase shift driving a real speaker, why wouldn't it sound different from the Pioneer? Does it?
Can the Axiom be distinguished from the Pioneer under controlled listening? Link?
If so, do the measurements readily explain it?
Not quite sure what you're getting at here Gene. Or what any of this might have to do with blind vs sighted listening.
One is the de facto standard of the scientific community and the other the standard of the audiophile community...who have magazines and jewelry to sell .

cheers,

AJ

My point is people tend to dumb down results of sighted or blind tests without doing any real statistical analysis to determine the merit of the test or its results. Several reviewers of my staff were convinced all amps sound the same so I in fact did conduct a listening test switching between two amps (a really poorly designed Class D from Panasonic, and a mid priced Yamaha receiver). Every listener in the room heard a clear difference between the amps most notably in bass and with speakers that exhibited low impedance below 60Hz. Both amps were not driven near clipped levels and the listeners had no idea which amp they were listening to at any given time.

How an amp interacts with a speaker is a complex topic and whether or NOT listeners can identify the sonic differences in a particular listening room and program material is even more complex.

I am all for blind and controlled tests but often see too many people religiously clinging to them as absolutes for all situations (ie. all amps sound the same, all CD players sound the same, etc).

Be careful to not fall into this trap that I've often found myself falling into in the past since my background has a heavy basis in engineering and science.

 

[Swerd]

My point is people tend to dumb down results of sighted or blind tests without doing any real statistical analysis to determine the merit of the test or its results…

An important point in understanding the statistical analysis of any test or trial is just how many people participated. With 50 or fewer listeners you must be careful to avoid making broad or large scale conclusions. To get significant and reliable answers for any listening comparison test, you may need to test at least 10 times as many people.

For example, in experimental drug testing, a clinical trial may test a new drug to see its effectiveness against lung cancer. Let's say that existing standard therapy is known to cause tumor shrinkage in about 15% of patients who get it. First, a small clinical trial involving 50 or fewer lung cancer patients is done to test the new drug. The bar for effectiveness is usually set high, let's say 45% vs. 15% for the standard therapy. If the new drug does actually cause tumor shrinkage in 22 or 23 out of 50 patients (roughly 45%), then you have positive results. But these results do not mean that the new drug is effective in 45% of lung cancer patients. What it really means that statistically you can feel confident in going ahead with a much larger (and more expensive) clinical trial involving 500 to 1000 patients. If the small trial fails to meet the goal of 45%, it is probably a waste of time and money to go ahead with the larger trial. So the only valid conclusions you can make from a small trial is it worth it (YES or NO) to perform a large trial.

Only with large numbers of patients can you confidently say what the measured response rate is. They very same applies to listening tests. I've never seen any listening test performed with large enough numbers of listeners to allow for truly valid numerical conclusions.

 

[majorloser]

Man, this thread has gone too far into the geek realm for me.
I guess next time my comments will be, "Both speakers sound like poop."


FWIW, I now have a new take on "speaker shootouts".

 

[AJinFLA]

My point is people tend to dumb down results of sighted or blind tests without doing any real statistical analysis to determine the merit of the test or its results.

Fine, agreed, but how people interpret the data from blind testing is an entirely separate issue from whether they are required for perceptual threshold or distinction testing. Hopefully we can agree on the latter as well.

Several reviewers of my staff were convinced all amps sound the same

That is absurd. Please don't take offense, but if true, you need new staff or lots of staff training/education.
Anyone with technical literacy knows different amplifier designs can (not necessarily do) sound different when driving loudspeakers. There is no mystery there whatsoever.
Or anything that measurements (or metrics, such as GedLee, etc.) can't explain.

I in fact did conduct a listening test switching between two amps (a really poorly designed Class D from Panasonic, and a mid priced Yamaha receiver). Every listener in the room heard a clear difference between the amps most notably in bass and with speakers that exhibited low impedance below 60Hz. Both amps were not driven near clipped levels and the listeners had no idea which amp they were listening to at any given time.

Was this a surprise?
Poorly designed Class D vs AB...differences heard. Even blind. Hmmm .

How an amp interacts with a speaker is a complex topic and whether or NOT listeners can identify the sonic differences in a particular listening room and program material is even more complex.

Sure it's complex. But it certainly doesn't mean it's mystical and unsolvable. It requires robust applications of the scientific method, NOT dismissal and a belief that only sighted, uncontrolled, bias overloaded "listening" is appropriate.

I am all for blind and controlled tests but often see too many people religiously clinging to them as absolutes for all situations (ie. all amps sound the same, all CD players sound the same, etc).

I have never seen a technically literate person state any such thing (ie. all amps sound the same, all CD players sound the same, etc). Never. If anyone could provide a direct quote or link, please do so.
What I have seen, is the so called "subjectivist" religion, erect that very strawman argument. That because 2 (or more) DUT's were found to be indistinguishable without guessing, under test conditions, that all DUT's "sound the same". They are the only ones I've seen make such illogical and preposterous assertions and conflations.

Be careful to not fall into this trap that I've often found myself falling into in the past since my background has a heavy basis in engineering and science.

Duly noted. And you be careful inserting prices before electrical components when asserting (soundwave) perceptual differences .
Now what was all this about a loudspeaker shootout Gene??

cheers,

AJ

 

[tom67]

Keep it simple....

yeah, the Klipsch won the competition by a small margin among a cross section of listeners....and, a cross section of listeners is what its all about and reflects the market more than a few trained listeners with a set bias...

 

[Ito]

Would have loved to see a few more brands in here, especially monitor audio, but great article none-the-less.