Why are Speaker Review so bad?

[Sheep]

If a car can't handle the 50MPH around the same curve, it would fly off the road.

If a car can't stop fast enough, it would hit the object.

A 2-wheel drive car will have less traction and slide a lot more on snowy/wet roads than a full-time 4-wheel drive.

It is black and white. There is nothing subjective about that.

Speaker measurements may be objective and may be compared to car measurements.

But how well you like the bass or imaging or soundstage of a speaker is still 100% subjective.

That is not nearly as black and white as you made it out. My Rav4 will not handle as well as my Mazda 3. I understand cars are an easy comparison but I would forego using them for future comparisons.

SheepStar

 

[Swerd]

I'm finding it hard to believe the 4 expert reviewers can't come into a typical living room, set up speaker A, and then speaker B, and not be able to at least agree that one has better bass and one has better imaging etc. Beyond that I'm sure they'll have their personal preferences to which speaker they would buy based on the entire package. Most of us realize there will always be flaws in any review.

You will not find any printed audio publication that would publish such a review. A few online sites, such as this one, have tried this in the past, with mixed results.

It will be difficult to find 4 "expert reviewers" who don't depend, at least in part, on income from the major audio manufacturers.

And such a group review will generate an absurd debate (forum fistfight) over its merits or lack of merits. Was it done blind or sighted? There are those who swear that any comparison done without blinding the listeners is invalid because the attitudes, beliefs, and expectations of the listeners will affect their choices. There are also those who swear that any blinded comparison masks the very subtle audible differences between speakers.

In this debate, I support the idea of blind listening comparisons. It is embarrassing that the audio industry/hobby is crippled by the foolishness of those who refuse this rational approach. But the debate shows no sign of going away.

Edit: Most of the audio magazines (especially those that promote expensive products) have hitched their wagons to the irrational side of this debate.

 

[AcuDefTechGuy]

I understand cars are an easy comparison but I would forego using them for future comparisons.

I would agree.

 

[psbfan9]

Of course, like with cars, we'll have preferences. If I could afford to drive a Ferrari, I wouldn't care if a some Chevy can beat me 0-60 because I'll be in a Ferrari

I think you may be surprised how much you would care. My friends uncle owned one the largest RV manufacturing plants in town. The uncle made a ton, a literal ton, of money. He owned a Ferrari and my friend owned a 67 Chevelle. The inevitable drag race happened, and Scott's Chevelle smoked the Ferrari. Guess which one one was put up for sale the next day?

My point being, driving styles are different. Ears are different. Rooms are different. I think having four reviewers agree would be classified as a miracle. Too many variables. How many car reviewers agree on every point. Remember when Atkinson said that the bass on the Phil 3's was "A little slow". What does that even mean?
I've never heard the Phil 3's, but believe they're great speakers yet ADTG sold his. I bought my DefTechs blind on ADTG's recommendation. Would I trust him after he sold his 3's?
Maybe ...

There's a point to this rant and video some where ...

I really thought the Dodge would smoke the Tesla. Would a better driver in the Dodge have won? Damn variables. They're every where.

 

[AcuDefTechGuy]

Remember when Atkinson said that the bass on the Phil 3's was "A little slow". What does that even mean?
I've never heard the Phil 3's, but believe they're great speakers yet ADTG sold his. I bought my DefTechs blind on ADTG's recommendation. Would I trust him after he sold his 3's?
Maybe ...

My case study of speakers bought and sold is a clear example of personal preference.

In a blinded speaker test, people will choose certain speakers based on preference.

There is no such thing as a blinded test with cars. You definitely don't want to drive BLIND.

 

[Dr. Bob]

The blind shootout seems like a great way to evaluate speakers, but I've only managed to find this single example from 2007 - unfortunately involving speakers which are unavailable in any stores near me.

I agree that most reviews are nearly useless, but at Stereophile's site you can find the "Recommended Components" where speakers are ranked A, B, C, etc. This can at least give an idea where speakers fall relative to others.

 

[Hobbit]

You will not find any printed audio publication that would publish such a review. A few online sites, such as this one, have tried this in the past, with mixed results.

It will be difficult to find 4 "expert reviewers" who don't depend, at least in part, on income from the major audio manufacturers.....
.

Nice post! Perhaps the car analogy wasn't best, but these reviews use professional drivers.

I remember the first time I took a real race car for a "spin" on a banked oval. I came in and the guy who's car I was driving asked if the car was tight or loose. I stared at him dumbfounded then said depending how I entered the turn it was one or the other. He looked oddly at me, then said good answer. The point being professional race car drivers can pick subtleties -and the SAME ones out- that the novices and even neopros can't. If I listened to different speakers all day I surely would be able pick nuances out that I can't now.

edit: I want to add that this doesn't mean all drivers want a neutral car. Some want it a little loose, some a little tight. It is a preference. Though, I'm sure to most of us this "little" is hardly distinguishable.

To PSB's remarks, the guy with the Ferrari should have tried to get the guy with the Chevelle to race him on the track! That is more of an apple to orange comparison - a track car to a drag race car. I'm sure a Bentley would win for ride comfort and quietness. In F1 I remember JPM in his BMW powered Williams taking poles from points leader Shooey in the Ferrari. JPM could manhandle that car using the BMW power for 1 lap. A few laps into the race the lesser powered Ferrari with the better chassis would go right by... But really, I used to live near a Bentley/Aston Martin dealership and I loved the Continental GT

 

[Irvrobinson]

Remember when Atkinson said that the bass on the Phil 3's was "A little slow". What does that even mean?

Assuming you're talking about John Atkinson of Stereophile, when did he ever say that? I searched the Stereophile web site, and the only reference to the Phil 3 (a report from an audio show) was very positive.

When I'm reading loudspeaker reviews I really only read the measurements section with any seriousness. Especially JA's work in Stereophile, which I find very useful for conventional box speakers. (Much less so for panel types.) I find a strong correlation between speakers that get good measurements from him and those I find appealing to listen to. Most of the time I find the subjective prose his writers contribute is not so useful, except for Larry Greenhill. This is a guy who owns and prefers Quad ESLs, which I think is very telling. IMO, if a box speaker sounds good to a guy who owns ESLs it is an opinion worth listening to. But most of the time I find subjective reviews or the subjective portions of complete reviews, even on AH, to be more entertainment than informative. Especially on subwoofer reviews.

This doesn't mean I would make a buying decision based on measurements, it is just an indicator of what I think adds value versus entertainment in a review.

 

[Swerd]

Assuming you're talking about John Atkinson of Stereophile, when did he ever say that? I searched the Stereophile web site, and the only reference to the Phil 3 (a report from an audio show) was very positive.

He said that at an audio show in Washington, the DC AudioFest, a number of years ago. It was in one of those informal write ups that get printed in some kind of audio show blog soon after the show.

I remember because I was there at the time he was in the room. Dennis Murphy knew who he was and gave me a sign, and later explained when he had left. I had previously helped Dennis find the dead and live bass locations in the room, as he decided where to locate the speakers. JA was sitting in one of the wrong spots.

I agree that JA's speaker measurements in Stereophile are good. But he can also be as wrong as the rest of us at times.

 

[lsiberian]

There is no way I'm walking, especially in 10F or 100F degrees.

What's wrong with 100F degrees?

 

[Seth=L]

Because it's 100 f'n degrees yo

 

[AcuDefTechGuy]

I'll bet most $1000 (or $5000) dollar speakers will end up performing more similar than different.

That's the key right there. If the reviewer really likes all of the speakers, he's going to have a difficult time judging the speakers. It is just his opinion.

Guys with multiple sets of speakers that they like can relate to that.

 

[Irvrobinson]

I agree that JA's speaker measurements in Stereophile are good. But he can also be as wrong as the rest of us at times.

Like when he's arguing that cables make an audible difference...

 

[Sheep]

I agree that JA's speaker measurements in Stereophile are good. But he can also be as wrong as the rest of us at times.

See that's something I have a problem with. When I find out someone believes in that side of Audio I have a hard time accepting anything they say, even if it is 2+2 = 4. I find that I won't take anything at face value and go check for myself (if its something I'm thinking of buying at least).

SheepStar

 

[andy19191]

See that's something I have a problem with. When I find out someone believes in that side of Audio I have a hard time accepting anything they say, even if it is 2+2 = 4. I find that I won't take anything at face value and go check for myself (if its something I'm thinking of buying at least).

SheepStar

I have seen nothing to suggest that John Atkinson believes expensive audiophile cables possess properties unknown to science/common sense but his job certainly involves promoting such cables. For example, when the James Randi cable challenge was in full swing, unlike his writer and despite the offer of a million dollars, he knew perfectly well he would not be able to identify the cables under blind conditions.

 

[Sheep]

I have seen nothing to suggest that John Atkinson believes expensive audiophile cables possess properties unknown to science/common sense but his job certainly involves promoting such cables. For example, when the James Randi cable challenge was in full swing, unlike his writer and despite the offer of a million dollars, he knew perfectly well he would not be able to identify the cables under blind conditions.

Thanks for making my point. He can be bought. If that's what he wants to go, good on him, but I'm not going to listen.

SheepStar

 

[psbfan9]

He said that at an audio show in Washington, the DC AudioFest, a number of years ago. It was in one of those informal write ups that get printed in some kind of audio show blog soon after the show.

I remember because I was there at the time he was in the room. Dennis Murphy knew who he was and gave me a sign, and later explained when he had left. I had previously helped Dennis find the dead and live bass locations in the room, as he decided where to locate the speakers. JA was sitting in one of the wrong spots.

I agree that JA's speaker measurements in Stereophile are good. But he can also be as wrong as the rest of us at times.

Yes, this is the quote. Thanks Swerd.

 

[psbfan9]

Like when he's arguing that cables make an audible difference...

I'm sure I'll take some hits for this but ... Here it is. I have no science to back this up but I believe this is true to a certain point. If someone hooks up their, lets say, EMPtek's, or, Revel Salons, with the same speaker wires that came with their HTIB, I think new thicker wires would make a difference. Again, that's just off the top of my head. Is there a difference between Blue Jeans top of the line cables and Cardas? I doubt it.
This goes back to the convoluted point I was trying to make about these guys reviewing speakers with treated rooms, 500 watt mono blocks and $300 a ft speaker wires, and so on. The average person reading that review will not have the same results with those speakers in their room.

 

[AcuDefTechGuy]

Thanks for making my point. He can be bought. If that's what he wants to go, good on him, but I'm not going to listen.

SheepStar

Can't we all be bought to some extent?

If my boss tells me to review some cables and fib a little, I'm not going to risk my job denying him the request because we are just talking about cables here, not a matter of life and death.

If AJ wants to continue working for Stereophile, he has no choice but do things that they want him to do, even if he doesn't want to do.

AJ made it clear on AVS that he doesn't like to review cables. But he does like to keep his nice job.

I like to keep my nice job. I think most people would like to keep their nice jobs even if that means fibbing about some harmless worthless cables.

Oh, BTW, I use Kimber Kable 8PR, which costs $6 per foot bulk. Hey, they look pretty.

 

[psbfan9]

Can't we all be bought to some extent?

If my boss tells me to review some cables and fib a little, I'm not going to risk my job denying him the request because we are just talking about cables here, not a matter of life and death.

If AJ wants to continue working for Stereophile, he has no choice but do things that they want him to do, even if he doesn't want to do.

AJ made it clear on AVS that he doesn't like to review cables. But he does like to keep his nice job.

I like to keep my nice job. I think most people would like to keep their nice jobs even if that means fibbing about some harmless worthless cables.

Oh, BTW, I use Kimber Kable 8PR, which costs $6 per foot bulk. Hey, it looks pretty.

Um, ok, but, um... If Stereophile loses credibility due to 'small fibs', 'white lies', or just out right lying, then there would be no job for JA to go to.
Credibility. It matters!