Study Shows No Correlation Between Price and Sound Quality in Headphones

[gene]

In a recent study of the frequency responses of over 280 headphones, it was found that price had no correlation to frequency response. Why is this a big deal? Earlier research found that frequency response was by far the greatest factor in headphone sound quality. So if price does not correspond to sound quality in headphones, what is really gained by spending hundreds or sometimes even thousands of dollars one a set, beyond build quality, comfort and features?

Read on to find out more about the research that prompts these questions.

Read: Study Shows No Correlation Between Price and Sound Quality in Headphones

 

[Joe B]

Gene,
Do you happen to have a link to "Earlier research found that frequency response was by far the greatest factor in headphone sound quality." I would love to see how other attributes of headphone sound reproduction figured in to the sound quality tested.

 

[shadyJ]

Gene,
Do you happen to have a link to "Earlier research found that frequency response was by far the greatest factor in headphone sound quality." I would love to see how other attributes of headphone sound reproduction figured in to the sound quality tested.

One of the most important papers in this field.

Other attributes of sound reproduction are addressed in the article. Time-related functions are not really a concern in headphones, since they usually use a single driver. Harmonic distortion is a problem for few headphones, according to Dr. Sean Olive. Differences in frequency response is what listeners respond to.

 

[TheWarrior]

One of the most important papers in this field.

Other attributes of sound reproduction are addressed in the article. Time-related functions are not really a concern in headphones, since they usually use a single driver. Harmonic distortion is a problem for few headphones, according to Dr. Sean Olive. Differences in frequency response is what listeners respond to.

Has Sean not yet finalized the measurement standard that he adapted from CEA2034 for headphones, I assume you would have mentioned it? (Or they updated 2034 to include it?)

They recorded the response of a number of popular headphones and then digitized the playback through a control set of headphones allowing listeners to perform a similar test to their 'speaker shuffler'.

 

[shadyJ]

I know how Sean measured his headphones and how Jeroen measured his, but I don't know how rigorously they adhered to any reference standard. There is this testing methodology developed by Olive, Welti, and McMullin, if that is what you are referring to, but that isn't quite like CEA-2034.

 

[tonmeister]

Has Sean not yet finalized the measurement standard that he adapted from CEA2034 for headphones, I assume you would have mentioned it? (Or they updated 2034 to include it?)

They recorded the response of a number of popular headphones and then digitized the playback through a control set of headphones allowing listeners to perform a similar test to their 'speaker shuffler'.

Let me try to answer how we measured the headphones. For around-ear/on-ear headphones we use a GRAS 45 CA with some custom pinnae that my colleague Todd Welti developed to more accurately reflect headphone leakage measured on real humans. see: http://www.aes.org/e-lib/browse.cfm?elib=17699

For in-ear headphones we currently use the GRAS RA0045 externally polarized coupler equipped with a GR0408 nozzle according to IEC 60318-4. This method provides a very good seal and we are able to predict listeners' preference ratings from these measurements because our replicator headphone used to simulate IE headphones also has a good seal. Whether this accurately reflects seal of real-world listening conditions is another matter.

In the last year GRAS has made some new anthropometric pinnae that provide more realistic leakage effects, and we are in the process of testing them.

 

[tonmeister]

Gene,
Do you happen to have a link to "Earlier research found that frequency response was by far the greatest factor in headphone sound quality." I would love to see how other attributes of headphone sound reproduction figured in to the sound quality tested.

There are a few papers where nonlinear distortion in headphones has been investigated and not found to be a big factor in the overall sound quality. There are 3 references mentioned in Breebaart paper which the article is about:

1. Fleischmann, F. , Estrella, J. , and Plogsties, J. (2014). “ A method for comparison of non-linearities of consumer earphones using equalized stimuli,” in Proceedings of the 136th AES Convention, Convention e-brief 141, Berlin, Germany.
2. Temme, S. , Olive, S. E. , Tatarunis, S. , Welti, T. , and McMullin, E. (2014). “ The correlation between distortion audibility and listener preference in headphones,” in Proceedings of the 137rd AES Convention, Convention Paper 9118, Los Angeles, CA.
3. Welti, T. , Olive, S. E. , and Khonsaripour, O. (2016). “Validation of a virtual in-ear headphone listening test method,” in Proceedings of the 141st AES Convention, Convention Paper 9658, Los Angeles, CA.

In reference [3] we compared binaural recordings of a) actual versus b) virtualized in-ear headphones where in b)only the magnitude and minimum phase responses of the headphones were simulated. In a) all linear and nonlinear distortions were included. Listeners rated both actual and virtual headphones and the correlation between the two was r = 0.98. In other words, any phase or nonlinear distortions present in a) did not have much influence on the overall preference ratings.

 

[AllanMarcus]

Wow, you are really stretching here. The study is literally called "No correlation between headphone frequency response and retail price" and you make some (possibly) logical conclusions, but the title of your article is totally misleading. The study did NOT conclude "No Correlation Between Price and Sound Quality in Headphones"! It concluded exactly what the title of the paper is called.

The article is better titled: Audioholics concludes No Correlation Between Price and Sound Quality in Headphones, and then you layout your arguments (as you have done).

There is also no evidence that cables matter, DACs matter, high quality amps are any different, or that hi rez is any better than red book. Does the internet cable matter? Does the brand of hard drive in the NAS matter? No. But people continue to soak up the voodoo that vendors sell with nonscientific studies, or even DBT that prove anything! We finally get a little academic rigor here, and you go and jump to non-scientifically proved conclusions!

Does the price of a headphone make it better? of course not. Are the best headphone the most expensive? No (well, yes). Are many of the most expensive headphones better? yes. This is true of anything! The Honda Civic is arguably one of the best cars made, so why do people buy more expensive cars?

I happen to own a few TOTL headphones, and I cannot tell the difference between one DAC and the next. Cables? other than how they look or feel, they all sound the same to me. but I can easily tell the difference between the Utopia and the LCD-4 and the HD800 and the HD650 and the MDR-6 (and many others).

Also, different headphone react differently to amps and volume levels. Distortion levels definitely change at higher volumes on cheaper headphones (and on a few expensive ones) to levels that are discernible.

It's good to have studies, but it's REALLY BAD to have journalistic sensationalism that purposely jumps to conclusions and misrepresents the study. The study shows one thing and your article title says it shows something different. That is purposely misleading.

 

[lovinthehd]

Could probably make the same argument for loudspeakers under the right room conditions...

 

[shadyJ]

Wow, you are really stretching here. The study is literally called "No correlation between headphone frequency response and retail price" and you make some (possibly) logical conclusions, but the title of your article is totally misleading. The study did NOT conclude "No Correlation Between Price and Sound Quality in Headphones"! It concluded exactly what the title of the paper is called.

The article is better titled: Audioholics concludes No Correlation Between Price and Sound Quality in Headphones, and then you layout your arguments (as you have done).

There is also no evidence that cables matter, DACs matter, high quality amps are any different, or that hi rez is any better than red book. Does the internet cable matter? Does the brand of hard drive in the NAS matter? No. But people continue to soak up the voodoo that vendors sell with nonscientific studies, or even DBT that prove anything! We finally get a little academic rigor here, and you go and jump to non-scientifically proved conclusions!

Does the price of a headphone make it better? of course not. Are the best headphone the most expensive? No (well, yes). Are many of the most expensive headphones better? yes. This is true of anything! The Honda Civic is arguably one of the best cars made, so why do people buy more expensive cars?

I happen to own a few TOTL headphones, and I cannot tell the difference between one DAC and the next. Cables? other than how they look or feel, they all sound the same to me. but I can easily tell the difference between the Utopia and the LCD-4 and the HD800 and the HD650 and the MDR-6 (and many others).

Also, different headphone react differently to amps and volume levels. Distortion levels definitely change at higher volumes on cheaper headphones (and on a few expensive ones) to levels that are discernible.

It's good to have studies, but it's REALLY BAD to have journalistic sensationalism that purposely jumps to conclusions and misrepresents the study. The study shows one thing and your article title says it shows something different. That is purposely misleading.

Many of these arguments are addressed in the article itself. Distortion levels are addressed, and besides that, did you read Sean Olive's reply directly above yours? Frequency response has been shown to be the greatest factor of sound quality in headphones by far. The article title does take a very slight liberty, since there was some very small correlation in price in the bass response of expensive headphones compared to cheap headphones, but it wasn't a a very tight correlation. It also may have been a attribute of the more expensive models' having a tighter and more consistent seal around the ears which would have an effect on the measured bass response.

 

[shadyJ]

Could probably make the same argument for loudspeakers under the right room conditions...

Yup. Check this study out.

"The target curves included the diffuse-field and free-field curves in ISO 11904-2, a modified diffuse-field target recommended by Lorho, the unequalized headphone, and a new target response based on acoustical measurements of a calibrated loudspeaker system in a listening room. For both headphones the new target based on an in-room loudspeaker response was the most preferred target response curve."

Listeners preferred the sound of headphones when they sounded like a neutral speaker set in-room.

 

[BoredSysAdmin]

A bit off topic, but brand of hard drives in nas does matter from reliability standpoint, but if using redundancy raid level, it much less important.

 

[TheWarrior]

Let me try to answer how we measured the headphones. For around-ear/on-ear headphones we use a GRAS 45 CA with some custom pinnae that my colleague Todd Welti developed to more accurately reflect headphone leakage measured on real humans. see: http://www.aes.org/e-lib/browse.cfm?elib=17699

For in-ear headphones we currently use the GRAS RA0045 externally polarized coupler equipped with a GR0408 nozzle according to IEC 60318-4. This method provides a very good seal and we are able to predict listeners' preference ratings from these measurements because our replicator headphone used to simulate IE headphones also has a good seal. Whether this accurately reflects seal of real-world listening conditions is another matter.

In the last year GRAS has made some new anthropometric pinnae that provide more realistic leakage effects, and we are in the process of testing them.

Thanks Sean. I must be confused on the purpose of the 'headphone sample area' you had constructed at Northridge. When I visited last August with Floyd I came away with the impression that the effort that had gone in to recording various headphones for comparison was intended to establish a measurement standard, not just annihilate the competition.

I may also be misremembering because I had too much fun 'testing the tests'. I recall a 'Huh!?' from you when the shuffler results gave me slight preference of Monitor Audio over Revel until you saw the listener consistency range gap - A last round "lets see what this does" score change.

 

[AllanMarcus]

A bit off topic, but brand of hard drives in nas does matter from reliability standpoint, but if using redundancy raid level, it much less important.

Reliability yes. Sound quality, no.

 

[BoredSysAdmin]

Agreed

 

[tonmeister]

Thanks Sean. I must be confused on the purpose of the 'headphone sample area' you had constructed at Northridge. When I visited last August with Floyd I came away with the impression that the effort that had gone in to recording various headphones for comparison was intended to establish a measurement standard, not just annihilate the competition.

I may also be misremembering because I had too much fun 'testing the tests'. I recall a 'Huh!?' from you when the shuffler results gave me slight preference of Monitor Audio over Revel until you saw the listener consistency range gap - A last round "lets see what this does" score change.

You are correct that one of the goals is to establish a standard for how we measure headphones both internally and hopefully externally. Our goal is to establish a headphone measurement that is accurate, repeatable and perceptually meaningful. It's not easy to achieve all three because of the leakage/fit factor. The most repeatable measurement is measuring a headphone on a flat plate with no head or pinnae, yet that condition most misrepresents leakage and the acoustic impedance effect between the headphone and the ear drum. That is what the coupler is for albeit it is generally agree they are not accurate above 8-10khz.

 

[<eargiant]

I'll say this, I have actual in-home/side-by-side experience with the Sennheiser HD600 and the HD800S and I can tell you that IMO the 800S are in a completely different league. Unmistakable from the first few seconds.

I'm sure that the HD600 have superb measurements that would (and do) make them a favorite (probably even in a study like the one that was linked) but the 800S outclass and outperform them in every way.

To put it another way, I have heard many speakers that sound like the HD600 but only few speakers that sound like the HD800S.

 

[jeffreynoah]

In the 1970s. some audio magazines used a ‘kunstkopf’ for objective headphone tests. Almost all audio columnists claim their memory of previous headphones is adequate for comparative evaluation. Absurd.

Headphone manufacturers must delight in such arbitrary claim.

One positive price vs accuracy correlation must be for electrostatic headphones. Almost all audio columnists praise ES headphone accuracy, if not bass response or dynamic range.

How about rank accuracy ratings of headphones on THD, etc., like Consumer Reports?

 

[TheWarrior]

In the 1970s. some audio magazines used a ‘kunstkopf’ for objective headphone tests. Almost all audio columnists claim their memory of previous headphones is adequate for comparative evaluation. Absurd.

Headphone manufacturers must delight in such arbitrary claim.

One positive price vs accuracy correlation must be for electrostatic headphones. Almost all audio columnists praise ES headphone accuracy, if not bass response or dynamic range.

How about rank accuracy ratings of headphones on THD, etc., like Consumer Reports?

This kind of objective testing and ranking makes enemies of advertisers very quickly...

 

[shadyJ]

In the 1970s. some audio magazines used a ‘kunstkopf’ for objective headphone tests. Almost all audio columnists claim their memory of previous headphones is adequate for comparative evaluation. Absurd.

Headphone manufacturers must delight in such arbitrary claim.

One positive price vs accuracy correlation must be for electrostatic headphones. Almost all audio columnists praise ES headphone accuracy, if not bass response or dynamic range.

How about rank accuracy ratings of headphones on THD, etc., like Consumer Reports?

Accuracy = frequency response. Electrostatic headphones wouldn't have any real advantage over conventional headphone design. Much like loudspeakers.

Inner Fidelity looks like it does good measurements for headphones in their reviews.

If I were headphone shopping today, I would just look for a frequency response that matches Olive, Welti, and McMullin's target curve, the 'neutral' response.