Another double blind study on CD vs mp3 quality - Can a difference be heard?

[Darkwing_duck]

Found this and thought id share.

Double blind study done by McGill University on music - CD quality vs mp3 - Can a trained listener hear a difference?

http://www.music.mcgill.ca/~hockman/documents/Pras_presentation2009.pdf

I know this topic probably has been debated to death.....but its a double blind study using reference equipment, speakers and an acoustically treated room evaluated by smart people. Just an interesting read to confirm what we already know.

The study suggested....No. At the highest quality available of either format, a difference could not heard but the studio engineers still prefered CD quality over the highest quality mp3 and the musicians couldnt prefer one medium over the other at the highest quality mp3 vs cd.

 

[BMXTRIX]

But, that can't be right... I know people who listen to their headphones when they jog, or drive around in their cars who always know that the CD sounds such much better than their 320kbs MP3 files!

With 50% as a baseline number, it seems like everyone who took part in the test really struggled around the 256/320 mark vs. the CD and that it really is 'good enough'.

I think it would be interesting to incorporate the HD audio formats available on Blu-ray to see how they would stack up in a similar test.

Seems like good effort put in and a solid study.

 

[j_garcia]

IMO, 320 IS "good enough" for "general" listening. IMO, on headphones, I can't really tell the difference at that point which is why I usually rip at 320. It isn't until you take that to the main rig that you may start to hear differences in the top end and bass IMHO.

 

[ReUpRo]

I have always used 128Kbps MP3s for my car and portable audio. Neither of these are critical listening setups nor am I looking for it while driving or exercising. At these times I'm looking for quantity over quality .

 

[its phillip]

lame v2 or v0 is the only way to go for mp3s.

 

[AcuDefTechGuy]

I agree with the study. I can't tell the difference when both MP3 and the CD/lossless files are very high quality.

 

[Pyrrho]

But, that can't be right... I know people who listen to their headphones when they jog, or drive around in their cars who always know that the CD sounds such much better than their 320kbs MP3 files!

With 50% as a baseline number, it seems like everyone who took part in the test really struggled around the 256/320 mark vs. the CD and that it really is 'good enough'.

I think it would be interesting to incorporate the HD audio formats available on Blu-ray to see how they would stack up in a similar test.

Seems like good effort put in and a solid study.

I don't think the HD audio formats would matter. When CD quality digital conversion is inserted into the signal path, people cannot hear the difference:

BAS Experiment Explanation page - Oct 2007

Boston Audio Society - ABX Testing article

This, by the way, is the best way to test such things as what the CD format does to the sound; if one buys different discs, one very likely will have different masterings, and as everyone should know, if they stick different stuff on the disc, then it may sound different, even with the same type of disc.


To the opening post: Even if the difference is not generally audible, the compressed MP3 is still not ideally suited for archival storage, as one may wish to convert the signal to some other format in the future, and that will be best done from the full signal rather than from one that has thrown away some of the data.

Also, some people are going to forget the fact that they are discussing less compressed MP3s than what many people use, and consequently it isn't relevant to those more heavily compressed MP3s. That is, MP3s are at different levels of compression, and some of them do audibly degrade the sound. So it is important to remember that when considering what one wishes to do.

Since the reason to use compression is to save storage space, if one has plenty of storage space (which is becoming increasingly common), then there is no reason to compress the signal at all and consequently one may as well keep the full CD signal rather than compress it. It is only if one does not have the storage space that it becomes desirable to consider data compression. And then I agree that it can be done in a way that will not be audibly noticeable. But it isn't always done that way, and so some care must be taken if one wishes to avoid degrading the sound.

 

[lsiberian]

I'm not sold that no one can hear a difference, but I am sold that the brain can compensate in most situations.

 

[ImcLoud]

I have a b tested with myself, between my cd player and my ipad with the smale songs at the same volume and I can't tell which is better, but I have terrible ears that are mounted on the sides of my head, not like most audiophiles, their ears are mounted in a much better position that allows them to hear the difference between $50 speaker cables and $5000 cables...

 

[lsiberian]

I have a b tested with myself, between my cd player and my ipad with the smale songs at the same volume and I can't tell which is better, but I have terrible ears that are mounted on the sides of my head, not like most audiophiles, their ears are mounted in a much better position that allows them to hear the difference between $50 speaker cables and $5000 cables...

I couldn't tell with most things. Only music that was extremely well mastered and I was very familiar with and believe me that's only up to a certain bit rate. I really don't care that much. I listen to CDs at home and AAC on my phone. For standard rock lower bitrates were just fine.

 

[PENG]

In my collection of discs the best sounding SACDs do not sound better than the best CDs. I do think I can hear the difference between CD and the best (320) MP3 at least that's what I found comparing my CDs to those ripped from the same CDs to 320 MP3. To me it is surprising to see that in the study it shows most people couldn't hear the difference. Put it that way, the difference betwee MP3 320 and CD is much more obvious than that between my amps.

 

[Darkwing_duck]

But, that can't be right... I know people who listen to their headphones when they jog, or drive around in their cars who always know that the CD sounds such much better than their 320kbs MP3 files!

With 50% as a baseline number, it seems like everyone who took part in the test really struggled around the 256/320 mark vs. the CD and that it really is 'good enough'.

I think it would be interesting to incorporate the HD audio formats available on Blu-ray to see how they would stack up in a similar test.

Seems like good effort put in and a solid study.

If you look at the last slide or so of the presentation you will see "Recommendations" and a bullet note that said to include higher res HD files for the next study

 

[herbu]

The result of any subjective experiment, (ex. At what point can you hear/see/feel/smell/taste the difference), will be a normal bell shaped curve. Any particular individual is likely to fall within +/- 6 sigma, but any particular individual COULD be an extreme out-lyer, (or out-liar).

There can never be a descrete definitive answer.

 

[cpp]

The way I look at these test or when people claim they hear a difference, I say good for them. Since our hearing abilities are different and our brains which compute what we hear are different nobody can criticize a person for what they hear, unless they are in fact that person.

 

[TLS Guy]

That was not a good study. The listening times were too short. One of the recordings was anechoic and had no ambient envelope.

The ability to hear a difference in sound is high program dependent. You have to listen over time, as there are discrete incidents that give the game away for lossy codecs.

The best source for finding it out is music from UK cathedrals. The high boys voices against the controlled restraint of the deep bass of the British cathedral organs is an absolute torture test.

Even at 320 kbs AC3 this type of programming will catch the codec out, with frank and obvious twinking and bass artifacts you could never miss. Although with that codec and bit rate gross incidents are far and few between, but quite common at 180 and unlistenable at lower bit rates.

On this rig it is not hard to find the shortcomings of these codecs, although in the main, and for most orchestral program 320 KBs AC3 produces a very enjoyable listening experience, but it still does not sound like a good CD, and especially not a good BD.

I admit my rig is very high resolution and above that of the speakers shown in that study, especially the bass definition, where MP3 has particular problems, which is why BBC engineering have ditched it.

For reference quality there is no substitute for loss less recordings, especially if they have a lot of sustained deep bass high HF content and a hue ambient envelope. That type of program will catch out any lossy codec I'm familiar with.

 

[AcuDefTechGuy]

I have a b tested with myself, between my cd player and my ipad with the smale songs at the same volume and I can't tell which is better, but I have terrible ears that are mounted on the sides of my head, not like most audiophiles, their ears are mounted in a much better position that allows them to hear the difference between $50 speaker cables and $5000 cables...

True audiophiles listen to their Speakers, amps, and cables.

I only listen to the music.

 

[Swerd]

I thought this test was decent. Compared to other attempts at blind listening tests I've seen in the great audio debate, this was very good. They proposed a question that could be put to the test, and their main conclusions were reasonable. Some people may be disappointed in what it didn't do or say, but that always is true with good science, it's not for the impatient.

They tested 13 individuals, 4 musicians and 9 studio engineers, who they described as "trained" listeners. And they performed 150 repetitions of the various trials (5 short musical passages and 6 different digital formats), on average a little more than 10 trials per person. The people who ran this study were in an academic psychology department, seemed to understand concepts of blind randomized studies of human perception, and knew how to apply statistical analysis, all without making any obvious errors. They presented the results in a clear fashion, and although I didn't work through the statistics myself, their statistical analysis did not look unbelievable.

Overall their conclusions looked valid to me, especially their first one:

Trained listeners can hear differences between CD quality and mp3 compression (96-192 kb/s), and prefer CD quality.
​

Nothing too surprising there, but they really did have the data to back up those conclusions.

Their second conclusion was fine, until the last part (I bolded & underlined it):

Trained listeners cannot discriminate between CD quality and mp3 compression (256-320kb/s), while expert listeners could.
​

Because they had so few individuals (13) they can't compare results from experts and non-experts. It might take at least 10 times as many individuals before they could make statistically valid observations about that. And they never defined what makes for a trained listener and what makes for an expert trained listener.

Other than that somewhat minor quibble, I liked this test. Thanks for posting it.

 

[ReUpRo]

(Based on side 17) Next time a studio engineer tells me to buy a specific speaker or stay away from another, I'm going to give it a lot of weight .

Can someone please explain slide 18 - Sound Criteria?

 

[Pyrrho]

The way I look at these test or when people claim they hear a difference, I say good for them. Since our hearing abilities are different and our brains which compute what we hear are different nobody can criticize a person for what they hear, unless they are in fact that person.

People have claimed to hear differences when someone pretended to change something, without actually changing anything. So the "difference" that they heard is in their head, not anything to do with actual sound waves. So people's claims about what they can hear and distinguish are simply unreliable. If someone makes a claim of having superhuman ability, no person of sense will take them seriously unless they have proper evidence for their claim.

The peculiarities of human perception are not limited to audio. There are strange things with the taste of wine as well:

Price changes way people experience wine, study finds

People think wine tastes better when they believe it is expensive. Same wine, but different thing going on in the person's head (see article at link).

So, when someone experiences something differently, it may be purely a matter of what is going on in the person's head, and have nothing whatsoever to do with the thing itself.

Therefore, when they make a claim about the thing itself, one may justly criticize the person for their claim.

 

[slipperybidness]

People have claimed to hear differences when someone pretended to change something, without actually changing anything. So the "difference" that they heard is in their head, not anything to do with actual sound waves. So people's claims about what they can hear and distinguish are simply unreliable. If someone makes a claim of having superhuman ability, no person of sense will take them seriously unless they have proper evidence for their claim.

The peculiarities of human perception are not limited to audio. There are strange things with the taste of wine as well:

Price changes way people experience wine, study finds

People think wine tastes better when they believe it is expensive. Same wine, but different thing going on in the person's head (see article at link).

So, when someone experiences something differently, it may be purely a matter of what is going on in the person's head, and have nothing whatsoever to do with the thing itself.

Therefore, when they make a claim about the thing itself, one may justly criticize the person for their claim.

You are making some very valid points here, and all of this has actually been scientifically proven.

A really good series to watch is "Your Bleeped Up Brain"
Your Bleeped Up Brain - Episodes, Video & Schedule - HISTORY.com

It is only 3x 1hr episodes and it is awesome!

One of the experiments that they did was to give someone lime jello but color it red. Everybody kept claiming that it was the fruit punch flavor. They saw red color so they tasted fruit punch, even though it most certainly wasn't fruit punch (I may have my exact facts a little off here, but you get the idea).