Why Live-vs-Recorded Listening Tests Don't Work

[tonmeister]

Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful at convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance. His secret weapon was this elaborate live-versus-recorded demonstration that managed to convince people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the listenings tests since they are the only true valid measures of fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article.

 

[highfigh]

Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful at convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance. His secret weapon was this elaborate live-versus-recorded demonstration that managed to convince people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the listenings tests since they are the only true valid measures of fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article.

As bad as his hearing was, it probably did sound identical to live. (edit) I posted this before I read the blog and in your second sentence, you also refer to his hearing deficiencies but when the majority of the public was lacking in any technical knowledge, as they were at that time, almost any description will be believable.

That quote is also why people who sell things that require a subjective test generally tell the customer how it will sound or look before the demonstration- suggestive selling brings in much more money. Look at all of the "high end" cables- they make the sound 'chocolatey', 'three-dimensional', 'transparent', etc. Let's face it- when the bills are due, things just seem to sound and look better to the salespeople.

 

[tonmeister]

As bad as his hearing was, it probably did sound identical to live. (edit) I posted this before I read the blog and in your second sentence, you also refer to his hearing deficiencies but when the majority of the public was lacking in any technical knowledge, as they were at that time, almost any description will be believable.

That quote is also why people who sell things that require a subjective test generally tell the customer how it will sound or look before the demonstration- suggestive selling brings in much more money. Look at all of the "high end" cables- they make the sound 'chocolatey', 'three-dimensional', 'transparent', etc. Let's face it- when the bills are due, things just seem to sound and look better to the salespeople.

You are an audio cynic I can see. I find your lack of faith refreshing. Welcome to the Bright Side (science).

 

[highfigh]

You are an audio cynic I can see. I find your lack of faith refreshing. Welcome to the Bright Side (science).

I started selling audio in early '78 and most, if not all, sales training involved the trainers telling us to tell people what they would hear. We were told to say that the speakers would sound different at home and if they got the speakers home and decided they didn't like them, we could exchange them. The demo rooms weren't particularly good but at least they didn't have more than a couple of parallel walls. Looking back, the speakers weren't usually exceptional, other than the Micro Acoustics brand, which are among the worst I have ever heard. Most were OK and some were pretty good but far from what's available now. That's mainly due to the improvements over the last 30+ years in the quality of drivers and knowledge of what makes a speaker/crossover sound good, though.

To be honest, I agree that the reason live vs recorded tests don't work is that the list of variables makes it impossible to be totally accurate (you called them nuisance variables, IIRC). How does one make sure the mic is accurate enough? How is that tested and how is the rest of the test equipment validated? I have to assume it's through reducing and averaging variations, similar to how a once uneven farm field is plowed flat over time. Then, the recording equipment needs to be as accurate as possible and so on. Someone once said "Electronics- always precise, never exact".

I think the quest for absolute perfection is futile because that just doesn't exist and as Vince Lombardi said, "Perfection is not attainable, but if we chase perfection we can catch excellence". However, when a collection of really excellent equipment, a great room and the right program material are assembled, it's pretty amazing. I think the quest for greatness in the equipment is very noble but at some point, the people who are only concerned with the technical side need to listen to the music because that's why this stuff exists in the first place.

'Audio Cynic'- I think I just found the new name for my company. Thanks.

 

[jostenmeat]

Something that I bring up every once in a while, that I haven't seen others mention really, is that musicians sometimes completely change the way they play for a recording versus playing for an audience in a concert hall.

In a large hall, the closest person to you might be as far as 20 ft away. The back row can be very far away. A soloist for example must be able to carry the sound to anyone in the room (or try their darndest), and have some semblance of usable dynamic range. The more packed the audience is, the greater the exerted force might have to be. Even at the expense of unwanted extraneous sounds, and less consistency of tone for any line/melody.

For a recording, the mic can be just a foot or two away. That means any tiny little extraneous sound, like moving your butt on the chair, or barely audible squeaking of a string, even just breathing, will come out like sore thumbs. Thus, musicians will sometimes completely alter their fingerings for any given work. This can mean playing the same passages on completely different strings for example, for better clarity (without any worry of maximum volume capability), or better consistency of tone of the lines.

This was just one out of about a hundred things that crossed mind when a couple of people were bashing musicians for not having a clue about what the audience hears. I couldn't respond because the thread is locked. The only homes that I have ever personally been in with acoustically treated areas belonged to musicians.

While I wouldn't go so far as to say that live vs recorded tests are a must, as I just don't know enough about all of the difficulties in controlling variable, I do think it's cool when it happens. I think it might have been Triad Dude saying they had some world class musicians (possibly CSO members) in one room, with speakers playing back their stuff in an adjacent room, and they were just going back and forth between the two rooms. Not blinded, IIRC. I still think that's pretty neat, even if one cannot use it as scientific evidence. No offense to the scientists.

 

[skizzerflake]

Live vs recorded doesn't even make sense. The only way it would is if a recording was made "organically" in the sense that all instruments were recorded acoustically, with live musicians playing all the parts all at once. If you were there when it happened, there would be a standard of comparison.

Hardly anything is actually recorded that way except a precious few audiophile recordings. Unfortunately, in those, the musicians are playing very carefully and a lot of performance fire is lost. In most recordings, some parts are "phoned in" others are added in post production to add something and featured instruments or voices are added like the voice-overs in a movie. Missed notes are pasted over digitally. There never was ONE performance that was recorded and put on the disc, so there's no way to really know exactly what the performance sounded like.

 

[AJinFLA]

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the listenings tests since they are the only true valid measures of fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article.

So how should a (high performance) electro-acoustic system reproduction of a recording be judged for "Fidelity" to the "live" acoustic sources?
How should the loudspeaker portion of that system chain be judged individually?

cheers,

AJ

 

[tonmeister]

I started selling audio in early '78 and most, if not all, sales training involved the trainers telling us to tell people what they would hear. We were told to say that the speakers would sound different at home and if they got the speakers home and decided they didn't like them, we could exchange them. The demo rooms weren't particularly good but at least they didn't have more than a couple of parallel walls. Looking back, the speakers weren't usually exceptional, other than the Micro Acoustics brand, which are among the worst I have ever heard. Most were OK and some were pretty good but far from what's available now. That's mainly due to the improvements over the last 30+ years in the quality of drivers and knowledge of what makes a speaker/crossover sound good, though.

To be honest, I agree that the reason live vs recorded tests don't work is that the list of variables makes it impossible to be totally accurate (you called them nuisance variables, IIRC). How does one make sure the mic is accurate enough? How is that tested and how is the rest of the test equipment validated? I have to assume it's through reducing and averaging variations, similar to how a once uneven farm field is plowed flat over time. Then, the recording equipment needs to be as accurate as possible and so on. Someone once said "Electronics- always precise, never exact".

I think the quest for absolute perfection is futile because that just doesn't exist and as Vince Lombardi said, "Perfection is not attainable, but if we chase perfection we can catch excellence". However, when a collection of really excellent equipment, a great room and the right program material are assembled, it's pretty amazing. I think the quest for greatness in the equipment is very noble but at some point, the people who are only concerned with the technical side need to listen to the music because that's why this stuff exists in the first place.

'Audio Cynic'- I think I just found the new name for my company. Thanks.

The accuracy of the microphones, microphones methods, and other equipment in the recording chain can be quantified with objective measurements, if (and that's a big IF) the live-versus-recorded apologists accept objective measurements. The last one I talked to doesn't, and would have to select the most accurate mics via a live-versus-recorded comparisons of different mics. But then they are trapped in the circle-of-confusion because at some point they have to accept the loudspeaker is accurate.etc It's quite comical to try and follow their circular arguments as they navigate through the circle-of-confusion maze.

The other problem with live-vs-recorded method is that it does not allow for the fact that many artists and recording producers do not want recordings that are accurate: the recording becomes part of the art. In that case you it no longer becomes possible to judge the accuracy of the reproduction based on how close it sounds to the live performance.

What about the other 90% of recordings made in studios by a series of overdubs, for which there is no live performance equivalent? Then the live "reference" becomes what the artist heard through the loudspeakers. All we need to do then is replicate in our homes what they heard and the accuracy argument is over.

 

[tonmeister]

Something that I bring up every once in a while, that I haven't seen others mention really, is that musicians sometimes completely change the way they play for a recording versus playing for an audience in a concert hall.

In a large hall, the closest person to you might be as far as 20 ft away. The back row can be very far away. A soloist for example must be able to carry the sound to anyone in the room (or try their darndest), and have some semblance of usable dynamic range. The more packed the audience is, the greater the exerted force might have to be. Even at the expense of unwanted extraneous sounds, and less consistency of tone for any line/melody.

For a recording, the mic can be just a foot or two away. That means any tiny little extraneous sound, like moving your butt on the chair, or barely audible squeaking of a string, even just breathing, will come out like sore thumbs. Thus, musicians will sometimes completely alter their fingerings for any given work. This can mean playing the same passages on completely different strings for example, for better clarity (without any worry of maximum volume capability), or better consistency of tone of the lines.

This was just one out of about a hundred things that crossed mind when a couple of people were bashing musicians for not having a clue about what the audience hears. I couldn't respond because the thread is locked. The only homes that I have ever personally been in with acoustically treated areas belonged to musicians.

While I wouldn't go so far as to say that live vs recorded tests are a must, as I just don't know enough about all of the difficulties in controlling variable, I do think it's cool when it happens. I think it might have been Triad Dude saying they had some world class musicians (possibly CSO members) in one room, with speakers playing back their stuff in an adjacent room, and they were just going back and forth between the two rooms. Not blinded, IIRC. I still think that's pretty neat, even if one cannot use it as scientific evidence. No offense to the scientists.

What you say is true of course. I used to be a trained classical pianist and had to work hard to adjust my playing before recitals based on feedback from my piano teacher who sat in the back of the recital hall. Playing in recording studios is a different experience all together, and what you hear in the headphone mix can affect how you play.

Interestingly, Thomas Edison was very fanatical about his recordings not containing any mechanical sounds from piano damper felts, key noises, pages turning, extraneous noises -- even sounds from singers breathing! In reading about him one walks away with the impression that he was not a very musically cultured person, and difficult to work for.

Apparently Edison's singers adjusted how they sang to match the sound of their recordings. I don't know how they imitated the surface noise,etc on the recordings. Some of the accounts suggest that the musician actually sang in unison with their recordings so that the surface noise and distortion was always a constant. At some point they would stop singing and only the recording was playing.

This would largely explain how people were easily fooled since the listeners' task is not whether they are the live and recorded performance are the same or different, but whether you can detect when the singer stopped singing along. If the singer was purposely singing below the level of the phonograph, it would be even more difficult to detect when the singer had stopped singing.

A much more difficult test would be doing A/B comparisons between the live and recorded performances and saying whether they were the same or different.

 

[tonmeister]

So how should a (high performance) electro-acoustic system reproduction of a recording be judged for "Fidelity" to the "live" acoustic sources?
How should the loudspeaker portion of that system chain be judged individually?

cheers,

AJ

Well, the answer to that question could be my next article

 

[highfigh]

The accuracy of the microphones, microphones methods, and other equipment in the recording chain can be quantified with objective measurements, if (and that's a big IF) the live-versus-recorded apologists accept objective measurements. The last one I talked to doesn't, and would have to select the most accurate mics via a live-versus-recorded comparisons of different mics. But then they are trapped in the circle-of-confusion because at some point they have to accept the loudspeaker is accurate.etc It's quite comical to try and follow their circular arguments as they navigate through the circle-of-confusion maze.

The other problem with live-vs-recorded method is that it does not allow for the fact that many artists and recording producers do not want recordings that are accurate: the recording becomes part of the art. In that case you it no longer becomes possible to judge the accuracy of the reproduction based on how close it sounds to the live performance.

What about the other 90% of recordings made in studios by a series of overdubs, for which there is no live performance equivalent? Then the live "reference" becomes what the artist heard through the loudspeakers. All we need to do then is replicate in our homes what they heard and the accuracy argument is over.

I'll address the second paragraph first.

Many producers and labels have their own "sound" or sonic signature for the recordings they produce and when heard by people who know the difference, they can identify who produced it. Mic techniques, room sound, EQ, use of some instruments, etc can be telltale signs of that person's involvement. If they went for total accuracy, people might very well hate the sound because close up, not all instruments are pleasing to listen to. I played trombone and I sure wouldn't want to be inside the bell when it's being played. Even worse would be right in front of a trumpet- as much as I like some horn players, I don't want to be that close. Recordings are an interpretation of what they want the music to be and it doesn't matter what genre is being considered. Two painters may be looking at the same scene and they'll produce drastically different works, even though they may be using the same paints, brushes, canvas, etc.

The in-studio recording with overdubs is one reason I have told people that they'll never hear the music the same as when it was recorded. Even if the bed tracks were left alone after they were recorded- the room, speakers and other electronics won't be the same. When the musician wears headphones while recording tracks or parts of tracks, it's often left to the engineer or producer to "get the sound" on the master. That further removes the musician from the sound of the final result. Also, even if the instruments are all acoustic, playing that instrument is different from hearing someone else playing it.

If electronically produced waveforms were viewed on an oscilloscope, played through high quality electronics, viewed again, played through speakers and viewed again so the resulting waveforms could be compared, accuracy can be seen. If a recorded instrument's waveforms can be substituted for the electronically produced ones, the problem then becomes "what about the sound that projects in other directions and isn't picked up by the mic? The speaker can't produce 360° sound the same way an acoustic instrument does and how the room reflects the energy makes a huge difference in the way it's perceived in different parts of the room.

It's a sticky situation, this 'High Fidelity' thing.

 

[AJinFLA]

Well, the answer to that question could be my next article

That ought to be a doozy
And in the meanwhile?

 

[tonmeister]

I'll address the second paragraph first.

Many producers and labels have their own "sound" or sonic signature for the recordings they produce and when heard by people who know the difference, they can identify who produced it. Mic techniques, room sound, EQ, use of some instruments, etc can be telltale signs of that person's involvement. If they went for total accuracy, people might very well hate the sound because close up, not all instruments are pleasing to listen to. I played trombone and I sure wouldn't want to be inside the bell when it's being played. Even worse would be right in front of a trumpet- as much as I like some horn players, I don't want to be that close. Recordings are an interpretation of what they want the music to be and it doesn't matter what genre is being considered. Two painters may be looking at the same scene and they'll produce drastically different works, even though they may be using the same paints, brushes, canvas, etc.

The in-studio recording with overdubs is one reason I have told people that they'll never hear the music the same as when it was recorded. Even if the bed tracks were left alone after they were recorded- the room, speakers and other electronics won't be the same. When the musician wears headphones while recording tracks or parts of tracks, it's often left to the engineer or producer to "get the sound" on the master. That further removes the musician from the sound of the final result. Also, even if the instruments are all acoustic, playing that instrument is different from hearing someone else playing it.

If electronically produced waveforms were viewed on an oscilloscope, played through high quality electronics, viewed again, played through speakers and viewed again so the resulting waveforms could be compared, accuracy can be seen. If a recorded instrument's waveforms can be substituted for the electronically produced ones, the problem then becomes "what about the sound that projects in other directions and isn't picked up by the mic? The speaker can't produce 360° sound the same way an acoustic instrument does and how the room reflects the energy makes a huge difference in the way it's perceived in different parts of the room.

It's a sticky situation, this 'High Fidelity' thing.

You certainly understand and explained well the conundrum behind live-vs-recorded tests and "accuracy."

If we compared the sound waveforms of Edison's reproduced recordings measured at the ears of a listener versus those produced from the singer's mouth they certainly would not have been the same. Yet people thought they sounded the same.

 

[yepimonfire]

Sean, your blog is going to be to blame for me not going to sleep on a reasonable time

 

[highfigh]

You certainly understand and explained well the conundrum behind live-vs-recorded tests and "accuracy."

If we compared the sound waveforms of Edison's reproduced recordings measured at the ears of a listener versus those produced from the singer's mouth they certainly would not have been the same. Yet people thought they sounded the same.

At the time, it would have been thought of as "accurate" by most people because of the average level of schooling, the real lack of technology and the primitive nature of what was available to most people. Before 1910, something like 10% had electricity, 1% had a car, nobody had a radio, most didn't have a telephone and IIRC, most doctors didn't go to medical schools. Anything that came as close to the real voice would be astounding.

 

[tonmeister]

Sean, your blog is going to be to blame for me not going to sleep on a reasonable time

Well, if it's any consolidation I have the same problem as you with my blog. Creating the content takes some time, and then answering all the questions takes even more time. But, for now, it's fun, and actually one of my measurables at work - albeit a small one -- so I console myself knowing I'm getting paid for it )