The Ethics of Product Review

[av_phile]

Here's a quote from one of Gene's responses on the Z9 thread. And according to the administrators, I should just post my questions here. Highlight is mine.

The RX-Z9 will handle NTSC and PAL, not sure about the variants you mentioned. I will ask Yamaha after they look over my review. There will likely be a review addendum after my review is posted to handle everyones questions, or we may just feature it on this forum if it remains highly active.

I am curious. Why should an INDEPENDENT reviewer allow his review to be looked over by the manufacturer of the product under test?

Having worked with film critics and consumer advocacy groups in the past on the side, I have never encountered a film reviewer submitting his review of the film to its director prior to publication. That would be like showing the pupils the result of their exams before finally putting in the grade. It is my undestanding that a review or test is supposed to validate the manufacturer's claims on the product. So it would be unethical for the manufacturer to have even a hint of the review prior to publication.

[gene]

As usual AV_Phile you take my commentary or others commentary out of context. Yes we do allow manufacturers to look over the technical aspects our reviews before posting to ensure we did not incorrectly state things about the product(s) in question. This is a professional courtesy we extend to manufacturers as do most credible product review websites and magazines. This is especially important to us because we like to provide indepth technical reviews/analysis unlike much of our competition that simply do cursory one pagers and want to make sure we have our facts straight. Its often difficult to get all the facts on products when you don't have schematics, firmware, etc to confirm your findings or suspicions and the manufacturers can be of great help in this regard.

This is also a policy we made explicitly clear and if you spent more time reading the review section of our site rather than obsessing over all channels power driven tests you would know this. If you cannot see the merit or logic in this, I don't know what to tell you. We are neither film critics or part of a consumer advocacy group so I cannot comment on their policies.

If you don't like or agree with our policy, you are more then welcome to not participate in our free forums or read our articles that we offer the public for free. It's weird that you are like this, especially since you spend more time on our forums than almost any of our members. Perhaps you just enjoy arguing with people. Unfortunately I don't have the time or desire to do so.

Most of the people on this site spend countless hours contributing out of shear joy and love for the hobby. You discredit their efforts and their work by your continuous compulsion to point fingers of foul play at work here both in these forums and in private messages you have been sending myself and my staff over the past week.

It's your choice to use this site, enjoy the forums, and participate. You usually offer very good contribution here, but quite frankly if you continue down this path, don't let the door hit your *** on the way out! There I vented my steam, nuff said :mad:

[shlomore]

as a member here i would like to say that not only this site is so professional,gene which doing great job is a very professional in what we all like in here..
thank you so much gene and everyone in this great site !!!













As usual AV_Phile you take my commentary or others commentary out of context. Yes we do allow manufacturers to look over the technical aspects our reviews before posting to ensure we did not incorrectly state things about the product(s) in question. This is a professional courtesy we extend to manufacturers as do most credible product review websites and magazines. This is especially important to us because we like to provide indepth technical reviews/analysis unlike much of our competition that simply do cursory one pagers and want to make sure we have our facts straight. Its often difficult to get all the facts on products when you don't have schematics, firmware, etc to confirm your findings or suspicions and the manufacturers can be of great help in this regard.

This is also a policy we made explicitly clear and if you spent more time reading the review section of our site rather than obsessing over all channels power driven tests you would know this. If you cannot see the merit or logic in this, I don't know what to tell you. We are neither film critics or part of a consumer advocacy group so I cannot comment on their policies.

If you don't like or agree with our policy, you are more then welcome to not participate in our free forums or read our articles that we offer the public for free. It's weird that you are like this, especially since you spend more time on our forums than almost any of our members. Perhaps you just enjoy arguing with people. Unfortunately I don't have the time or desire to do so.

Most of the people on this site spend countless hours contributing out of shear joy and love for the hobby. You discredit their efforts and their work by your continuous compulsion to point fingers of foul play at work here both in these forums and in private messages you have been sending myself and my staff over the past week.

It's your choice to use this site, enjoy the forums, and participate. You usually offer very good contribution here, but quite frankly if you continue down this path, don't let the door hit your *** on the way out! There I vented my steam, nuff said :mad:

[Yamahaluver]

av_phille,

The crux of your problem doesn’t really lie with Gene’s review but your strong and apparent bias towards anything Japanese, especially any product bearing the Yamaha name. If we were to analyze all your previous posts, one can easily come to the conclusion that either you suffered a real sad experience with Japanese products or Yamaha as you have not even a single good word for them. It is perfectly fine to have your biases and opinions, however strong that may be as we as humans are entitled to that freedom but imposing them on others or trying to shape it to your personal preferences reeks of dogmatism which was quite prevalent in the Soviet Union.

You would rather that Gene rants and raves about your favorite snob appeal names like Rotel etc. rather than dare to mention the dreaded Yamaha word. The reason Gene sent out the review for perusal by the manufacturer shows of AH’s concept of fair play which is what drew to me for the first time and I continue to do so as long as Gene and others allow me. This gives the manufacturers to point out any technical anomalies in the review or a chance for the manufacturer to clarify certain points, I along with others would have it no different way and if this irks you, there are so many other forums catering to snob appeal, non Japanese brands and as a matter of fact, the mere mention of anything Japanese on those particular forums would get you barred and detested to say the least. Why do you still hang around here and get really frustrated is beyond my comprehension. Sadly it seems that the editorials as well as the general atmosphere of the forum doesn’t suit your disposition, one thing you should get by now. We all value our opinions however slanted, stupid they may be and to change them would be a totally futile exercise on your part.

Sadly you remind us of an ex-member who is thankfully not here anymore as to him, anything from the east has no right to be in the audio forum. You are slightly different as your concepts are much clearer than him, only that your bias clouds your position from time to time. Learn to live with that and we all would have a good time.

[steve]

Greetings AVPhille,

We are sorry you feel that we only favor Japanese products. But I don't think we have any special favoritism to Japanese designed and/or made products. In fact, we did an in depth article on the Parasound Halo C 1 and C 2 and the review was very positive. We encourge you and others to read this review and see that we did our best to find ways to work around some of the missing elements on these aging products because we felt they were worthwhile products to consider in a vast, rapidly changing, highly competitive marketplace.

Parasound Halo C 1 and C 2 Review (http://www.audioholics.com/productreviews/avhardware/ParasoundHaloReview.html)

If memory serves, the only non-Japanese Processor we hit pretty hard was the Sunfire Theater Grand III. Our review of that product was indeed critical, but given it's pricetag, we simply felt it had too many shortcomings to be competitive. You're welcomed to read that review as well.

Sunfire Theater Grand III (http://www.audioholics.com/productreviews/avhardware/SunfireTheaterGrandIIIP1.html)

One thing you may want to consider is that we do try to be objective when possible while also supporting our subjective comments with facts and/or experience. This is the very reason why we provide measurements when possible and why we created an AV Processor/Receiver Checklist.

AV Processor/Receiver Checklist (http://www.audioholics.com/productreviews/avhardware/AVprocessor_checklist_v01.html)

This list was completed after careful consideration and thought combined with input from many people in the field from companies in America, Sweeden, Japan and other places. We tried to compile this list by listing product offerings and assigning them a logical score based on what we felt was critical, important, or bonus features. We then use this table to score the products under review. I'm not sure how much more objective we can be beyond this as all Processor or Receivers are placed under the same scoring process. In fact, this is an adaptive score card as each year there new offerings that are added to the list.

As for our measurements, we have a growing list of test equipment that we use in speaker analysis, cable analysis, video analysis, amplifier analysis and other analysis of many products. This growing list of equipment can be viewed in our evolving table.

Audioholics Test Equipment (http://www.audioholics.com/techtips/setup/Setupcalibration.php)

We even completed several training classes including ISF, HAA, RF, and several others, all in an effort to gain more knowledge to provide better and more complete reviews.

Now grant it reviews can be subjective too, including ours. That's just the nature of a review and why we are not just a test lab. We do our best to support our subjective writings with objective facts as often as possible and as best we can. But even magazine publishers author articles that are subjective and may be considered biased by some. I don't think there's a steadfast solution to this misconception, but we're willing to listen. Much of our subjective views are based on our experience with tons of products, and typically not from the seat of our pants.

Thanks for continuing to visit Audioholics and discussing in our forums. We do hope you continue to provide valuable input and comments. But we ask that you consider not being as abrasive to us when you disagree as we find that it's counter productive. We always value and welcome constructive criticism as it helps us improve our site and services. There are time however, that we get a bit taken back by argumentive comments in place of critical evaluations. Perhaps this is due to the fact that we are all Engineers and not Philosophers.

We do wish for you to continue to provide valuable feedback and comments, but please be a bit more tactful in your approach as it will go a long way with us.

[Dan Banquer]

As one who has had his company (www.redesignsaudio.com) reviewed by a number of different magazines I can tell you that it is not uncommon for reviewers to show the review before it is published for technical editing/comments. Since many reviewers are technically illiterate they need to get certain facts straight. I do not get to comment on any of their subjective impressions.
This web site is anything but technically illiterate and the purpose of showing the review on a technical level is to allow the company to address the problems the staff here has found. Note that many of the products reviewed here may need some software changes which the manufacturer was initially not aware of. This results in progress, not animosity if the parties involved are rational.
Dan Banquer
R.E. Designs

[RLA]

Hi All
I see nothing wrong with sending the review to Yamaha before publication
There is a distinct difference between reviewing a performance (Movie)
and reviewing a very sophisticated piece of electronic equipment
proof reading for technical correction or user/reviewer set up would be the main reason for the proof read

[Westrock2000]

It also gives the reviewer time (or the option) to include a rebutle from the manufacturer. Whether that rebutle be simply thanking the reviewer for a positive review or trying to explain aspects that were frowned upon. Although not particulary common in reviews, I think its a good addition because it allows us to see how the manufacturer feels and/or their attitude.


I thought this was going to be something like posting a review based on second hand experience, or a complete lack of faith in a product for the wrong reasons. Like saying that a speaker will suck simply becuase it has a vented bass driver, or that one reciever is better than the other based on wattage.....thats unethical reviewing.

[Westrock2000]

Heh, one thing that gets me about some reviews in magazines is that of speakers. Have you ever noticed that sometimes the domes on the tweaters are pushed in on the sample picture? Whats up with that? Are reviewers that careless? Did they recieve them like that? It just looks plain bad for everybody. Becuase then you wonder if they listened to them like that (although many claim it affects are nill), or if the company has bad Quality Control.

[Physique]

Dear Av Phile,

As an engineer who used to head the testing in the Technology department of a top Networks company, I can tell you that the technology industry is a very different one from the film industry. Part of my job was to study different Internet connection technologies and develop standardized methods of testing for each one. Another part of my job involved reviewing each of these technologies. In both instances, I always contacted the manufacturer of the product/technology under test after my test report was finilized as this accomplishes two purposes:

1) It gives the manufacturer an opportunity to validate the test setup. Sometimes faulty logic in test method or incorrect firmware/software loaded in the product may have been the reason why a result did not come out as expected.
2) It provides the manufacturer with good feedback on their product if the findings are indeed valid.

There is nothing unethical about this. As a matter of fact, as Gene said himself, this is a professional courtesy if nothing else, and serves to preserve the validity of the review.

Comparing the technology industry with the film industry is like comparing an apple to an orange. A film review, does not need to be validated as it is subjective in nature. While it can be argued that parts of a technology review can be subjective as well, if you wish to make a strong case for your opinions, your test setup and technology/product facts better be straight or else your case will fall faster than water at the Niagara Falls. Ergo, the reason for a peer review by the manufacturer of the product in question. What you are doing here is just setting up a classic strawman's argument and at such a contest you stand at a disadvantage.

Best of Health,
Hugo A. Rivera
President
HR Fitness Inc.
http://www.hrfit.net

[zipper]

I enjoy reading the reviews on this site. I find that they are objective,in-depth,& professional. When there are criticisms of a product,they are done tactfully & without malice. Thank you Gene,Steve, & Clint. You provide an avenue for less experienced audio enthusiasts such as myself to gather ideas & information without having to wade through a ton of BS or egotism.
Just about all the regulars here have more experience & knowledge of products than I do,so I thank them for sharing.This includes you AV_phile. But I don't understand why you're so negative. Yamahaluver is obviously pro-Yamaha. Just let it be. You could change your name to Roteluver & pump Rotel. I got no problem with that. But you seem to ***** for the sake of *****ing. IMHO,you have much to offer on a site like this but remember that,like the rest of us,we are guests here.We don't own it. Lets not denigrate it or it's owners for trying to provide a free service & platform for sharing.

[Rip Van Woofer]

Okay, let's get down to the REAL question:

How much are Gene's reviews influenced by his brother Steve's homemade mead?? :eek:

[Dan Banquer]

"How much are Gene's reviews influenced by his brother Steve's homemade mead?? "
Good Question! Enquiring minds want to know! :D
Film at 11?
d.b.

[gene]

It's not the wine thats getting to me, its sleep deprivation. Damn I need a vacation or more hired help :eek:

[Rip Van Woofer]

You know what they say about self-employment and small business, Gene: you work 16 hours a day for yourself so you won't have to work 8 hours a day for someone else!

[gene]

Or you work 8hrs/day for someone else and 16hrs/day for yourself at the same time. I am hoping to eliminate the prior very soon! I don't mind working around the clock for myself, especially since I love doing this :)

[Yamahaluver]

Or you work 8hrs/day for someone else and 16hrs/day for yourself at the same time. I am hoping to eliminate the prior very soon! I don't mind working around the clock for myself, especially since I love doing this :)

Gave up working for others long time back, most unrewarding and constant nagging made me switch to self employment and I have never looked back.

[gene]

Yamahaluver; I hear ya. I don't mind the extra work involved in working for yourself and the many hats to have to wear to be successful in it. It beats sitting in a Dilbert cube doing work you hate and answering to someone you would rather have no affiliations with whatsoever. I love working at my own pace and setting my own priorties and being in total control of my own future. Although more risky, it is much more enjoyable to me.

[Clint DeBoer]

This is what Gene does every day... :D

[Dan Banquer]

You mean Gene has some software weenie measure cables? :eek: OH boy we're in real trouble here folks. Does the Wayne kerr Analyzer have a fat screen? :rolleyes:
d.b.

[av_phile]

After about a week-long sabbatical, I am not entirely surprised at the responses to my question. All your points have been well received.

It may be the policy of this forum to do so as its notion of courtesy dictates. I have no quarel with that. You've made your point. Now let me make mine.

Some have opined that a film review is different from a product review which is done for very technical products like the equipment we use in our hobby. But that is entirely beside the point. The objective of a review is the same, regardless of what is being reviewed. And that is to enlighten the consumer so he/she can make a more informed decision or choice. Doing so requires validating the claims of a manufacturer made on its spec sheet that is available to consumers. A reviewer works to test those claims.

To say that equipment reviews are often done by "technically illiterate" people is a disservice to consumers. Letting the manufacturers see the result of a review in order to check on the veracity of the results made by these "illiterate" reviewers who are not confident of their test results smacks of irresponsibility. Why allow a review to be done by incompetents?

But I have to ask, why do you have to have the manufacturer check on the results of your test? Are you not sure? What facts do you have to check with them AFTER the tests are done? Shouldn't you have done this BEFORE?

I submit that it is the responsibility of the reviewer to get his facts straight PRIOR to making a review. At this stage, a reviewer can liaise with the manufacturers to straighten his facts as part of what is called due dilligence. And he should ethically advice the manufacturer of his testing methodology and all the software and measuring hardware he intends to use in his test.

But once the testing is done to validate manufacturer claims, that's it. The knowledge and technical expertise of the reviewer bears down on how he conducts the test on the supplied data from the manufacturer. A reviewer is testing the Data supplied by the manufacturer which should be no different as what is made available to the public. Nothing more. And, because a reviewer knows what he is doing or not doing, stands by the results of that test for the benefit of the consumer. I don't see the point why any part of his review needs to be shown to the manufacturer after the test/review is made. The manufacturer can see it in published form and does whatever correction is needed to better the product. And in many magazines publishing such reviews, I often encounter published retorts from the manufacturers themsevles arguing an aspect of the test results. But the publishers/reviewever often stood their ground standing pat on their testing methodology.

The goal behind testing a product so that consumers are not led astray by their makers is simple: To validate manufacturer claims. When you validate a claim, it is like GRADING a student's test results (the claims a student makes on how well he has learned) It carries a degree of FINALITY that cannot and must not be altered by the subject of the test. To say that other reviewers do so fails to convince me that the practice is sound. Just because many are doing it doesn't consitute a valid justification. Perhaps within the ambit of your daily work, you do find them. But not in mine. I have previsously worked with film ratings and consumer advocacy groups and I don't recall submitting test results to manufacturers prior to publication. The reviewers I know submit their test methodology and the product data claims they will test PRIOR to testing. And if they hear nothing from the manufacturer after a certain date, they proceed with the test all the way to publication.

So I may be a odd man out decrying this. Fine. I am merely pointing out my obvious disagreement with the practice from my own viewpoint. And while I am very open to opposing views that can change my opinion on the matter or prove me wrong, so far I haven't read any that will. But like what Gene said, i've "vented my steam, nuff said."

[gene]

AV_Phile;

Hmm I wonder if it takes a film critic 40++ hours to review a film, and research the details of how the film was made, or do they just watch the film and voice their opinion in a cursory summary?

Obviously you have little to no experience dealing with hardware manufactures and the channels one has to go through (especially with the larger companies) to get technical details. Most of the time we usually don't have design schematics so we take our best guess based on logic, experience, and feeback (if any) from manufacturers.

Maybe you should look elsewhere for your product reviews that will better suit your alleged needs. I certainly don't want readers whom think our review practices are unfair to feel "led astray". Perhaps cursory reviews found in other places will make you feel more comfortable.

[zipper]

What if the product in question was rattled during shipping & is now somewhat defective? I'm sure the manufacturer sent the tester a piece of gear that they were confident was good but if said gear was no longer functioning as intended wouldn't the tester want to verify this with the manufacturer before submitting his review for publishing? How embarrassing would it be to call something sub-standard(or worse) only to find out it was defective?
Audioholics did a speaker face-off for HT. I can't recall the brand name(atom or something) but the reviewer thought so poorly of them that he talked to a rep from that company who gave some BS excuse about setup or need for the matching sub to be used. AFTER giving them a chance for rebuttal or correction,it was published( in a courteous manner) that these were overpriced junk.
Face it,professional courtesy courts professional courtesy.If some clown continually thrashes a variety of gear in his reviews w/o checking to make sure he's got his facts straight,how long do you think it will take before he's got to BUY EVERYTHING he wishes to review?
I don't believe there is anyone in bed with anyone here.If Audioholics maintains their professionalism,I'm sure their reputation will afford them the opportunity to review more & more products,which will be a benefit for all of us.

[gene]

I don't believe there is anyone in bed with anyone here.

Zipper, my wife often has this complaint since I am usually up late at night working on reviews :rolleyes:

[surveyor]

Zipper, my wife often has this complaint since I am usually up late at night working on reviews

Gene, I volunteer to use your test system so you can get some rest.
MuHaHa

Cheers :D

[Unregistered]

AV_Phile;

Hmm I wonder if it takes a film critic 40++ hours to review a film, and research the details of how the film was made, or do they just watch the film and voice their opinion in a cursory summary?

Obviously you have little to no experience dealing with hardware manufactures and the channels one has to go through (especially with the larger companies) to get technical details. Most of the time we usually don't have design schematics so we take our best guess based on logic, experience, and feeback (if any) from manufacturers.

Maybe you should look elsewhere for your product reviews that will better suit your alleged needs. I certainly don't want readers whom think our review practices are unfair to feel "led astray". Perhaps cursory reviews found in other places will make you feel more comfortable.

You may have missed my point. I couldn't care less if it was 10 second review or a 10 year review. the objective remains the same. and the ethics behind it doesnt' change.

The more complex the product that has to be reviewed the more due diligence is needed to ascertain the facts about it PRIOR to any formal test is made. NOT AFTER the test. That's all the point I was making.

Other than the reviews you made on cables, I do look elsewhere for the product reviews I want. Certainly not here. Especially after learning about your policies. I had the impression this forum was about pursuing the truth in audio. I sincerely hope that is still the case.

[av_phile]

ooops, forgot to log on. that's me above.

[Clint DeBoer]

OK, we get your point and appreciate your views. Thanks for voicing your opinion.

surveyor - if you want to come up here and do the next 3 product reviews I've got scheduled for this month, my wife and I could use a vacation! :D

This is going to be a very productive month I think for product reviews... lots of fun stuff coming up!

[Yamahaluver]

Zipper, my wife often has this complaint since I am usually up late at night working on reviews :rolleyes:

As my name suggests, my wife clearly knows I sleep with Yamaha.
:p

[surveyor]

Clint, I could not touch you guys!
If I could, my wife would not allow me to be around all that fine gear that you guys use ($$$).
Remember that I work in the DSP industry (tiny income).
Please keep up the fine reviews and continue with your straight forward ethics!

Cheers :cool:

[gene]

Other than the reviews you made on cables, I do look elsewhere for the product reviews I want. Certainly not here. Especially after learning about your policies. I had the impression this forum was about pursuing the truth in audio. I sincerely hope that is still the case.

AV_Phile, we apply the same policies to the cable companies whose products we review or write about. So following your logic, you may wish to look elsewhere for cable reviews as well.

[surveyor]

As my name suggests, my wife clearly knows I sleep with Yamaha.
:p

I prefer the DAC's on my older DSP A1 compared to my Z-1 and Z-9.

Carefull with how many Yammies you sleep with you could get an AVR transmitted disease!

MuHaHa :o