First and foremost, I completely agree that it seems to be an oxymoron. But the confusion may arrise from the context of specifically which part of the test is subjective, and which specific definition of subjective may be used in the context(subjective has several definitions).
Perhaps some confusion arises from what is known as the age-old stereotype of what subjective is usually asociated with in audio: purely uncontrolled perceptions, probably the Forer effect in many cases. But a subjective evaluation can be performed with controls in place, to limit the extent of subjective response or what influences such. The use of this term could also be used in a capacity meaning to create a subjective experience, as in subjecting person to particular samples; subjected(submitted) to a certain stimulus.
Not entirely true. You can quantify preferences, through statistical value, via a controlled test environment in order to find a pattern of (subjective) reactions to a controlled experience. The scientific component rests in the elimination of external factors/variables to create a properly structured experiment, that shall analyze only the intended factors.
For example: You could find out which two experimental soda formulas were preferred by submiting a randomized blind test to many subjects. The actual act of tasting and preference is subjective, but all external factors would be eliminated leaving only responses based only upon the taste of the product. The statistical score of the completed tests would demonstrate a percentile quantification of preference for each formula, within the narrowly defined conditions of the test scenario(s).
The issue is not to measure perception, as a quantification. It is to find a preference in this case. There is no reason that this could not be accomplished, if the proper protocols were put into place. One could set up an experiment similar to the bandwidth test, but compare direct phonograph play to phonograph routed through a PCM 44.1kHz/16 bit ADC-->DAC convertor with very low latency. A standard random A B vs X experiment could be used to switch between the two for the blind testing. The ADC-->DAC convertor, of course, would be measured and ensured to be working under known human thresholds for the relevant parameters(THD, level matched, frequency response, etc.). If with statistically significant results, the subject or subjects could reliably identify analog from digital processed version, then it would demonstrate strong probability of difference(tenative, and dependant on peer review/critical review of the test procedures). The test could also be expanded to include a form/questionare component, that when applied under these blinded/controlled conditions, could be used to see if a reliable/consistant response could be achieved. The actual judgement/evaluation on the human interaction level would be considered subjective, but yet this would be a scientifically valid experiment due to the established controls/experimental structure/statistical power. In this experiment, the subjects would have no reasonable probability of cheating or being affected by unintended/external stimuli. Of course, if the test ended up with null results(this is probable, based on current research and knowledge of human auditory senses), then you could argue that it was not a subjective test since preference never entered the equation; i.e.; no audible difference was detected thus no valid subjective opinons would be present on the questionare portion of the blind trials.
-Chris
