In a book I read by Rodger Driscoll, who has quite an impressive CV (lecturer, reviewer for Gramophone, consultant for Philips) 'Practical Hi-fi Sound', the major limiting factor are the speakers in most hi-fi systems.
Certainly things sound much better through headphones on my system, even though I use reasonable budget speakers (B&W DM601's).
This would lead me to believe that the CD player component is not the limiting factor in a hi-fi system, unless you're using speakers of distortion <0.1% (which maybe electrostatic speakers are capable of?). This is assuming that CD players do not introduce some distortion that cannot be measured, which of course would change things, and maybe explain why more expensive CD players are thought to sound better by some people.