The Pioneer's "Auto PCM" and "Digital", in the case of your signal from your CD player, do the same thing. Your CD player outputs PCM, so that's what is being handled regardless of how the selection is made. Even in terms of how the player is connected, coax or optical, it's still just PCM, and once it's in the AVR, it's all handled identically, which is to be internally up-sampled to 24/96 and processed through the SHARC DSP.
Analog is an interesting situation in this case, if you're actually playing an HDCD. If so, the internal processing of the Rotel is necessary to properly decode the HDCD and get the benefit of that format. The only way that's coming out of the Rotel is the analog jacks. Now that's not a real problem, because your Pioneer AVR will re-digitize that signal at 24/96 (but that performs more like an optimistic 20/96) which should be a fairly good match to the 18 bit HDCD performance (as your Rotel manual states), especially considering HDCD's "Peak Extend" function. But, all that considered, there still be not much to be gained from HDCD > Analog > Pioneer. The reason I say the Pioneer, even though it's spec'd at 24/96 for analog inputs to its ADC, there are no inexpensive real 24/96 ADCs in the world. Most barely make it to 20 bits, with many performing up to 18 bits. A real 24 bit-performance ACD is a beastly expensive device that wouldn't be found in an AVR's rarely-used analog inputs.
So what you've got is an HDCD with an optimistic 18 bits at 44.1 being sent via analog to an ADC running at barely 20/96 performance. The end result will likely be closer to 17/44.1 performance after all of that, and now you're pretty much close to a high-end standard CD player with no HDCD.
Anyway, the short story is, for HDCD playback you should use the analog outputs/inputs. For standard CDs, the coaxial connection (PCM) would be better, slightly.