Hi...
i have an older Arcam 137 dvd player that i was going to use only for CD playback but i noticed i have several audio output options (going to a marantz AV7704). I have optical, stereo out (which i assume is analog?) and HDMI so which one to use?
Here's the deal, your RCA stereo connections are analog. It has been used quite successfully for years and I find it hard to believe that people are routinely hearing problems with this format. Consider that the cables musicians use to plug their instruments into amps are still analog. The exception is if you are in a very electromagnetically noisy environment. If you run and tie wrap your RCA cables alongside your speaker or AC power cables you will have the type of noisy environment, but as long as you keep them a few inches from each other, you should be fine!
Digital (which includes HDMI and optical) was developed because, by encoding the analog signal into an on-off (0's and 1's),you are no longer concerned about the magnitude of the signal. In other words, say you have a abnormally long run of cable, an analog signal might lose 2% of it's strength over that distance. That would mean it would reduce the SPL of the music by 2% effectively reducing the dynamic range. However, in the land of 0's and 1's, a 0 reduced by 2% (or increased by 2% from external RFI is still recognized as a zero (or off),and a 1 reduced by 2% is still clearly a 1 (or "on") for a system designed to recognize only 0's or 1's (in other words it effectively rounds off the magnitude to either 1 or 0 whichever is closest to to value received - as you can imagine it would take substantial interference to cause a 50% or greater error in magnitude).
So, digital is essentially a noise reduction technology when you are talking cable choice (obviously, it is also a recording technology and a medium in which you can more precisely alter the signal, but that does not effect your choice of cable format).
So TL;DR:
All should work perfectly (without an audible difference) in a normal environment!
However, because poop happens, I am inclined to use digital because I could accidentally end up with low-level (RCA) and high-level (speaker or AC) cables next to one another.
Between HDMI and TosLink optical, I would go optical as long as the bandwidth of your information can be carried. I like optical because there is only one signal (one strand of "clear fishing line" in the cable) which makes it a very simple cable that either works or doesn't. Simple is almost always better from a reliability standpoint. Optical is also immune to Ground Faults!
HDMI has multiple (maybe 18?) signals/wires with terminations to sockets at both ends. Add the stresses of flexing this bundle of metal wires and my ignorance of what information is carried among the different wires, I don't think it is as reliable and I worry that I could get a short or intermittent connection in one of these wires that would be hard to troubleshoot.
On the question of bandwidth, I believe it is a matter that 18 electrical conduits/wires can carry more than one optical conduit/strand.