Cables do not cause jitter, period - they are a conduit for sending binary data either electrically over coax or optically over optical cables. You can have interface jitter where there is a small timing error (jitter is a timing error) between reading/preparing the data and sending it to the interface to be transmitted to its destination but no cable can either improve or worsen that. Besides, it is so vanishingly small that it is inaudible - usually on the order of picoseconds.
Coax vs Optical is simply a matter of trade-offs:
1. Coax is more sturdy and RCA connectors fit more tightly, but it could be subject to EMI/RF interference. Any reasonably well made cable will have adequate shielding to make interference another practical non-issue.
2. Optical cables are immune to electrical interference because it is pulses of light, but the toslink connectors are small and fragile and if the cable is bent too far or obviously crimped, it will affect the data stream. Don't bend or break it and it will work flawlessly.
In HT applications, they are interchangeable and each performs as good as the other.