Audio Quality
Audio quality-wise, you might be able to tell a difference between the two, and you might not - it may just come down to the convenience of running one cable instead of six.
The 5.1 connections are analog outputs from the DVD player, while the digital coax is a (I know it's obvious from the name) a digital connection. The material on the DVD is digital to begin with. The signal is converted to analog before being amplified and sent to your speakers. You have at least two choices as to where to do the digital-to-analog conversion: in the DVD player, or in the receiver. In theory, you'd want to use the one that gave you the best end result - which is a combination of the D/A converter being used in each unit and any potential influence of the signal over the analog lines coming from the DVD player to the receiver.
The above is just meant to be some explanation. In practice, you would just connect it both ways and see which one you like better. You should be able to connect it both ways and switch between them using your receiver/DVD menus.
Features
With some equipment, you need the DVD player to convert the material. For example, my DVD player will only output DVD-Audio material over the 5.1 analog outputs. I used to have those connected, but I don't ever listen to DVD-Audio, so I'm just using the digital connection now.
