'Composite' refers to video, which would be color coded yellow if the cable uses the standard color coding. As highfigh said, the red and white colored cables and jacks are simply analog audio.
If you use the analog cables, the receiver will see a 2 channel analog signal and the only way to get "5.1", regardless of whether the source format was DD, DTS, or even 2 channel PCM (like from a CD), is to use a matrix surround decoder in the receiver, such as ProLogic II, DTS Neo:6, etc. The DD/DTS light on the receiver will not illuminate because it is not receiving DD/DTS - it is receiving 2 channel analog because the DVD player decoded those formats to 2 channel analog before the receiver ever sees it.