The sub out on the dvd player is for the LFE channel when using the 5.1 analog outputs connected to the receiver's 5.1 analog inputs. You would only use it if you use the 5.1 analog outs and have the dvd player set to do the decoding ('PCM' setting in the menu for DD/DTS). You would also use those analog outs if the player is actually a universal player that can play DVD-A and/or SACD discs.
If you use the optical cable to connect to your receiver and have the dvd player menu set to 'bitstream' for DD/DTS then you will get the full 5.1 (for DD/DTS discs - not for DVD-A or SACD).