Seeing as you have a 5.1 speaker system, I'd choose Dolby Digital. Dolby Pro-Logic II would be the one to choose if your Yamaha receiver didn't process Dolby Digital 5.1.
Dolby Pro-Logic II can have up to four channels of sound - surround, front centre, and front left and right. Dolby Digital is more advanced and can have up to seven channels of sound - left, right, and centre surround, front centre, front left and right, and a low frequency effects channel.
Dolby Pro-Logic II is under the analogue section because it is not digitally coded when it is sent from the XBox.