If your Xonar is doing the decoding, the Onkyo isn't going to be in a mode. It will just detect the incoming signal and output the channels accordingly.
For example, if it detects a stereo signal, it'll either go to Pro Logic or Stereo, depending on what you want it to do. To me, that setting is your preference.
If it detects 7 channels of audio coming in, that's what it's going to output. The decoder won't be doing anything to the incoming signal.