I would bet it has to do with settings in the bluray player, firstly. See if there are settings to choose SPDIF/optical for output, and set Dolby/DTS to bitstream.
Some bluray discs might come defaulted as 2.0 PCM, if rarely. If you select something like uncompressed PCM on the disc's audio choices, you might be getting 2.0 PCM there (downmix), but I'm not sure if that is necessarily the case. (Optical can carry only up to 2.0 in uncompressed PCM).
Therefore, to repeat, make sure the player is bitstreaming, and choose the correct codec on the disc.
DTS MA will work just fine, since the "core" bitstream will pass thru the optical.