Only 2 channel PCM can be passed via SPDIF (Toslink or Coaxial). In order to receive 5.1 surround sound you must set the PS3's Dolby Digital and DTS bitstream outputs to on. I would deactivate Dolby Digital Plus and Dolby TrueHD if possible, as you will not be able to use it with your current setup.
The only things that SPDIF can accept are bitstream audio from Dolby 2 channel, Dolby Digital 5.1, Dolby EX 6.1, DTS 5.1, DTS-ES 6.1 matrix and discrete, DTS 96/24, and 2 channel PCM. 7.1 LPCM and new age surround formats Dolby Digital Plus, Dolby Digital TrueHD, and DTS-HD Master audio cannot be supported via SPDIF due to copyright protection.
So know that there is no problem with the receiver, it is performing as it should.
(HDMI 1.1 and 1.2 can support 7.1 LPCM and HDMI 1.3 can support Dolby Digital Plus, Dolby TrueHD, DTS-HD master audio, 7.1 LPCM. In order for these new audio formats and 7.1 LPCM to be decoded on a receiver the HDMI hardware on the receiver should be the repeating type and not pass-thru.)