Audio and Video are separate and unrelated concepts. The fact that you are using an HDMI->DVI adapter has nothing to do with 7.1 vs 5.1. Optical (s/pdif) connnections cannot support uncompressed 7.1 audio, the type you'd get from a Blu-ray disc (the bandwidth exceeds the capabality of the interface).
You will not get the 7.1 PCM signal from a Blu-ray (PS3) without using HDMI and a receiver that is capable of dealing with 7.1 channels of uncompressed PCM audio.
There is no bandwidth limitation that prevents 7.1 from going over optical. It's just that the newer high resolution audio codecs and PCM greater than 2 channels are not part of the specs for optical and coax.
To the OP, the only way to get 7.1 into this receiver is to use a Blu-Ray player that has 7.1 analog audio outputs (e.g., the Samsung BD-UP5000, the Panasonic DMP-BD10/30/50, the Denon 3800BCDI). The other alternative is to keep your PS3 and get a receiver that accepts 7.1 PCM over HDMI like the Onkyo TX-SR604,605,606,705,805,875,905, etc. (the TX-SR5xx are pass-through only) and the Yammy RX-V663. You can find good deals on the TX-SR604 and 605, but those have been replaced by the 606.