You are doing things incorrectly.
The red/white connectors next to the HDMI input is for analog left/right INPUT into the TV, not audio output! This is in case you are using a DVI video source, like an older Oppo DVD player. Great digital video, but audio is not carried over DVI, so you have to put it in over analog audio connections.
With the PS3, you want to set your video to HDMI and your audio to A/V multi. This should give you your video as you expect it and then deliver audio over the red/white (and now worthless yellow) connector that comes out of the back of the PS3. Connect the RCA red/white cables from the PS3 directly to your TV and then loop the TV's audio out jack back to your A/V receiver (if your TV has audio out). If the TV does not have audio out, then you should take the PS3 RCA red/white analog audio jacks straight to your A/V receiver and use your A/V receiver all the time for listening to audio.