The first thing obviously to do is connect the receiver and PS3 via both HDMI and optical. On your receiver, you need to assign the HDMI input and optical (digital input) to the source you want (page 22 of the manual). That should allow you to get both video and sound to the correct input on your receiver.
On the PS3, you need to go into the setup menu and select video output as HDMI and turn on the optical output. The video instructions are on page 18 of the manual. Basically you go to (Settings), then (Display Settings), then [Video Output
Settings] and follow the prompts. For the audio, there should be a prompt at the end of the video setup that asks you what connection you are using. Choose Digital Out (optical).
I'd suggest connecting the PS3 directly to the TV to do its part of the setup. Once that's done, plug it into the receiver to see if you set the receiver up right. Hope this helps!