Welcome to the forum!
I couldn't find much info on those two products, so what I did find might be faulty. Assuming that what I found was correct (which differs slightly from what you stated), here's how I would go about connecting things up:
1. Connect the TV's digital audio output (digital coax, which looks like an RCA jack) to the RCA's coaxial digital audio input.
2. Connect the PS3's optical digital audio output to the RCA's optical digital audio input.
You might need to set up the TV to output sound over the digital audio output. Also, I don't know which input selections on the RCA are associated with the digital audio inputs, so you might need to flip through them (TV, VCR, etc.) to see if any of them give you sound.
Adam