Welcome to the forum!
Have you assigned the digital input that you connected to on the back of the Denon to the source selection that you are using? For example, assigning OPT1 to "DVD". If you haven't, check out page 41 of the owner's manual.
Also, have you set up the PS3 to output audio through the optical output?
If you have, just let us know.