Phono and line level RCA aren't the same thing. This player does not have phono outputs; it has regular line level RCA outputs. Which analog inputs did you use on the 770? The assignment for that input selection (CD for example) to use analog is likely buried somewhere deep in the menus, making it not obvious, as I have experienced with my Marantz. Does the Denon have a "pure direct" mode or something similar that would force analog?
*edit - Looked at your model. It shows two analog inputs as assignable. That is what you should connect to and you will need to assign it to the input selection you want to use. It will not work with the Phono input.