HDMI cables carry both digital video and audio. The fiber optic cables carry digital audio only - but they can't handle the new lossless audio codecs like HDMI can.
So, HDMI is better, but some receivers can't process the audio coming over an HDMI connection. Your RX-V663 can, though, so you don't need to use the fiber optic connection from the PS3 in addition to the HDMI connection - just use the HDMI connection.