Make sure the switch on your sub is set to on instead of auto, set the gain to roughly 50%, and run the YPAO again.
What sound profile/decoder is the Yamaha using when you're playing media (ie. Pro Logic, THX, 2-ch stereo, etc.)?
EDIT - also, any low pass filters on your sub should be disabled, let the 663 set your crossover point. This sounds more like a sub problem than a receiver problem.