I have the AV8802 back Fresh with the AV8802(A) upgrade. I have the GE Triton One Speakers. Not knowing much about the Yamaha CX5100, I can't give you a very helpful answer at this point.
DACs are different: AK4490 32-bit DACs for the Marantz
ESSES9016 on Yamaha (Save as the Previous Geneartion 5000)
What does that mean? I don't have a clue. I will have to defer to someone who has a much greater grasp of that piece. Looks like the both have all the same codecs. At the end of the day I am sure both will sound excellent, and I doubt very highly you will be able to hear any noticeable difference between the two.
That being said, the List Price Difference is $500 not 1K $4999 vs 4499. I paid $2500.00+Tax for my AV8802 from an authorized dealer who quoted me the wrong priced and honored it. This was before they had the HDCP 2.2. I paid $60.00 to ship to Marantz and just got it back last week. They even put an 'A' Sticker on my box. I will say the build quality is superb and that I did notice a difference between that and my Integra 80.3 (Perceived I know) on music. I did no measurements to quantify this. I have had 0 problems on the software side of things and it is quiet as a mouse with mittens with my Halo A51s and A21.
I have not used a Yamaha since the RXV-2000 or 2100. I really liked Yamaha, but it was well before they had YPAO I think.
I do know that I am addicted to Audyssey and tweeking it out with my room setup. I do not know enough about Yamaha's room correction to make a direct comparison, but Yamaha has never been one to slouch on features.
I will run mine in full Dolby Atmos, only because I built my room from scratch. 7.4.2.
PENG stated it correctly, I believe Marantz can get away with higher pricing because of brand loyalty. In my case, Price was excellent and all things being equal or slightly more expensive I would go with Marantz again, just because I have become fairly proficient it setup and room configuration and have no desire to learn something new with room correction lol. I think back to the AV7705 and it really killed everyone back in they day for Price/Performance for a pre/pro. Their Flagship has always commanded a really high cost.