The A-S801 is an AVR based integrated amp, not a real thing in my book. It does have a more powerful amp section with a larger power supply that is more suited to 4 ohm speakers (not much a factor though if you sit close enough). Imo, many got sucked into it based on Yamaha's excellent marketing material, forum hearsay/review, plus perhaps to some extent, the good reviews on its predecessor A-S700, that is what I would consider a real integrated amp. That is, one that is not based on AVRs.
You Monitor Audio gold bookshelf can only handle 120 W right, and that's 4 ohms so the RX-A870 should still be adequate for the job if you are sitting close enough to the speakers. Your photo seems to show a bed room system, but what is the seating distance anyway?
I highly doubt it could sound really better the way you connect them together even if it is a power output related issue. If it is not about power, then I have no idea why/how it could do better with two preamps in the signal path.
If you want to use an amp with an AVR you should stick to a power amp, or an integrated amp that has the HT bypass feature as other have mentioned. Please also note that Yamaha AVRs, at least the ones below the RX-A1080, typically don't do well as preamp because of their relative high distortions when the output voltage exceeds 1 to 1.2 V. Gene mentioned something to that effect in his review of the RX-A860.