I struggle with understanding exactly how this system was wired.
One channel of an amplifier is designed to power ONE SPEAKER.
By the accounts, it sounds like multiple speakers are being driven off of a single channel of the amplifier. It sounds like a 'stereo zone' is being wired off a single channel setup of the amplifier. It sounds like the A&B channels of the amplifier are being used to drive 9 speakers... instead of 2 or 4, like it was designed for.
I think the wiring to the speaker selectors (16/4) is fine. I think the wiring to the speakers (16/4) is potentially correct as long as it's one pair of wiring per speaker.
I think that the wiring back at the amplifier may not be correct and that this will lead to multiple problems down the road.
I would STRONGLY recommend that instead of a 2 channel amp and a SIX ZONE speaker selector (which is what should be used at a minimum) that you instead use a 12 channel amplifier from a quality manufacturers like Sonance, Speakercraft, or Niles.
ie:
https://www.ebay.com/itm/Sonance-Sonamp-1230-12-Channel-30-WPC-Whole-House-Amplifier-Tested/313388463681?hash=item48f7687641:g:cTQAAOSwjJVgCENf
An amplifier like this will give you 12 total channels of amplification which you will need for your NINE separate speakers.
The 'Master', makes zero sense as that's a single mono speaker. This isn't something I ever recommend, especially in more budget oriented systems. Everything else is stereo, so how did you really think this 'one' speaker was going to work? It makes no sense.
Still, there are solutions and options. Realistically, unless you are keen on stereo everywhere, I would run the whole system as mono and just get good audio to every speaker with independent amplification. Make sure the amplifier has good airflow and isn't stacked in with 30 other pieces of electronics shoved into a drawer, which will kill everything. COOLING MATTERS!
While I think the foundation is decent, I think the implementation leaves a lot on the table and should be considered as the most likely culprit for your woes.
Remember, you have a 2-channel amplifier right now, don't overdrive it!