Sounds like the amp isn't too happy with that arangement...
...and is expressing it's displeasure in the only way it knows how. Here's a few possibilities to keep in mind.
Possibility 1. Generally, if a receiver doesn't have a built in a/b switch, it isn't designed to handle more than one set of speakers.
Possibility 2. When a signal is sent to two speakers, the available power is split between the two speakers.
Possibility 3. That the A/B switch you're using probably connects both sets in parallel, which causes the "impedance", or resistance seen by the amp to be halved. Many amps don't play well with low impedance loads. See possibility 1.
Possibility 4. Consider yourself lucky that your receiver only distorts and loses volume. Generally, maintaing any of the three possibilities stated above could cause your receiver to cease to function.
You may want to rethink the A/B switch.