On most receivers, using A+B puts the two sets of speakers in parallel which lowers the impedance seen by the amplifier.
Because you are using a speaker level hook-up the impedance of the sub comes into play and the sub's impedance is going to be very low (likely lower than 4 ohms). If the front speakers on the A terminal are 8 ohms and the sub is 4 ohms, you get 2.66 ohms (1 / (1/8+1/4)) which is nearly impossible for a receiver to drive adequately. When you used the surround speakers with a much higher impedance, the total impedance was lowered but probably still above 4 ohms and the receiver could handle it.
That is a pretty odd sub if it has both low-level and high-level inputs but only low-level (RCA) outputs. Typically you would connect the speaker wires from the front A terminals to the sub and then speaker wires from the sub to the front speakers.
The possible solutions I see:
1. Run the speaker wire from the A terminals to the sub and then run cables from the RCA outs of the sub to an external amp and connect the front speakers to that amp.
2. Run the front speakers and sub in series from the A terminals. Someone recently posted a diagram of series hookup and I will search for that and post the link rather than explain it in words.
3. Buy a new sub that has speaker level outputs.