Het, my guess is one of the subs is producing a peak that’s causing trouble. I use three subs in my main system. I use my spl meter to set each one individually to about 68db(if memory serves). Then when combined I get fairly close to 75db which is what audyssey wants. It makes it easier to calibrate imo even if you have XT32ht, which shouldn’t struggle with dual subs, but ya just never know.
Basically what audyssey wants is to set every speaker, including subs to 75db. Simply put, if the subs are too hot, audyssey can’t trim them low enough, by going quieter than -12.(max). Conversely if the subs are too low, audyssey will try to hit 75 by boosting. If that’s the case, you might find the sub channel in the positive. Like say +5db. This is too high, as you can introduce distortion into the signal, and also you have no room to bump up a few db ost calibration, or run a house curve.
Another possibility is simply that due to the different port, driver, and cabinet tunings of the different subs, the peaks and valleys could be more erratic than if they were matched subs. I know RSL said they would work together, but imo, the unpredictably of totally different tunings can cause many more issues than multiple subs have the potential to fix. I know you intend to keep them both so I would recommend trying as many locations as you can afford. Also, whether or not you find satisfaction with audyssey, a umik and rew is always highly recommended. That way you can see where you’re flying!
ETA: The sub test to the right of the green means the sub/s are too loud. Manually lower the trim/s directly on the subs and retry until you get to maybe -8 or -9.