Yeah, it seems the room, placement, and insufficient calibration all conspired to give you deep nulls where you listen from. Too bad you already sold the Monoliths, a tweak of the phase control or placement may have salvaged them. But they were too big, placement options too restricted, and they're already gone, so no sense in sweating it now.
If an array of ssmaller subs is in the cards, it seems a valid path to pursue. More subs with more optimal placement will definitely provide more effective modal smoothing, before any corrective eq is applied. And your small room justifies the space savings of going sealed. Small rooms load up, with cabin gain kicking in at 35-40hz, which is quite complimentary to the shallower roll off inherent to sealed subs, particularly 12's of decent pedigree, not requiring heavy handed headroom robbing eq to get deep extension in-room. Three or four of them should bring copious headroom from 25-30hz on up.
The Denon lacks independent sub tweakability. Subs (or sub amps if you pursue DIY) with sufficient built in calibration tools like variable phase controls might allow you to avoid the expense of a miniDSP via manual tweaking. Those miniDSPs are so expensive it may be more cost effective to swap to a higher tier AVR that has multi-sub eq of some sort instead. Start with the subs, and cross that bridge when you come to it.
