I had attempted to swap RCA cables at the start; in addition to both RCA cables having the same problem in this system, they both work perfectly with known-good components so I ruled out an RCA cable issue. I don't think it's an issue with power levels because when I use a 330W Polk soundbar connected via the 3.5mm jack and turn it up all the way the sound is crystal clear, whereas the Niles is a 2 x 100W amp and buzzes at any volume. When the isolator had failed to rectify the issue I did attempt to connect the components' metal bodies together (the Denon has a very convenient signal ground terminal) and it had no effect. This is why I'm fairly confident that it's not a grounding issue; directly connecting the metal bodies of the two creates no change. This is why I'm intending to send the amplifier back to Crutchfield, even though they don't seem to think it could be bad.
That's what started me down the path of figuring out what else I can do with that pre-out. Frankly, I could just leave the Polk soundbar connected, it actually sounds decent. I just don't want to have to deal with mounting it above the TV. But it got my mind turning, and I was thinking that if I do go that route and replace it with the Nakamichi Shockwafe 9.2 system I was running back before I started the whole home theatre project and I set the Shockwafe to stereo mode, I could bring the Height2 pre-out signal through all channels. And if
that's feasible, why not just a whole other system? It seemed like a kind of wacky idea though so I wanted to get the opinion of some folks with more experience
Thank you to everyone responding for the insight, much appreciated!