The wiring seems like it is wrong from the start.
There is no negative connection in an unbalanced signal. There is signal (+) and ground. If an amplifier is wired with left on positive and right on negative, then the audio will cancel itself out as the amp flips the signal and sums the two. This often can eliminate vocals and mess significantly with the audio output levels.
I wouldn't expect anything more serious in terms of signal degradation.
If the incoming audio is mono, then the signal should be signal (center conductor) to + and ground (outer wire) to the ground connection. I would expect if this is for LFE, then it is a mono output from the source.
I assume that the two RCA jacks are connected to the LFE outputs of the Marantz, and that the amplifier then has the signal (center) on the two + connections on the amp. One RCA on one input, then the second RCA on the other input. Either move the ground to the ground connection, or jumper the ground connection to the two negative connections to get a solid signal.
I wouldn't expect a huge jump by putting a unbalanced to balanced converter in line with the amplifier as we aren't dealing with a mic level signal here.
Maybe you are thinking of a balanced 2-wire output (both conductors have the same impedance to ground).
With such a connection if 1V is present on the hot pin and 0V is present on the negative pin, the difference between them ... 1V ... will be the signal level.
If you connect the amplifier L ch Hot to the speaker and the amplifier R ch Hot to the speaker, you will get the phase-altered signal that was used to create pseudo 4-channel sound (as in the Dynaquad system).
I have not heard of a system connecting the L ch Hot and the R ch Ground, but I suppose it would work the same way, just 180deg out of phase.* That phase difference should not matter, however, because the information you will get with either wiring is out-of-phase information in the first place.
With a single-ended output 2-wire system (which describes the AV preamp outs he has) you cannot have 1V on the hot (+) conductor and 0V on the negative (-) conductor. The (-) has a lower impedance to ground than the (+) conductor does, thus it does not meet the criteria for balanced output.
Most balanced outs are 3-conductor, but they do not have to be. However they do have to have the same impedance to ground: (+) to ground and (-) to ground.
Finally, although it does not apply here, you can only have Balanced
outputs. What are colloquially (and incorrectly) referred to as balanced inputs are technically not balanced but are differential inputs.
So this discussion has nothing to do with balanced anything; it's a single-ended output into a differential input.
Now, someone could say "well, you know what I mean when I say 'balanced input' " and you would be right, I do know what you mean. But that's not what some posters said; instead they said 'differential input' is incorrect. But that is not true; it is perfectly correct.
* Correction: on second thought it would not be the same. In a single-ended output, the (-) L and R are connected together. It would be as if you were connected only to the L channel, with no R channel information.