Replace the speaker wire with shielded cable. Use a speaker wire for a line-level signal and you're sure to get a lot of buzz and noise.
Not to mention, you can't solder speaker wire to an RCA, especially of it's 16 ga. or larger. It's physically too thick for the barrel to fit over the cable. Plus, the wire itself is too thick. The amount of heat it takes to solder wire that size will ruin the nylon bushing inside the connector (that the center pin sits in).
Regards,
Wayne A. Pflughaupt