I would try to help you troubleshoot, but need to be clear on the following:
1) You refer to "sound", what kind of sound? Is it hum, or hiss and if hum, does it sound like 60 Hz, 120 Hz, 180 Hz, or 240 Hz?
2) When you said disconnected everything except power cord, RCA cables, speaker cables, did you mean 3 pairs of speaker cables, and 3 RCA cables all connected?
3) Were the AVR-X3800H and the CA-5200 connected to the same wall outlet directly, without any other gadget such as isolating, noise suppression, UPS, power conditioners etc.?
4) If yes to 3), have you try to plug the CA-5200 to a the same outlet, just for troubleshooting purposes.
5) Have you tried the process of elimination, that is, have nothing but the power cords connected, no speaker wires, no RCA interconnects, then proceed with connecting one RCA interconnect at a time, and so on, until you hear that "hum" (assuming it is a hum sound for now)? If it is caused by one or more of the RCA interconnects, then it will narrow things thing for solution.
1) is important, you need to know for sure if it is a ground loop "hum", just "sound" doesn't tell us that.
First of all, connecting a wire "behind screws" of two preamp, power amp never, or rarely work, but the myth continues on the internet. That's because without seeing the schematics and wiring diagrams of both units, it would a hit and miss thing. Thing about it simply, if that works, why wouldn't manufacturers tell you right there in their manuals and even provide such ground screw in the first place. Such chassis ground connections do exist in many old gear, mostly those that have phono preamp built in, but the vast majority don't provide such things, yet people will look for those other "screws" not really designed for the purpose. So I would suggest you don't waste time on that one, as you have tried already (yes in some cases it could be effective).