Consider the thousands or millions of metal to metal boundaries a signal will encounter on any PCB and you have to realize that your logic is fundamentally flawed.
Metal to metal contacts are not the issue. It is when 2 different metals make contact with each other in the signal chain that you could introduce artifacts. Any time 2 different metals come into contact, a voltage develops b/w the 2 metals. This is the basis for thermocouple technology.
In an ideal world, you would have all of your wiring and interconnects on a signal path made from 1 continuous peice of metal (from source to voice coils), either etched or forged or stamped, etc, with no disontinuities or irregularities. Obviously there are many hurdles to that approach.
This is analagous to engine exhaust systems on high-end sports cars. If you are serious about performance on a sports car, your exhaust from the heads to the tailpipe will be bent out of 1 pipe. That way, there are no irregularities from welds etc that can cause turbulent flow. When you move into electromagnetism in physics, they always take you back to the concepts that you learned from mechanics (fluid flow), as it is analagous.
If you are serious about signal quality, then you take every step that you possibly can to eliminate turbulance or noise, thus I keep connectors out of my signal chain.
Now, does it really make a difference in the real-world? Quite probably no. But, am I an OCD audioholic? Quite possibly. If I can simplify something and avoid the possibility of noise, then I will.