Exactly...
Receivers are not made for handling video, they're about audio. There are typically some basic video connections, like S-Video, sometimes but not always allowing for switching. But if you really want to benefit from MPEG-2 sources like DVD, it is much better to configure RGB or Component transfer directly to TV. Utterly few receivers have plugs for this.
Signal degradation is unfortunately to be expected for switches, internal or external. Cables are made according to industry standards, matching electrical conditions in equipments. Just compare a 2 meter/75 Ohm RCA cable with 2 enlaced 1 meter cables . Resistance, capacitance etc. change, and it's like putting a filter in the middle. To avoid this, switch has to be active and of good quality, as good as your equipment.