I think that if I did that, then I would run each connection through a double pole switch. Always flip the switch off for the input that you are not using at the time.
If you forget to flip the switch and have both hooked up, then you are back to the same problem noted above
So, bottom line, it might work but it might be risky.