You can do what ever you want, but.....
It sounds like you plan on moving the speakers around. You probably can do what you want, the way you want (i.e. just run the cables all connected to the same place at the source). But to alleviate any potential problems you can get two speaker switches, one for the mains, and one for the centers. Good ones with 5 ways are hard to find and expensive, cheaper ones with spring loaded clips start at $30. A few in the slightly less than $100 have screw terminals, older JAMO JSS4(6)-HP use to have 5 ways but current have spring clips or screw terminals.
Or:
For the left and right, you can connect the cables to the B connectors on the amp/HT receiver. When you go from location A to location B just remember to push the llittle buttons.
For the center channel, I'd recommend a four pole, 10 amp non-shorting (break before make) rotary switch ($70 plus).