You will have EDID issues and possibly have signal blanking anytime a HDMI matrix switcher is added to the mix. Hopefully you didn't run any HDMI cables without putting cat5e/6 cables right next to it to ensure reliability should a HDBT unit be necessary.
The HDMI matrix switchers simply don't have enough reviews out there. Most blank, many more have HDCP reauthentication, and almost all only allow for lowest common denominator audio and video. So, if one connected source only allows for 1080i and stereo audio, then that's ALL you will get.
After enough time banging my head against the wall with other people's problems with more simple setups and doing my homework, I'm going to get a Crestron solution in my home. Pricey, but solves the issues. HDMI input cards with DSP audio downconversion and scaler output receivers which will allow my non-1080p displays to work in this day and age. Gonna cost an arm and a leg, but is truly reliable.
I think you will be okay with what you are describing, but be aware of the caveats of HDMI matrixing.