As far as AV goes, the remote is the obvious "hub", but since it is typically only me (with occasional use by my daughter or GF who are already in my "shelter at home" pod) I don't see it as a threat.
For elevators, speech commands (like on Star Trek, though there you are telling the central computer where to take the elevator) should work fine given the limited vocabulary required (with a simple list of the command words posted in the elevator). I guess you could leave the control pad in place for people with speech disabilities.