I don't think you'd ever be happy going to the little KEF speakers from what you have.
Polk SDA-SRS
KEF KHT 3005
Search around and either decide to keep what you currently have, which most likely sound incredible (I've never heard them), or find some new speakers which will compare in sound production. I would stick with towers for the front and a solid center channel (matched, of course). I love my Aperions and really enjoy my buddy's B&W speakers. I could never go to anything like the little KEFs.
Remember to consider the size of your room. If it's large, the smaller speakers will never be able to keep up with the space. There's a reason that speaker systems for rock concerts are massive. Size matters.
You'll most likely get lots of suggestions for companies and some really solid advice regrading the Polk speakers and what would be a good match to those in the current lineup.
Happy searching and listen, listen, listen (to any speaker before you decide to keep them).
-pat