I personally would never do away with 5.1 for movies and gaming though I do listen to music normally in 2.0 or 2.1. While a 4.1 (phantom center) might be worth a try I believe removing the surround speakers really takes something away for most movies. I have watched some TV movies in stereo, as that was the way it was broadcast, so I'm aware of the difference. Realistically, running surround speakers doesn't add much cost to the system, they don't take much power and even "sub par" speakers, when compared to the mains, will generally do the trick.
Steve