Welcome to the forum!
Yeah, "audiophile" has a certain connotation to it. I think that most people (especially on this forum) are just like you, though.
Personally, I like the effect of surround sound, so I'd lean towards getting the center channel and rear speakers. Running stereo speakers is also just fine if that's what you like. My receiver (Pioneer 1015) does a really nice job of downmixing 5.1 material into stereo, and I've accidentally watched shows in stereo without noticing.

With some movies, though, it would have been obvious because they have more sound going to the rear channels.
Modern day receivers can do a really good job, IMO, of automatically equalizing the frequency response of speakers, so it's not as critical as it was ten years ago to get timber-matched speakers. You could always get an external equalizer and get all fancy, if you were that interested. So...you don't need to get matching speakers to have some benefit.
You are totally right about the rear speakers. They are not as critical. However, you shouldn't just ignore them. I noticed a distinct difference in quality when I upgraded my rears from $10 speakers to $300 speakers. Okay, yeah, that should be expected - but I'm just giving an example.
So, I think that you are on the right track. Get a good center channel because that speaker is used a lot in shows, and get decent speakers for the rears. I'd recommend buying from someplace that will let you take them back - that way, you can try them out and decide for yourself if they are better or worse than just using your two front speakers.