Actually, I think that is a good way to start, especially if this will be mainly a music system. Two good channels will beat 7 mediocre ones any day. Also if you have legacy equipment there is no compelling reason to have an AVR.
If your interest is music, get the best speakers you can afford. Get a decent used amp and preamp. With an HDMI switcher you can switch video. You will have to switch the audio separately, which is not a big deal.
That way you can add speakers, and you will have a decent amp to power your mains from receiver pre outs as you expand your system.
This system, I put together so I could enjoy music at the fireside in the winter, and also watch TV and movies in that space if I wanted.
The system was put together largely from items I already had lying around. The Quad FM 3 tuner and Quad 33 preamp go back to the early 70s.
Speaker position is not optimal, but this is an architecturally sensitive space.
The system ended up being successful and fills this large space well. Originally I powered it with a legacy Quad 405 2, but added more power with a Quad 909 and added a couple of subs I built back in the 80s powered by the 405 2
It turns out that the center image is in fact very stable and dialog is easily heard. So yes, you can start with a two channel system. In any event this space would not be suitable for multichannel audio anyway.
Admittedly this is secondary to the system in my signature, but this little system is very pleasant, and I'm glad I put it together.