Cab,
You sound like a person who knows where he’s going, and is willing to take a different route to get there. I put my first system together in a similar fashion 25 years ago. I started with source w/headphones, then added pre/eq, amp, and speakers as money allowed. It may take a while to get there, but eventually you reach your destination.