How much memory is enough? What's the best way to connect the htpc to the system?
With memory, and I'm assuming you are talking storage, not ram, is totally dependent on your DVD collection. Average DVDs ripped as an ISO are 3-6 GB, so you can see that will add up quickly. I would start off with at least a 1 TB drive.
My media center is actually a modded XBox original right now, so it's hooked up using component cables and a digital optical cable. But if you're building an HTPC, you'll usually use HDMI or HDMI and a digital audio connection if your receiver doesn't handle audio over HDMI(like mine).
Here are a couple cheap options for a XBMC based PC:
ASRock
The ASRock would still need an OS and probably a larger hard drive, a TB external or replacing the internal drive would probably work.
Acer Aspire Revo
This comes with an OS and small hard drive, but you would still want to add an external drive. Also it doesn't have digital coax or optical so you would need to have a receiver capable of audio over HDMI.
I have not personally used either one of these products but based upon all the reading I've done both of these would work well with XBMC but I don't know about other programs or applications.