In that price range you can get a pretty nice rig. I know the AH staff uses RBH speakers as their reference, but of course not the cheaper ones. I've never heard any of them; all I know is they've been well reviewed.
In any event, you'll want to start with a good sub. This will give you the foundation for music and HT, plus relieve your main speakers of reproducing bass. This helps dynamics considerably. Many people advise that if you only want to play music you can get by without a sub. I tend to disagree- a good, accurate sub enhances music considerably. However, you'll not get a good home theater experience without a good sub.
There are several good ones available for $500-600, or even less. You don't have to break the bank, but you shouldn't skimp too much. Buying enough sub can allow you to get by with smaller and cheaper speakers- it all comes down to tradeoffs, no matter what the price range.
My standard recommendation is a
Hsu Research sub. They've received more critical rave reviews than almost any sub I can name, and I myself have a pair of their best. They make some of the best on the market, yet they're pretty affordable. Their customer service is superb (very important, IMO) and the sound is great. Best of all, they work well for HT and music both. Their STF-2 is sufficient for many systems, and their STF-3 is enough for even fairly large theaters. They make subs for almost any conceivable system.
SVS also has an enviable reputation, especially for HT use. Their subs are powerful and affordable, and their owners are often rabidly loyal. They'll go as loud and low as you could ever ask of a sub in that price range.
The
Adire Rava also merits inclusion on your 'short list.' Who else can deliver a 60 L cabinet with a 12" Shiva driver and a 250 W amp, all for $400? The Rava is a superb sub so cheap you could scarcely DIY it for less. The Shiva driver is a legend in the DIY sub community, and the company is good to deal with. You can get a sealed or passive. This is a quality veneered sub for the price of a Best Buy 'partical board wonder.'
As for speakers:
Pardigms are pretty good for the money. There are several Canadian companies that make very good affordable speakers. PSB is another Canuck. I'd say overall the PSB is among the most widely loved of all the reasonably priced brands. The Absolute Sound has chosen their
Image towers as an Editor's Best Buy, and many other mags have similarly praised them. For my money they sounded a tiny bit "dark," but they had a nice midrange, good bass and were pretty dynamic.
Wharfedale doesn't get much ink in the States, but the British press has gone ga-ga for several of their models. They have giant killer rep for sound and value. In America,
Audio Advisor carries some of their speakers.
Axiom is also famous among web-savvy audiophools. I've sadly never heard an Axiom speaker, but many here can advise you in that regard.
Infinity actually does make some decent entry level speakers, too.
Athena gets an awful lot of praise for their inexpensive speakers, like their
Audition line. They really are damned fine for the money, and they can be had at considerable discount as many Best Buy stores carry them. It's hard to top them for really cheap speakers, and although you've budged considerably more, as you say, why spend more than you need to? They're worth an audition if you've got a seller nearby.
That leaves my favorite, Monitor Audio. They're a bit "livelier" or brighter than PSBs, but not overly so. The
Bronze B series are giant killers, too, and well within your price range. The only bad thing is it's not always easy to find someplace to audition them. And they're not a direct sales maker, so you may not find a dealer who'll ship and let you return them if you don't like them.
There are others, but these are all good places to start researching. You're so lucky! I've said this so often, but it really is true- we're in a Golden Age for audio and home theater gear. Never before has so much good sound been available so cheaply. A carefully selected $2000 speaker system can blow your doors off! This site is certainly a great resource to help you in your search.
BTW, I think the Denon AVR-3805 is an outstanding choice. I've got the older 3802, and I'll likely upgrade to the 3805 myself at some point.