Considering how speakers are the most critical, preferential and varied component to an A/V system, and how important it is to fully audition them with the correct set-up/environment, I'm surprised there aren't any "speaker only" higher-end B&M stores around (if there are, I never heard of one).
If a store eliminated all the other less crucial gear easily available (and returnable) online or at mass market stores, and focused solely on speakers, they could carry far more brands and types of speakers than the average high-end store, and have rooms and staff fully dedicated to speaker know how. Of course, there can still be room for all those wonderful speaker accessories (cables, stands, etc.)
This is all assuming of course that:
1. The staff is very knowledgeable and that the speakers are set up and displayed to the best of their particular advantages.
2. The staff were not pretentious, stuck-up, intimidating jerks and didn't hassle customers on returns.
3. The store was in a larger city.
As many have stated here: who wants to buy speakers based solely on hearsay, then deal with the serious hassle of re-packing, re-shipping large speakers that you didn't like, plus all the potential re-stocking fees, etc., damage, insurance, etc.
I think something like this could be successful if done correctly.