No I'm not set on towers, I think I went that way because of the current implementation is poor at best and wanted to not to repeat the same "mistake".
Yeah, my first system was an RCA home theater in a box. You'll be surprised at how good a good pair of bookshelf speakers can sound. The advantage of towers is deeper extension, and possibly higher sensitivity if the towers have more drivers. But you can still get transparent, revealing sound from a good (not just any, but good) pair of 2-way bookshelf speakers; and the low-end extension will be handled by the subwoofer. You've got a fair bit of space to pressurize there, so don't skimp on the sub.
That being said I made a number of mistakes when setting up the current systems, eg wire/cable choice, speaker location.
Possibly -- but ultimately, good sound comes from good speakers. You're not going to get good sound from a home theater in a box, regardless of how optimal your wire and placement are. Be careful not to be oversold on your wiring and cables.
Monoprice is great for both. Get oxygen-free copper speaker wire, using
this guide to determine the gauge. For speaker wire, if it doesn't say OFC, it's probably copper-clad aluminum -- which is only about 2/3 as efficient as OFC if I recall correctly. (Amazon Basics speaker wire is CCA, for example.) If you must have boutique cables, then shop
Blue Jeans Cable. Remember this mantra: good sound comes from good speakers.
My displeasure is probably not all the equipment's fault.
It most likely was. And that's OK. Your home theater in a box was probably miles ahead of TV speakers. And the system you're shopping for now ought to be miles ahead of your HTIB. By the way, your Denon receiver is a good one. Keep using it, just with better speakers.
My question is; should I build a MDF enclosure around the speaker? It would be reasonably easy to accomplish and would keep the blown-in insulation out of the speaker if nothing else.
Gene says a back box is a good idea, although a full MDF enclosure is probably overkill. His install uses the sheetrock as part of the enclosure and 2x4's as the sides. There's probably not enough pressure generated by surround speakers to cause even a sheetrock enclosure to flex enough to affect the sound.