Your AVR already has very high-quality burr brown dac's built in that are quite capable of very accurate results. They may not be perfect but you would need a very good room and speakers to have a chance to hear any issues.
Most audible differences like this warm studio feel you are talking about are going to come from either a new setup that fixes the frequency imbalance from their old system so it sounds better or it is just altering the sound like another eq in the signal path giving a boost to a frequency range. Many people also hear what they want or expect to hear so count this experience as a positive learning experience as you have not been sucked into this trap.
You just need to make sure the dac you use is good enough so it won't be the week link in your audio chain and then you can stop worring about it and concentrate on the important parts that you can change to get a better experience.
Listen to different speakers to find some high quality ones that you like and work with the music sources you listen to. Speaker placment can have a big effect so play with that. Room has a big part to play so treat your room where possible. And make sure your source material is up to scratch.
Another thing to try just for the hell of it is to try running the dac output direct to the power amp and bypass the AVR. I don't think it will make a big or noticeable improvement in which case just concentrate on better speakers. If you prefer this setup then maybe look into better equipment options.
You may also want to test with something other than the ATV4 and it's fixed sample rate output. Even a cheap chromecast audio could be used to test 96/24 output.
Thanks for the response.
The DAC is a Bryston BDA-2 and has been very highly rated from various publications.
The power amp is the Bryston 9Bsst which also has incredible ratings. We're talking close to $9000 for the two pieces. Clearly my Klipsch best buy specials are inferior and are so sensitive that the power and clarity the amp gives doesn't improve the speakers results..... imo.
I do notice a slight difference in where my ears perceive the instruments from when switching from HDMI to analog sound. Also my sub isn't as boomy in analog as it is hdmi when listening to bass strong recording.
I guess I was just hoping for that "wow what a difference factor" when it is more like "oh yeah it's a tiny bit better"
I've been switching back and forth on different recordings between analog and hdmi. Mainly the instruments seem more detailed. Also, I agree the ATV4 locked in output is limiting me. it's just nice to have one unit that does everything, (movies, music, netflix, apps, games for the kids)