@Methodical I hear ya, but I feel like I've already been there. I'm still open to any suggestions, but at this point I feel like I have a pretty decent understanding of how the input routing works, and I can consistently get HDMI audio, digital optical, and analog to all work when using a particular HDMI source for video. Also, I know the digital coax output works on the computer because it works on my old receiver and it works on the new Denon if I convert it to analog first via the adapter I mentioned in the original post.
Since my last post, I have contacted Denon support, and they had me do a firmware restore as a last resort, and unfortunately the same results -- digital coax inputs are not working on the default firmware or the updated version. They were really cool about everything, and told me that there were no reports of this issue but did acknowledge the volume issue that I've seen reported in other threads, which I thought was pretty honest of them. They said that if the firmware restore did not work, then the next step is to RMA the receiver. I will of course update this thread once I get my hands on the replacement.
@TLS Guy Thanks, here's a screenshot from the web UI. I also included windows showing that the digital output is selected as default playback device and that it is playing audio. I've also tried other HDMI inputs instead of the 8k input with Digital set to Coax1 (and Coax 2 just in case maybe the one port was bad) and Input Mode set to Digital.
View attachment 38765