Excuse me for the delay in responding to your post. I've been dealing with some serious health issues. Hopefully I am past that now.
I have Philharmonic HT R & L, Philharmonic HT Center Channel and Philharmonic True Mini Monitors for rear speakers. The MLP is approximately 11 ft. from speakers and the speakers are 11 ft. apart. The listening room is of moderate size - 32x25. I have the system set up in the middle of the long side. I don't listen to it very loud, maybe -10 max to -20. I haven't measured the sound levels but will soon.
Denon AVRs typically has diagnostics that would tell you the reason for the shutdown events. If you cannot find the steps to display the fault, I can take a look of the service manual of the X3700H (available) and hope that the same procedure would work with the X3800H. Unfortunately the SM for the X3800H is not available, but if I recall, there was some YT video that mentioned diagnostics. In the meantime, you should contact Denon and they should be able to instruct you how to get the fault code/or display the causes for the shutdown.
There is a good chance that you are over driving the unit, volume of -10 could be too high, for those 4 ohm nominal speakers at 11 ft and in a large room; and would be worse if the trim levels are set to >0. So I suspect the causes of the shutdown would likely be overcurrent, not thermal but current spike that may be high enough to trigger the protective circuit. How did you get out of the shutdown mode, by turning the unit off and on?
Try to keep the volume below -20 and see if it still shutdown while watching those same movies.