I read the
manual. The "mute" scenario is unlikely because if pressed, it is supposed to mute the volume completely. This thing does have a weird volume feature, you can set it to variable, fixed, fixed 01-95, and fixed max. When set to fixed, the gain will be fixed so the OP would have known right away if he had set it to "fixed" or "fixed 01-95" accidentally.
That means the likely scenario is, as mentioned before, the unit may be malfunctioning.
Before contacting the dealer or Rotel support, the OP can try a "Factory Default" reset first and see what happens.
"• FACTORY DEFAULT: This sets the unit back to the original state as when it left the factory. Press the + button on the front panel or RIGHT arrow button on the remote control to select then press the “ENTER” button on the front panel or on the remote control."