If you want to control volume from the iMac you should use the analog input on the R-N301. As irv mentioned in post#4, you need a 3.5mm jack to rca (stereo) cable. The newer iMac models would switch the output to analog automatically but I am not 100% sure about the 2008 iMac. If not, take a look of the settings to see if there is something you have to do.
All the Macintosh laptops and desktops that use the combined analog/optical input & output connectors will switch automatically. When there are multiple audio out options (eg you have an optical cable and an HDMI cable connected, or an analog cable and a USB DAC, etc) you use the Audio Midi Setup App found in the Utilities folder to choose the desired output (and if applicable, volume and sample rate / bit rate).
Although it seems popular to diss the various model Macintosh's AD/DA converters (Cirrus Logic 24/96 and 24/192 ADCs),they are not as bad as some make them out to be. Many reasonable cost external DACs simply benefit from putting some space between the noisy computer environment and their chipsets, rather than any inherent superiority to Apple's digital audio BoM),and despite the "massive differences" some claim, few people can hear the difference in my experience (although some are much better than chance, it takes a trained listener to discern the differences).
Put another way, a lot of people say they don't like the built-in ADCs simply because they are told that's what some "authority" proposes. And I've heard external DACs I didn't like as well as the one in either my Mac Mini or the Macbook Pro. And then there is the not insignificant number of people just hate everything Apple, period, and that's been around for 25 years and in some cases 35 years.
I'm reminded of the double-blind DAC shootout held at the 2017 Rocky Mountain Audio Fest (RMAF) where experienced listeners chose from ten options*. One comment " ... There were differences, but what stood out was how similar they all sounded. Who worries about this stuff? ..."
I suggest you try the analog out, upsample your content to 24/96** via the Audio Midi app and see what you think. You may surprise yourself.
* Actually, four options per listener, chosen at random from 10 possible candidates. Even with the limited number of DACs it took six hours per listener to run the DBT,
^^ Or 24/192 if your model supports it. Purists would suggest 24/88.2 and 24/176.4. Do some listening and see what you think, not what someone else tells you to "believe". Your ears are the only ones that matter.