I've tried both sub out just above the surround back and the pre out ones just above the center out on the picture. In the pre out it hummed very low and the other 2 were completely dead. Shouldn't it give a test signal in setup even though the receiver is in straight?
Hi. The black "SUBWOOFER" jack above the surround back speaker terminals is NOT an output jack - it's an INPUT jack. So, don't use that one to connect to your subwoofer.
The white and red ("FRONT" and "REAR") "SUBWOOFER" jacks in the "PRE OUT" section are the ones to use. Either should work for you, but if it were me, I'd use the white one if I just had one sub.
As for menu settings, the defaults should have worked for you - but because you may have switched things while trying to get it to work, this is what I would use (refer to page 116 of the Yamaha manual):
(*) "Subwoofer 1, Subwoofer 2" set to "Use" and "Normal"
(*) "SWFR Layout" set to "Monaural x2"