A subwoofer setting of Both implies Mains set to Large. Most receivers will not let you choose Both (Double Bass, LFE+Main, Plus, whatever) if the mains are not set to Large. Maybe the Yamaha automatically sets the mains to Large when you choose Both for the sub setting, maybe it is finicky. Older Yamaha manuals I have read specifically state that as the case.
When the Mains are set to Large, they get a full range signal. With Both, any bass below the xover that is sent to the mains will also go to the sub. Generally you want all channels Small and subwoofer set to Sub only. Note that a xover is not a brickwall filter and some frequencies below the xover point will still go to the Mains even if they are set to Small - they will just be greatly attenuated.