Or is bass from front being played twice by fronts and sub?
Yes, bass below the xover is being played by
both the sub and the fronts (hence the term 'both' or Onkyo's more descriptive term 'double bass').
A speaker set to Large by definition plays the entire frequency range of the signal it is given, no xover at all - so fronts set to Large would definitely play bass below 80 Hz. The 'both' option means also send the sub the bass below the xover from the front speakers to the sub, because normally any channel set to Large would
not send any bass from that channel to the sub.
Yamaha receivers/manuals are second only to Denon in confusing the user. I understand this stuff and yet still scratch my head sometimes when reading Yamaha user manuals (don't get me started on their min/max/std settings for dynamic compression).
A Small setting means to send the bass below the xover to the sub, so how can setting the fronts to Small accomplish 'both' or 'double bass'? With a Small setting, the bass for the fronts would be redirected to the sub and therefore the fronts
would not be playing any bass below the xover and it would not be 'both'.
The simple fact is that setting the fronts to Small or Large is irrelevant when the 'both' setting is on - they are Large - and the receiver ignores the actual setting, unlike receivers like Onkyo that enforce the rule that 'double bass' cannot be set unless the front speakers are set to Large. I'm sure there is a footnote in tiny print somewhere in the manual that advises of this fact but it would be far less confusing if they had just implemented it properly and not allow the user to set a setting that will be ignored.
Forget about LFE (the .1 channel in DD/DTS soundtracks) - it
always goes to the sub if you set subwoofer=Yes. It gets confusing because people will use the term 'LFE' as a catch-all for bass frequencies when in reality it is a discrete channel, band limited to 120 Hz and below, that only exists on DD/DTS soundtracks.