It seems to me that you really need to step back and spend at least a bit of time reading about surround sound setups and such.
The subwoofer, if you're a movie watcher, is one of the most important speakers which you will be using. If you have mid-range speakers, a good sub is almost the most important speaker in the system as it adds presence to the viewing. A high end subwoofer, which does cost a fair bit, is the main speaker which I think is worth every penny spent. Why? Because it adds presence. When you walk onto an airplane, you FEEL the motors, you feel the presence of the motors without so much as hearing them. The deep, low rumble as they kick into motion, how they fill your entire body with their presence. That's what a good subwoofer does. It completes that feeling of presence which speakers can only deliver up to a certain point.
In your Yamaha settings, you would set all your speakers to 'small' and let the receiver handle the crossover of higher bandwidth audio to the speakers and low frequencies (LFE) to be passed to the subwoofer. You would connect ONE RCA cable between the Yamaha and the subwoofer using the LFE input on the subwoofer. You would ignore every other connection on the subwoofer as extraneous. It sounds weird to leave them all disconnected, but I'm sure there is a lot on the back of your Yamaha which isn't connected as well. It's okay.
Once everything is is in place, run through the auto-setup of the Yamaha and let it do its thing. Then if you want a bit more volume in surround sound scenes in a movie, you can go into the speaker settings and turn up the volume of the surround speakers a bit more.
Make sure any surround speakers are connected to the 'surround left/right' outputs of your AV receiver and not something else.