Haven't even had enough time to run YPAO to do a general calibration/setup? That's probably a good start in any case...but do read the manual
I looked for spec on a Swan Diva 5.6 and got references to some other model numbers, so not sure what it is. I doubt the Swans are anywhere near as capable as your subs in lower bass, and could well lack in the bass range you prefer, too, especially if contrasted with the sub's capabilities but if you haven't even calibrated yet wouldn't make too many judgements until you do.
Hard to know what your preferences are. In any case I'd still use the subs and start with a crossover of 80hz and not use the Swans full range, but you do have a feature for double bass in the Yamaha (called Extra Bass by Yamaha, see pg 122 of your manual) where you can set your Swans to full range and still redirect some bass content to the subs as well.
LFE only happens with content that have recorded such a channel so is it with movies you're finding a lack of bass or is it music?