You don't have to use direct to set your mains to large and still have the sub in the mix. Likely you are getting boom because you have too many speakers that playing down to frequencies that are localizable and conflicting due to locations. If properly crossed over, the sub should handle the majority of the bass, however a VTF-1 seems like it is insufficient of a sub compared to the rest of your speakers.
If you haven't calibrated the system to see if/where you have humps or dips in response, then you can't really set the crossover properly the way you have it configured. Basically, without knowing where your sub overlaps with your mains, you don't know what you are getting. I would use the receiver's crossover with it set as low as possible so that you don't have any overlap bass between the sub and the mains, but it may be that you simply need a better sub to get the most out of your setup.
In this configuration, you don't have an LFE channel; that information is simply sent to the R & L mains AND the sub. Using the receiver's x-over does not preclude you from using Direct mode in certain instances as well, to temporarily set your speakers to large, but you would also need to be playing a multichannel signal that has an LFE channel for the sub to receive signal (such as movies, SACD, DVD-A, etc...). Stereo signals are just that: no LFE.