You should actually set your subs to 250Hz or whatever the "full range" setting is. This lets them take in the full LFE channel and only stuff under 120Hz (with a drop-off) will go to the subs from your speakers.
I think 70-80Hz is best if you can pull it off since that's what most home theater speakers are designed for when at volume.
BUT, that's not exactly true for every use case. Your case of saying your speakers sound muddy could very well be the case. The issue is that as we increase frequency, the ability to localize those sounds becomes more prevalent. 120Hz is pretty high. I'd try 90Hz first and see how that works for you.
---
My front speaker woofers are 8", but the sides and surrounds are all 6.5" like you said. It's possible you're more sensitive to that sound on the front speakers. Ideally, you'd keep your base layer speakers all at the same crossover so when sounds pan around, they don't disappear and come back.