Your x-over shouldn't affect the speakers that are set to large at all, which makes me thing something else is set wrong.
Make sure you have it set to SUB = NONE/NO/OFF.
As you said, bass should not get better when you raise the x-over, so the only explaination for that might be that your receiver is underpowered for what you are asking it to do. Bass requires the most power to reproduce and the benefit of a sub is that your receiver doesn't have to do all that work to reproduce it because the sub's amp is handling the lowest parts. Large speakers being driven by the receiver and producing plenty of bass can quickly push a receiver to it's limit.