Male voice with too much bass, is usually caused by speakers with incorrect bass tuning from the cabinet volume and size of the port tube. It usually is a deliberate choice of the speaker designers to tune the bass to be too hot. It also causes music in that bass range (such as electric bass guitar) to sound muddy. The B&W 602 is known to suffer some from this problem.
What you're doing with the sub woofer is probably the only way you can deal with that. Try 70 Hz or higher as a crossover point to see if you can minimize this or make it disappear. As a guess, I think you might have to go to about 80-100 Hz.
Wayne also pointed out speaker placement as a possible cause of this problem. Are your speakers inside a cabinet, on a bookshelf, or near walls or a corner? That can also cause bass to be overemphasized. If so, try temporarily moving them so they are further away from reflecting surfaces and see if that helps.