When you say it's distorting, what type of sound are you hearing? The reason I am asking is the sound of a driver bottoming usualy sounds like a thunk or a clunck. If the driver beyond its normal travel range, but not bottoming you get a lot of doubling ( 2nd, 3rd, 4th, etc distortion). That is, if a 20hz note is played you end up hearing it as a 40hz, 80hz note instead. If the amp is runing out of power you kinda get a sharp or harsh sound.
The box itself can be making the distortion too. What material is it made of? If the walls are flexing or vibrating you will hear it. The surface area of the box is a lot greater than the driver, so if its moving only a fraction of a mm, your going to hear it. If you have a disk with test tones on it play them one at a time and go over to the box and feel the walls of it. Move your hand from the corners to the center and see if you can feel it vibrating at certain tones. If this is the case, we can figure out a way to save the box.
Let us know.