There are two likely scenarios that occur to me:
1) You're overdriving your sub. No matter how potent the sub, it can be overdriven. The fact that the distortion went away when you inadvertantly reduced the input by incorrectly connecting it lends weight to this notion.
2) You're engaging a room mode with a frequency low enough that your previous sub didn't excite it. Room acoustics dictate the sound quality you get more than any other component in your system, perhaps including your speakers. Below 100 hz it's not uncommon for peaks and nulls creating a variation of 20 dB or more. An easy test would be to get a disc of test tones (cheaply obtained from Stereophile or Rives Acoustics, or from various websites at no cost) and an SPL meter (also pretty inexpensive from Radio Shack; it's not a lab grade instrument, but it'll get you in the ballpark).