I've run into this before with DIY speakers. It would rattle like the woofer was blown with very low frequencies... rarely with music but several movies could get it to rattling even with low SPLs. We swapped drivers, reconed drivers, bolstered the cabinet, sealed the cabinet again, and it was finally solved by plugging one of the ports. It was very odd since both cabinets were indentical, but only one did it.
At any rate, try plugging the port and see what happens. Even if this is not a solution, it's a step towards finding out what's causing it.