If the grill material is pretty transparent I doubt any negative consequences would be audible. You might be able to measure them, but probably not hear them. If you hold the grill up to a light can you see through the material with relative ease, or does it appear to be very dense? Either way, I doubt leaving it off is a good idea in your case. If it turns out to be a problem perhaps you should consider a down-firing subwoofer instead. That could mitigate your concerns.