It's said that loudspeakers sound different with and without the grills attached. When you listen to your system, trying it both ways, do you hear a difference?
If you do, do you prefer one over the other?
If you don't, then it makes sense to play your system in the safest mode, that is with the grills on the enclosure.
I almost always hear a difference, but it's subtle and somewhat dependent on the other equipment in the system.