i usually take 'em off
the sound just opens up spatially and the highs, while they are SLIGHTLY brighter, thats not the dominant sonic effect. sounds just get more spacious and free.
i even did this to my old dynaco a25s and it helped. the problem is that the high freqs tend to bounce off the grill cloth because of its proximity to the tweeter and mid. i queried richard vandersteen about this and wheter he thought it affected his speakers such as the 2CEs etc. he responded that because the drivers in his speakers were distant from the cloth, that there was no such reflections. his speakers tend to sound quite open so it is consistent.
some of the speakers can be plug-ugly without grills but for me, the beauty is in the sound. when there is a sig-other its prudent to put the grills on when not listening or especially if company for her is coming over.
my kids could never get past the razor wire i put up around the stereo to do any damage.