Usually, there is very little, if any, difference in sound quality whether the grills are on or off. Manufacturers do try, after all, to use acoustically transparent materials for grills. In fact, I have never seen any double blind tests showing that anyone could hear the difference with any speaker, but I have never seen anyone bother to test this.
With that in mind, unless you and everyone else who visits your house is invariably careful, you should leave the grills on because they protect the speakers. If you ever decide to sell them, it will significantly affect their market value if the woofer centers or tweeters are pushed in or dented. Even if it makes no audible difference in the sound (which, if that doesn't, you are going to have a hard time convincing me that you really hear a difference between having acoustically transparent grill cloth between the speakers and you or not).