I've had mine on lately on the JBL monitors. Even immediate near field is hard to hear a difference. If there is one, it's not wrong, either way, and whatever subtle difference there may be, it's lost on me 30 seconds in and the music cancels it out. In other words, it's too subtle to focus on.
When I am listening to them with the grilles on, nothing compels me to remove them. On these JBLs, the grilles are open on all 4 edges as they are suspended off of the baffle.