A Blush Brush (natural fibers) and gently brush away.
something like this......
http://www.eshadesofperfection.com/xcart/customer/product.php?productid=16183
But buy it at your local supermarket.....or ...CVS...or Walgreens...or Walmart. It will be a lot cheaper.
If you have a vacuume cleaner with a narrow crevice tool, you can use the crevice tool to help you suck up the dust and grit, as you clean the corners around the woofers with the blush brush. But DO NOT get the vacuume too close to the actual woofer diaphragm. Use the blush brush for the diaphragm. Be very gentle and work slowly and gently.