I don't use threaded inserts for any driver smaller than 8". It requires a drill press, which I don't have, to get all the holes aligned. If just one is off, you can't tighten all the screws.
What kind of threaded insert are you using? T-nut, hurricane nut, or other? There are various tricks to install them that help keep them aligned better. Never hammer in a T-nut or hurricane nut!
With drivers smaller than 8", and a hand-held drill, I find it easier to use pan head deep thread screws like this #8 × ¾" from Parts Express.
If you pre-drill a smaller diameter hole before driving in the screw, they hold in MDF front baffles and I can't screw it up like I can with threaded inserts. I also prefer the square drive screws over the phillips head screws. With a square drive you cannot slip and poke a hole in your woofer or tweeter
.