Magnetic field of a ferrite magnet is all around the magnet, and that is why that in the CRT TV screen era, speaker drivers had to be shielded to prevent them from affecting TV image.
With Alnico magnets, the problem did not exist as their magnetic field is concentrated on the voice coil area. I suspect that a neodymium magnet would have a similar behavior to that of the ferrite magnet, but I haven't been able to find the answer on the web.