There is no need to speculate about this. It has been measured. It is basically a non-issue:
Speaker Break In: Fact or Fiction? | Audioholics
There are tiny, tiny measurable differences, but they are not going to generally be audible.
A couple of choice quotes:
From the foregoing analyses, it's reasonable to conclude that suspension compliance changes arising as a consequence of initial driver burn in has little effect on the performance of a loudspeaker system.
And:
Normal production unit-to-unit driver spec variances can affect final amplitude response of a system to a larger degree than that expected from normal pre- post-burn in driver suspension compliance changes.
In other words, if you are not ROUTINELY hearing differences between different speaker drivers of the same model, you are NOT hearing differences from speaker "burn in."
In other words, the idea that "burn in" affects the sound is essentially pure BS.