Breaking-in is very reasonable from a materials point of view. Any string instrument player will tell you that the instrument's sound changes over a period of playing. Therefore, the wood cabinet in a speaker is also very likely to improve in sound with age.
Brasswind and flute players also zealously state that their metal musical instruments change over years of playing, the sound waves "aligning" the metal.
Based on observations of musical instruments, I believe that the materials within a speaker, such as the cabinets and drivers, do break in after a period of playing as the sound waves changes the molecular alignment of the materials.
The wood in an instrument, like a violin, is made to resonate. It is very different from the typical speaker box, which is made to be as inert as possible; any sounds the box makes are distortion, not the signal from the speaker.
As for the claims of artistic types for what changes over time, that is no way to determine facts.
With a wind instrument, or a brass instrument, they are constantly getting spit in them, as one cannot help but release some spit occasionally while blowing.
Also, with something like a violin, if it uses the old fashioned gut strings, sweat and oils from the skin are absorbed into it, which affects it. Wood also absorbs such things, though with a proper varnish, that effect is minimized. That is not an issue with properly handled speakers, as one does not touch them normally.
However, I expect that a speaker might sound different if you sprayed it with sweat and oil, or spit, though it will all depend on how much sweat and oil, or spit, and what, exactly, the speaker is made of.
So, your analogy is not a good one at all.