I agree with you, but would like to see measurements of the magnitude of the effect from the second pair. The first several speaker comparisons I did, I swapped locations of the two pairs and established (to my satisfaction) that the difference associated with location (when side-by-side) generally was small compared to the difference between speakers.
However, as I concluded when comparing amps, the difference associated with location exceeded the difference between amps so as to render my attempt at a comparison under those conditions as worthless (other than to conclude that the difference between amps was slight enough as to generally look at changes in other parameters (like speaker location) instead of amps)!