Eagle covered some of what I was gonna say. Bass response can make speaker A sound better without a sub, but speaker B might sound better with a sub in the mix.
Another thing that's worth noting is speaker sensitivity. If speaker A is more sensitive it might sound better than speaker B if you compare both using the same setting on your volume dial. The way we hear a slight difference in volume isn't always obvious, but it's enough to skew perceptions. So it's good to level match with a spl meter so you know which volume setting will be the same spl switching between speakers.
I gotta say, you're going about this the right way, and it's good that you recognize how short audio memory is. You do forget pretty fast. If you have some way you could instant switch that would be best.