Obviously, there are limits as to what is practical!
The main argument against large drivers (as Slippery says) is "how big is too big" Most don't want to put dual refrigerators in their listening rooms! It also reaches a point where the cost of the cabinet, shipping, and the ability to get it in our house and move it around limits the size.
OTOH, there are indeed limits from a performance standpoint! Speakers would have near perfect imaging if we could buy a single 18" driver that could produce sound up to 20kHz! No worries of crossover artifacts.
I understand what you are saying about moving a lower mass quicker, and I think it is a theoretically valid point, but it seems that in practice, assuming a typical crossover point around 80 or 100Hz, a shorter stroke for a 12" or 15" driver is easier or equivalent to moving a 10" driver enough farther to be at the same SPL.
Martin-Logan probably has some of the best experience on this. They use a 8" driver on their smaller electrostatic panels which crossover around 400Hz. I suspect a 10" or 12" might not do as well at that high of a frequency, but I don't know specifically where the driver becomes too heavy to perform well!