Buck, while the gist of what you said is correct, you have the sensitivity numbers inverted.
IMO sensitivity is still important. Take for example a speaker with sensitivity of 88 db/w/m (fairly typical of many speakers these days) and compare to something with sensitivity of mid 98 db/w/m (this would necessarily be a large speaker with low WAF). For normal folks, listening at 70-95 db range, the first speaker will require ten times the power as the second across the board, which may come into play if you're using an AVR with limited dynamic reserves, where the second would demand a watt or less at all times, leaving copious amp headroom available should the need arise.
It starts with drivers that are engineered to be more sensitive to begin with, such as using more powerful magnets, narrower magnetic gaps for the voice coil, light weight cones, etc. On top of that there are other methods to increase sensitivity such as horn loading. Sensitivity tends to come with lots of girth, due to Hoffman's Iron Law as mentioned previously, if the speakers have any pretensions of providing extended bass.