500W for 1db! - talk about diminishing returns! (not that I'm in that extreme scenario)
My opinion on amps is this
Every doubling of power yields 3db more.
So from 1w to 2w, you gain 3db
From 1w to 128W, you gain 21db.
from 128W to 256W, you gain 3db <--- this is where the diminishing returns start to hit! Not only that, it's also where speakers' thermal management starts to become a problem.
In my opinion, your speaker "should" be close to 93db/w/m efficient if you need reference level 105db peaks at 10 feet away. For a 4 ohm speaker, that actually means a high sensitivity at 96db/2.83v/m. Less efficient speakers will only convert more power into more heat. but yes you can probably get away with ~90db/w/m efficient speakers because few of us really watch movies at exactly reference. 102db peaks at 10 feet is pretty good for 128 watts. But most 90db/2.83v/m speakers are 4 ohm, which means they're actually 87db/w/m. the JBL LSR 6332s are 90db/w/m or 93db/2.83v/m
..... if your system ever sees benefit from a kW amp, then your speakers are probably at fault.
It's tough for an LFE subwoofer though, because of the 10db higher headroom requirements. So for a single 93db subwoofer, you would need 1280 watts!

And if the subwoofer reaches down to 20hz, 93db efficiency is beyond optimistic. The only drivers I know with efficiency like that are expensive 21 inchers like the B&C 21sw152 and Mach V FTW-21 - and if THESE drivers need over a thousand watts to hit reference on their own, you can only imagine the 10 and 12 inch sealed subs with smaller magnets and attempts at deep, low extension!
So overall I think you're right - we really should NOT need kW amps for our main speakers. The speakers either won't handle it without compression of the signal, or they simply won't need it due to their efficiency.