Any time you have an impedance mismatch, whether it's electrical, optical, mechanical, or acoustical, you'll lose energy. That's why light reflects off water--much of the light is transmitted, but due to the impedance mismatch between air and water, some light gets reflected. Because energy must be conserved, any light that is reflected is energy that is not transmitted. And it doesn't matter whether you're going from high impedance to low, or the other way around, energy is always reflected at the interface of an impedance mismatch. So, using the water and light analogy, if you go under water and look up, you'll see that light gets reflected in that direction too.
To offer another analogy: tie a heavy rope to a doorknob and then tie a thin rope to the end of that heavy rope. If the ropes are very mismatched and you try to wiggle the end of the thin rope to make a sine wave, you'll notice that the fat rope hardly wiggles at all, because all the energy gets reflected back at you when it hits the thin/fat interface. If the ropes are closely matched in thickness, all the energy will make it to the doorknob.
Hooking a 6 Ohm speaker to an amp with an 8 Ohm output impedance should let most of the energy through, but there should still be significant loss. Effectively you'd have to turn the gain up higher to get the same amount of energy through the speaker. And I'm not sure how amps react to energy coming into them from the output. I understand the physics of some of this stuff, but I have no sense for what my ears will hear. Also seems like you could burn up an amp with an impedance mismatch.
Are my physics books too much about theory and not enough about reality?