I listen at about the same volumes, averages in the 80's, peaks mid to high 90's, and that seems pretty loud to me. Every once in a while I'll crank it to realize the full dynamics of a good recording (think of really bombastic Mahler or something), and for those times the loudness doesn't seem excessive, even on the super loud crescendoes. I listen to blues and rock at louder average volumes, but the peaks aren't tremendously greater than the average. I rarely listen at average levels >90 for more than very brief periods. And I use earplugs at amplified shows, just can't stand 100+db levels for very long. Damn, that makes me feel old. :/
You don't need excessive volume to get extremely pleasing sound, and you can listen all day at moderate volumes. It's nice to have the capability to rock it, however, for when the urge strikes.
My issue with the woman is getting her to stop blaring that damn Dave freakin' Matthews at concert levels.