The biggest differences I've noticed was when upgrading earphones, headphones and bookshelf speakers. I remember starting out with 50$ Sony in-ear headphones and when I moved to Shure SE420's a few years later, YES, the difference was véry obvious. The same happened when moving from a Logitech Z-5400 surround multimedia system to a Yamaha integrated amp with ELAC B6 speakers and a Klipsch subwoofer. And again when moving from a Sennheiser gaming headset to Shure SRH1540 headphones. And last but not least when moving from the Shure SE420's to Shure SE846 in-ears.
But when I use my relatively high-end SE846's and SRH1540 to try and find a difference between (decent) amps.... I... just...can't ...tell.... And I've tried hard. As a matter of fact, I can't hear the difference between my dedicated portable DAC/AMP Oppo HA-2, Yamaha AMP headphone out, Schiit Modi Multibit DAC + Magni 2 amp or iPod 6th Gen, except of course for the difference in output power. And any difference I believe there is, is nullified when I'm blind testing.
I do consistently hear the lesser sound quality of my Samsung Galaxy S5 smartphone, where audio was clearly an afterthought.
My personal conclusion is this: save on DAC/amps and spend on speakers and headphones. My Yamaha amp was +/- 500$ and that is the absolute maximum I would ever spend on an amp. But speakers/headphones, I'd definitely spend 1000$ or more if they get rave reviews (think SVS Ultra bookshelf speakers, or the upcoming Sennheiser HD820 headphones for example),because that you can REALLY hear.
Don't even get me started on the 'difference' between 320kbs MP3, 16bit/44.1Khz WAV and 24bit/96kHz WAV (hint: for 90% of the people, there is no difference, including myself).
Oh, and one more thing that is often overlooked in these discussions: the importance of an acoustically treated room, and proper physical setup of speakers/woofers. A 20.000$ dollar system listened to in a bunker is going to sound a lot worse than a 500$ system in a well treated room.
Glad I got that off my chest
PS: I'm an electronic music producer, if that means anything.