There are quality differences in amplifiers, whether they are driving loudspeakers or headphones. Exactly how far you need to go in that regard is mostly a personal preference. Like almost any audio application, sometimes you need to be exposed to a truly outstanding amp and appropriate speakers or 'phones to know that there is, after all, something you might be missing.
Certain types of headphones are more demanding than others. In-Ear Monitors tend to require extremely low output impedance from the headphone amp, a feature that is far from universal. Electrostatic headphones require some interface that differs from conventional amplifiers. Generally speaking some headphones are more forgiving of drive quality than others, so it also is somewhat device specific.
People who prefer some rather exotic headphone amps often use them as their primary listening device. I like a good headphone but prefer in-room reproduction. My best 'phones are IEMs (Ultimate Ears 4S, no longer available) and I have a couple of AKGs over-ears that are decent but far from outstanding. For open backed phones I like Sennheisers, but don't own any. If they were more comfortable, I might have bought some STAX electrostatics over the years, but I find them too heavy. Sound great, though.
There is a staggering amount of choice in the market today, but there is also a tremendous amount of market diversity ... many phones are not intended for the "Audiophile" crowd. Beats sells more than half of all "serious" headphones sold in the world*; the rest of the market is divided amongst dozens of manufacturers.
Yet many of those offer products that sound better than anything Beats has in the catalog; as always Sound Quality is rarely the number one criteria if you go by what people fall off their wallets for, even though consumers consistently claim that it is.
* models selling for $100 or more
