It's not just a matter of the headphones presenting a load to the amp, part of the problem with using the headphone output on most devices is really due to the fact that even on expensive equipment, the headphone stage is an afterthought....using just consisting of some cheap 25 cent op amp.
Almost every set of headphones will benefit somewhat from a quality headphone amp. If you spend most of your time listening to headphones, then this is a good investment (you can use it forever even as you change your preamps). And lastly, if you ever upgrade enough to high quality separates, you might find that they don't even offer headphone jacks.