If your ears hurt, or you hear ringing in your ears, you are listening too loud and are damaging your hearing. As for how loud will damage your ears, it depends upon how long you hear it. Anything over 85 dB is
known to be able to cause permanent damage if it is long enough. See:
http://www.cdc.gov/niosh/topics/noise/abouthlp/noisemeter_flash/soundmeter_flash.html
One thing to remember in this is that hearing loss may not be noticed right away, and may only cause trouble for you later on in life. So it can be difficult to tell exactly how loud is actually safe. It may be that slightly less than what is known to cause damage may actually cause damage, but due to the delay between exposure and hearing loss, it may simply not have been documented. One never proves that a particular level is safe; it is only proven that certain levels are harmful, and no harm has yet been noticed for less than a certain amount. The same, by the way, is true of carcinogens consumed, damage from ingestion of poison, etc. So if you want to be actually safe, you probably want to not push things to the limits of what has been known to cause damage.
Also, how much damage may occur is not purely a function of how loud the sound is. The particular frequencies matter a great deal in how much damage the sound does to your hearing. For OSHA standards relative to frequency, see the graph at:
http://www.osha.gov/pls/oshaweb/owadisp.show_document?p_table=standards&p_id=9735
One thing to notice is that very, very deep bass is less damaging than midrange frequencies, so turning up a subwoofer too high and making the music boomy does not pose the same hazard as simply turning up the volume to achieve the same level of bass. (Of course, the overall volume would be higher that way as well, due to there being the same amount of bass and simply more midrange and treble. And turning up the subwoofer excessively makes the music sound like crap, but quite a few people like things to sound like crap.)
As highfigh correctly points out, quite a bit of music is at a fairly constant level. Most popular music is not very dynamic, and consequently the peaks are not usually much higher than the average SPL.
If you want to read a fairly lengthy document on this:
http://www.cdc.gov/niosh/docs/98-126/pdfs/98-126.pdf
Judging from what it states in that document, it would appear that, for periods longer than 8 hours, sounds softer than 85dB may cause damage. And one will find that following OSHA standards does not guarantee no damage; it is only reducing the possible damage to what they consider to be an acceptable risk. So if you want to actually be safe, you should keep things softer than OSHA allows in the workplace.
