As with any sound, it's about SPL and duration. Your hearing can tolerate momentary peaks at higher levels without damage. I would think traditional sub high peak levels are operating under the idea that it's ok because it's not sustained for more than a few seconds. That does make sense. Sustained levels in a lot of music is higher up the frequency band, but not all. It depends on the content. One loud explosion is probably fine. Play some bass heavy music that's pretty consistent at 110db for an hour or something and it's a different story.
The study results I looked at before posting earlier cited 63Hz at 90db for several hours caused significant hearing damage across a wide range of the hearing range. Assuming that information was accurate, loud bass is dangerous for your hearing too (possibly worse), just like prolonged exposure at other frequencies. But really, that just seems like common sense that any thing at extreme levels for too long is going to be bad for your ears.