I go to Key West fairly often, because I live in Miami and its just a quick trip in a boat down there. Driving isn't too bad either, you could stop at Gilberts for lunch and a beer, its the first stop really in the keys there is a little outdoor bar and a few tables by the dock. Burgers and conch fritters are pretty good. Just don't go down on a Friday afternoon or back on a Sunday, because the traffic will kill you. On a ferry it will be pretty easy, I even think I know the ferry you are talking about taking.
Make sure to eat some hogfish while your down there (its a local snapper, flakey white fish, very very good).
Key West is a drunkfest though, just to warn you ahead of time. People fish during the day, and then get bombed at night/afternoon. If they aren't bombed, they are stoned. It's a fun place to go, but, if your looking for a more calm experience its not always the best. If you want to relax, grab a few brews and cruise around, its a good time. The hotels are kind of dumpy, even the "nice" ones, but, that's just the keys.
