We watch the 55 from the bed straight at the center of the screen. At a 5' distance we would have to set the TV on the bed itself. I think we are OK.
No question, this is okay, because it is what it is and it works for you.
I think there is a huge difference between having a TV which is the focal point of a space, and a TV which is casual viewing as an addition to a space.
We have a similarly sized 52" (or so) TV in our master bedroom. We watch it casually. Folding clothes, lounging about, etc. We don't do serious watching on it. We do MORE serious watching in the family room. Watching broadcast TV and some sports. Maybe a movie or two. That's on a 85" TV we view from about 15' away. It is too small. Not even a question of opinion. It's nice, it's viewable, but it is NOT immersive. It is acceptable, and better than the 65" we had there before, but the room is large and the TV is about as big they get within a reasonable price. The $20,000 100" Sony TV would be a great direction if I had bottomless pockets.
But, our basement hosts a 161" front projection setup. We sit about 16' away from that. It's about four times the area of what we use in our family room, and it is where we go for movie premieres. When the impact and quality really needs to be there, this is where we go. And it delivers. It's in line with THX specifications and gives the immersion of a true movie theater without the sticky feet or missing the best part because of a bathroom run.
Should you have gone up to a larger size in your family room? Yes. Yes, you should have. When you can, you should. But, only if you want to. Which is what it still comes back to. While I can point out and say, with absolute certainty, that you should have a larger TV, it is not me who has to live with it.