I think the answer is each individuals personal experience. I do a lot of critical listening with other people, either with the local guys or at shows, and my evaluations are quite often in line with others listening along as well. And in setting like that we all share similar past listening experience and histories. But when I have friends over and they listen to my designs, people who aren't audio guys, the ways they describe what they're hearing often seems strange to me. Their point of reference is so different.
Before I got into building speakers I had quite a few that I though were clear, detailed, smooth, focused, etc ... some Wharfedales, Paradigms, Energy, Mirage, and a pair of Dynaudio monitors. It was literally like flipping a switch when I finally did put something in the same room with them that WAS clear, detailed, smooth, and focused. Suddenly I realized just how off I was before, but I wasn't wrong for thinking that way before, I just didn't have a frame of reference to show me otherwise.