Depends. If you’re ear height isn’t more than 10-15 degrees above the center it shouldn’t be an issue. If it doesn’t sound wonky to you and things can move from left to right or vice versus across your center stage it’s not a problem. If things don’t sound “off” then there’s nothing to fix.
Yes, ideally you’d want it the same height, but within a certain +- degree it shouldn’t make a huge difference.