First of all, I am a video noobie, and am unfamiliar with the PS3, but...
Is it set to output at 24 fps?
If so, is your display capable of handling 24 fps natively? (no pull-down applied, or no 60hz display, etc). I doubt it.
If not, disable 24fps, and see if the judder is smoothed out by avoiding possible "pull-down" issues or otherwise.
Good luck.