FM has limited frequency range. FM is not CD quality. You would get better quality with a direct connection.
I use an FM transmitter in my car with my MP3 player because I saved $20 when I bought the head unit and got the one with no auxilary input.
It does work well though and the sound quality is decent though I would definetly use the direct inputs if I had the option.
I do have a min-jack to RCA cable hooked up to my receiver and I just plug the player in and it sounds great. With WMA/MP3's at 320k and lossless songs the sound quality is equal to the original CD.