I can definitely tell a difference between 128kbps (the original default for iTunes Store songs) and 320kbps when I listen on a good enough system like the one in my living room. On my iPod with the original headphones, it's tough to tell a difference.
The iTunes Store started selling higher bitrate copies (called iTunes Plus, I think) a while ago, and that's probably why you have songs with higher bitrates from there. I used to have to really, really want a song before I'd pay even 99 cents for a 128kbps song from the iTunes store, so I was quite happy that they finally started offering higher quality versions.
