This is a tough question to answer.
Growing up my family never had the best stuff. I never had surround sound, or even a big TV.
In the past I would have jumped quickly to say that I would skimp on audio, because I never had great audio and I was all about seeing things bigger and better.
Well I bought my HDTV about a year ago along with a Blu-ray player and was amazed at the quality and detail of the picture (and I now had a big TV that I was happy with). It didn't bug me that I didn't have a sound system, I was perfectly happy with the picture.
Then, I had friends starting to get 5.1 packages, so I started to look around. I knew that I didn't want an HTiB because they weren't all that great (at least the ones offered in the big box shops, BB, CC, Fry's, etc.). I had a budget in mind and I listened to everything around that was within my budget and eventually bought my system. After setting it up I was amazed at how much of the experience I was missing by not having the sound. The house shaking bass, the crack of the guns, the clarity of instruments, the sounds going on all around.
Now, I don't know which one I would skimp on. Of course with my system I bought everything at separate times (about 6 months or so apart) and found the best that I could within my budget for each of the components.
I would want to say that if you don't have a lot of money that is probably the best way to get yourself a quality setup. Just set money aside for a while and as you have enough for certain components, buy them. Though, when thinking about this process and if I were to start at nothing, I would most likely buy the TV first.