Here are the biggest contributors to sound quality
Speakers
Room (includes speaker positioning in the room)
Source recording (quality of recording engineering is biggest factor)
Then much farther down the list, assuming the conditions mentioned by PENG are met, is difference in amplification.
Why do I believe this? I once went to lengths to do a proper comparison of two amps (a pro audio amp vs. a consumer audio amp) in the same room with two pairs of the same speakers. I fed the amps using 2 identical avr's so I could level match and quickly switch between them. I did about everything I could think of to make all things equal. When I switched between amps, I DID hear a clear difference and got pretty excited! However, to be thorough I switched speaker connections and found that the difference in sound was between the speakers, not the amps. Since these were Identical speakers, I investigated further and found it was the position of the speakers that made the difference I was hearing! I had the speakers sitting side by side with about 10" between the tweeters and that 10" difference in location had a greater audible effect than any other factor!
So, no offense, but based on my experience, I am inclined to believe if you have heard a clear difference between two amps, they were most likely with other (much more audible) factors changed, such as different rooms and/or different speakers... of course, they would sound very different!
Of course, the other big factor is psychology! If I sell my $500 Yamaha stereo amp and replace it with some lovely McIntosh mono-blocs with the big meters on the front, will I perceive a difference? Damn straight! We are human, and unless such a comparison is done blind, it is inhuman to be able to ignore the psychological factors.
One of my favorite analogies (assuming you are a typical heterosexual male) is there are two women who are identical in the way they kiss. One is your ideal image of beauty, the other is not very attractive to you. Would you perceive a difference if you kissed them both? Of course you would.
Which also leads to the question of whether or not you would actually realize greater enjoyment out of a different amp even if it did sound the same. I think the answer may well be yes. but if you are really looking for an improvement in sound, spending time tuning things in your room or switching speakers are your best bet.