Well, DACs are rated with a THD+N spec. So they are theoretically imperfect. Amp transistors are not perfectly linear. Then there's crossover distortion - just to name a few examples of imperfection.
The question is not whether audio components lack distortion, because obviously they do have distortion.
The first question is whether two audio components sound different. There's no reason to insist they can't, when they are imperfect.
What people have troubles proving is whether audio components sound different enough to tell them apart. A number of amplifier listening tests demonstrated that people, even people who thought they had golden ears, simply could not reliably tell amps apart.
So, that leaves us nowhere really. People will tell you they hear differences, and other people will them to prove it.
The truth probably lies somewhere in the middle. Audible differences seem likely to exist because audio components are imperfect. But they seem to be subtle in any hi fi gear. I am sure most of us could tell a transistor AM radio vs a stereo playing into decent speakers
When people tell me they switched out a CD player, pre amp, amp etc. and were blown away, I refuse to believe they are being honest with themselves. If that was the case, we should be able to set up unsighted listening tests and easily pick out components. When I read up on such tests, people admitted how hard that is.