It's been done (many times). BRM ("Meridian")* probably being the best example ... the system they offered in the late 1990's was digital right to the individual loudspeaker driver, and employed proprietary switch mode amplifiers in the speaker enclosure that accepted a digital signal. They've offered products such as a $USD 20,000.00 DVD player.
Digital State-Of-The-Art is expensive, and depreciates rapidly and to a low bottom limit (5~10% after 5~10 years). That is much higher than audio equipment in general , at least up to today. You can barely or even cannot give away those Meridian systems today, let alone sell them to recover some of your investment.
You had better prefer a digital ecosystem, because you are basically saying goodbye to analog aside from the original performance and your listening, which are analog and always will be, forcing the two language translations (A-D and D-A).
That's not to say you can't use, say, terrestrial radio as a source, but you are inserting additional ADDA steps that are un-neccessary with the pure digital two-translation signal flow you have essentially embraced with your choice.
Translations from one "language" to another introduce errors that a system that avoids such translations does not have to deal with. Whether that is a problem or not is specific to the situation but it can never be perfect, the very best that can be achieved is "good enough to not matter", and that may or may not be achievable.
Further complicating matters is there are likely to be moving goalposts ... for example the vacuum state amplifiers of the 1970's were equaled by transistor amplifiers of the 1970's in ways that satisfied most audio enthusiasts, but the vacuum state amplifiers of 2018 are not those of the 1970's. So the goalposts have moved.
Because Digital involves a state change (aka language translation) it too must be able to continue to improve to equal the analog of 2018 and the analog of the future versus the analog of 1983 (the first CDs) or the analog of 1977 (the first all-digital recording chains suitable for quality audio). This should not be confused with potentially artificial tech advancements (eg THX certification, or MQA audio) which exist mostly to drive new sales. They may, or may not, be actual improvements, but they do serve to make existing digital ecosystems obsolescent.
I think that anyone new to Audio is well advised to consider an all-digital signal path, as it offers advantages likely to appeal to those new to audio, in particular compact size, low cost (hardware; the software is expensive as all rents are) and convenience.
I think digital of 2018 has reached the same point that the transistor electronics of the 1970's reached ... for most people, the value is there (by that I mean consumer level gear offers better Sound Quality than they need, at a reasonable cost).
But nobody is going to be looking to buy your iPod Classic in 30 years, unlike the 1970's MacIntosh Amplifier, which actually has a chance to be a collector's item in 2048, so don't expect your all-digital choice up to the loudspeaker driver to be the lowest cost approach. If you get tired of replacing your throwaway digital gear, analog might still be an option many years from now.
* Meridian invented encoding/decoding technology used in DVD-Audio, Dolby TrueHD and Blu-Ray, and they are also the developers of MQA.
Note:
obsolescent: surpassed by newer technology
obsolete: made useless by existing technology
There is a significant difference between the two, and they are often used incorrectly. We might be seeing the advent of electric vehicles making internal combustion vehicles obsolescent, ironically after IC vehicles made EV's obsolescent 90 years ago. Neither is obsolete as they both still are effective.
Similarly, the iPhone that was introduced in 2007, (the same year the iPod Classic 6th Gen was introduced) is obsolete ... it won't even make a phone call in most parts of the world as the cell network tech it supports no longer exists. The 6th Gen iPod, on the other hand, is obsolescent ... you can duplicate it's function with a modern phone, but it still works just fine, and in many ways superior to a modern cellphone, as far as listening to stored music goes. It is not obsolete simply because it hasn't been manufactured for 4 years; or because it's not a cellphone itself, that's not what obsolete means.