First, true five channel DVD-A will not work through a standard optical or coax digital connection. Not sure what your other player was doing, but best bet is that it was finding the Dolby Digital tracks.
Take a look in your player's menus. There is likely a setting for DVD discs that tells it whether to use the DVD-V section, or the DVD-A section. If you choose DVD-V, your digital connection should work.
Once you get a set of 5.1 analog cables, then go back and reset the player for DVD-A.
While some may disagree, there is no need to blow a wad on cables. Any regular red/white audio cables will work fine. If you don't have anything, then you can buy something at Radio Shack (stay away from Monster!), or on-line at Parts Express (I like their house brand; Dayton Audio).
Brian