I have made real world test. Here's one way to set it so it should work, failing this, you may have a bad TOSLink. (If it has a kink, or has previously been kinked, it may be defective).
Make sure if your using Coax 1 for a device, to use Optical 2, just to make sure they aren't a shared ressource or conflict with each other or something. (Just to be safe, I'm not familiar with your AVR's quirks...)
Samsung BDP-1600 and Harman Kardon AVR144.
Component video + TOSLink
Go into Samsung's setup menu.
Set
Digital Output to "Bitstream" or "Bitstream audiophile"
Disable
PCM downsampling
keep
Dynamic compression On
Select surround compatible
downmixing mode