Does your AVR have an ARC HDMI port? If it does you can connect the Smart TV, provided it also has an ARC HDMI port, to the ARC port on your receiver thus negating the need for a Toslink cable. However, if you do not have a ARC port on both devices, and want to use the features of a SmartTV over those provided by the BD player, then you will need a Toslink cable connecting the digital out from the TV to the AVR.
Having both devices, a SmartTV and 'Smart' BD player, will not cancel each other out. Personally I moved my BD player into our sons bedroom since he does not have a smart tv so he can watch Netflix Youtube etc. Our living room TV is internet connected as well.
I believe the newer Samsung SmartTV's have HBO-Go, and if you want to use that feature then you will want to use the internet connected TV over your internet connect BD player. There may be over differences in the apps available between the two as well that may influence which you prefer using if both are located in the same room.
Also, if you do not want to spend extra for a smart tv, I believe Google sells the Chromecast for about $35. You plug it into a USB port on your TV, connect it to your home's wifi, and boom, you have a 'SmartTV' with HBO-Go, Netflix, Hulu etc etc etc.
Hope this helps.