You could, but imo you shouldn't.
I don't own the device, however my educated guess tells me - no.
Since the icon next to lime port on x-fi shows headphones, this usually means small amount of amplification is added to drive passive speakers such as headphones. That means, in addition of very likely source of SQ loss - guaranteed different volume level, than rest of the channels.