As we have told you already, you AVR RX-A8A has XLR for both inputs and outputs, in order to do what you want to do, use a XLR to XLR interconnect cable, connect one end to the XLR outputs (not inputs) at the RX-A8A's XLR pre outs, and the other end to the XLR inputs at the buckeyeamp.
Of course you can also use a RCA to XLR interconnect cable and in that case you already know to connect the RCA end to the RX-A8A's RCA pre outs and the XLR end to the buckeyeamp.
Both ways will work, but the XLR to XLR way is better for the buckeyeamp because the pre out voltage from the AVR's XLR outputs is higher, 2 times higher.
The buckeyeamp is most likely fine, just how you hook things up is likely the issue, but again, it is better to do factory reset on the AVR first and then start from the beginning.