The list price of the Yamaha R-S700 when HTHF reviewed it in 2011 was $549 so it might be cheaper than the OP's cheap Audio Source amp!
Having said that, the R-S700, based on what I saw in the SM and Dr. Rich's comments, I would consider it a better amp that the current A-S701 and A-S801 integrated amps. It is the only "cheap" 2 channel Yamaha receiver that I know of, that uses balanced input/output in the preamp section. It also has pre outs and sub outs and the thing can be used as separate preamp or power amp.
I wish Yamaha had continued the same approach in their design of their <$1,000 amp, instead of those AVR derived A-S701. So I agree with you that the OP should remove the Audio Source amp, though that may not likely solve the problem, but it could at least narrow it down to whether the issue is the Audio source amp or the Yamaha receiver. If both are trouble free, the Yamaha on its own should perform better sonically, in theory...