The RCA output from the receiver does not have any power to drive a speaker, it is signal only. If your sub is not self-powered, then it can't do anything with the signal you have provided to it (not a good idea trying this, BTW). If the sub has a R&L speaker level (wire) inputs, you can connect it directly to your R&L speaker outputs, then in the setup menu in the receiver, set your main speakers to "Large" and it should work.
If the sub only has one + and - input, or if you want to use your new receiver's bass management capability, you can connect the RCA to an input with a Y connector to an old/cheap receiver or amp and power the sub with that instead. This will keep your main receiver from working as hard to drive the sub itself.
A line level to speaker level adapter could also be used, but if the sub isn't powered, this won't work either.