What you should do depends on the materials of which the contacts are made, as well as how much oxidation we are discussing. For example, gold plating can be scraped off if one is too vigorous in one's cleaning.
If there is light oxidation, many times, simply removing them and reconnecting them will be enough. For more, there are various polishes and cleaning products that you can buy. DeoxIT is a popular product for electrical contacts.
Personally, if they do not look corroded, I would simply unplug them and replug them in, rotating RCA connectors as one pulls them off and puts them back on. The same for banana connectors.