It will not damage the receiver if it has been running that way for years. Likely means you don't listen loud enough for it to be an issue, which may be your saving grace. Also, if they are the only pair of speakers in the system, the amp isn't seeing as much load.
Very few receivers can actually handle 4 Ohm speakers. Pretty much all of them will work, but if you expect high output, then you'll need something on the heavier duty side. Amp might be advisable in your case. I had high sensitivity 4 Ohm speakers and even a very hefty receiver wasn't enough to handle them, though I was also pushing 5 of them. I had to initially offload the front 3 to external amps and eventually all of them.