You're exactly right.
It puts more load on the blower motor.
Over the years I've spoken to every HVAC guy I see on various jobs, and they all say the same thing.
Use the cheapest filter,(read least restrictive) just change them more often.The other issue is that many AC service calls stem from frozen "A" coils due to reduced incoming warm air.
Also, running without a filter would clog the coils with dust reducing their efficiency.
This works out perfectly in the Boring Thread.