They're actually pretty simple, but some people like to make them out to be really complicated. It needs a base that dampens vibrations, whether produced by the motor or something outside of it. The tonearm also needs to be free from vibrations and should be light enough to react to any undulations in the vinyl. The motor needs to keep stable speed and not cause audible vibrations.
The cartridge is a special exception. That is all-important in several ways. The compliance needs to match the mass of the tonearm and its resonant frequency. If you use a high compliance cartridge with a heavy tonearm, it will feed back and will never track warps. If you use a low compliance cartridge with a low mass tonearm, it will have its own problems. Think of the compliance as the spring rate for shocks on a car.
Cartridges don't sound the same, from brand to brand, at all. You'll need to listen to some in order to find what works best and in the age of online shopping, you'll have a hard time doing that. If you audition equipment at a local shop, at least have the courtesy to not just use them for the listening test and then buy it online, based on price.
I got a turntable for a customer a couple of years ago (my suppliers don't really sell much phono equipment) and it wasn't very expensive. It's a Pro-Ject, distributed by Sumiko. I get the upgrade cartridge, too. It sounded very nice and it was easy to not pay attention to the fact that it wasn't a CD/DVD playing. You should be able to get the same combination for less than $500, too.