Well, an amplifier usually has a fixed gain. Which also means a fixed noise floor. You hook up the amplifier to your speakers, and depending on your speaker sensitivity and the noise floor of your amp, you either hear something or you don't.
Likewise, your pre-amp has a noise floor. Some of the noise may occur before the "volume stage". So when you turn up the volume, you turn up the noise as well. Some of the noise may occur after the "volume stage" and so turning up the volume doesn't change anything.
If you have a pre-amp with audible noise before the "volume stage" then a lower volume setting on your pre-amp would be good. In which case you may want the signal going into your pre-amp (i.e. from your DAC or CD player) to have a higher voltage if you like to listen loud.
In my experience, typical single-ended RCA voltage from a CD player is around 2 Vpeak. Still, usually people end up sending less than a volt into their amp because that's often plenty loud enough already. In other words, picking a higher voltage level isn't likely to be necessary.
Of course, just like for the pre-amp, your CD player or DAC will have a noise floor. But you have no control over that, other than getting a different source. Same as with the amp; if the amp noise floor is too high for you then you need to get a new amp. If the advertised specifications are reliable, then you want to look at the signal-to-noise ratio.