Max output mode will give you more output from 25 to 40 Hz. Max extension more will allow the sub to dig down to 18 Hz. I would say if you really want to crank the subs hard, use max output mode. So if you want to impress friends or just rock out, that is what you want to set it in. However if it is movie night and you want to watch something that has deep bass, set it to max extension mode.
One thing to remember is make sure a port is plugged with the EQ switch is set to EQ1! When the VTF-3 is set to 'EQ1', the driver is behaving with the expectation of more backspring pressure at low frequencies (where the rear side of the cone is met with air resistance due to the enclosed nature of the cabinet). When you open up both ports, there is less backspring pressure and so the driver is able to move around much more freely at low frequencies. The 'EQ' switch is partly a filter that prevents the driver from bottoming out when both ports are open. Bottoming out the driver can damage it, and it is not cheap to replace- and may also not be covered under the warranty since it pretty much takes misuse for that to occur. This is explained in the manual, but I think it should be emphasized. When you learn how the sub works, you can use it to its potential much more easily. This sub can be pushed hard with no problem- just make sure the settings are correct before you let her rip!