In most cases, you are correct. However, some low cost pre-built speakers, though rare, have superb quality drivers and good crossovers, but are held back by very poor cabinetry systems. One such example is the Primus 360. It has a superb driver set and an excellent crossover. On and off axis response is superbly linear. Energy decay of the drivers is superb. The cabinet, however, is a pile of horse doodoo. Like most cabinets, the thing is resonant, and if that is not enough, it does not even use acosutic dampening material internally that is sufficient to absorb reflections/standing waves in the bandwidth of relevance. You could go so far as to put lesser quality drivers in a high quality cabinet, and end up with higher quality sound as compared to higher quality drivers in a low quality cabinet. The cabinet has that much influence. Ideally, the cabinet should have no influence(be neutral/inert) overall, but that is not the case with the vast majority of speaker systems, and that includes high cost 'audiophile' speakers.
-Chris