Speaker systems are designed as a compromise between the enclosure size, the particular drivers (particularly the bass drivers) and the crossover to assure a smooth transition between them. Much enginerring is done to asure the proper balance between them to get the best sound from those three components.
Generally, replacing individual rivers in a system doesn't work out too well but "some people" love it simply because it causes a "change" in sound. This is true but there is a difference between a "change" and an "improvement". ...and "some people" can't tell the difference.
PS. Among other things to consider, you do realize the Polk has 5" drivers and those HiVi's are 4", don't you?
