Okay, I know you are clear, but let's try to be even clearer. If I understood correctly you are saying:
1) The towers are now set to small.
2) XO at 80 Hz
3) The phase switch pos. does make a difference.
If you can hear a difference by flipping the phase switch with XO at 80 Hz, you are getting bass cancellation. The cancellation effect will be much more audible if you had the mains set to large because the mains will then be getting full range, i.e. down to 20 Hz and below. Since the RTi10 can reproduce 20 Hz, it will cancel the subwoofer's bass if the signals are not in phase.
Two questions:
1) Have you tried setting the mains to large, and flip the 505's phase switch back and forth and see if you could hear a difference?
Yes I have and yes there was some audible difference. It seemed like the sub got louder or deeper/bassier (if that's the correct word)
2) How did the towers sound if you set the mains to large, or listen in pure direct mode, with the 505 power turn off completely.
I tried this and the mains actually sounded pretty good and tight with the EQ off. I listened to the music in Pure Direct with the eq on and off. I need to do some serious EQ tweeking as I have them set way to loud. The music, with the EQ on with my current settings, although louder, was not as clean sounding as with the EQ off. I just found some info on the web that explains each frequency range and the affects it has on music/movies, therefore, I will go back and tweek the EQ settings some more.
The thing is, my Veritas are less efficient than your RTi's, and I have no problem with bass when using the 3805 on its own. The 3805 is not as powerful as a 150WPC power amplifier but it should have no trouble driving a pair of RTi10, especially in two channel music.
One more thing to check, make sure you L&R channels are connected in phase. That is, from the amp output to the speaker terminals, red to red, black to black, for both L&R channels. If not, you will get bass cancellation between the two tower speakers. Good luck!
Yes, they are connected correctly.