As long as the amp/speakers are not over driven, there will be no distortion. Remember, I went through a practical life experience with this.
To add to this, in that same house that I sold, I forgot to turn on the center channel and I only happened against it through the menu setup of the AVR after re-running YPAO and looking at the results and I found no setting for the center channel. I must have been missing the center channel for about a month without noticing it not being there.