Have you tried using just the avr to run Audyssey? Then just manipulate those results (if different) with the app? I have faint memories of some issues with early versions of the app in terms of levels but its been a while.
How low are you setting gain on the subs? The -12 is indicating the avr is at its limit of turning the sub level down, that it is set too high on the sub.