This is my first modern A/V receiver. I was using a Marantz 2-channel receiver (SR-4023) before. I only have one subwoofer for now, but can I use this Audyssey thing to set up for 2.1 or 2.2 channel in regular stereo mode? I not in any hurry in getting a full surround set-up at the moment but it's nice to have the options.
You can run 2.0, or 2.1. You can run dual subs as well, but it will not be 2.2... each sub will NOT receiver a discrete signal. This may impact phase negatively unless you deal with that ahead of the signal. You may need to play with setup a few times and keep doing it until the phase match between the two subs works.
When running auto setup, you normally want the microphone to be close to ear level or tweeter level so that it doesn't take vertical off axis measurements (where few speakers truly excel). A silent room is ideal for setup (and in general). It's best to do it at night so that nearby traffic noise etc doesn't interfere.
In order to use your subwoofer for music, you need to run the setup first, and if it determines your speakers have extension to 60hz or so then you will need to manually switch them from the automatic setting of "large" to "small". Doing so will allow you to cross over to the subwoofer.
When the receiver is in source direct mode, the speakers alone will receiver a full range signal. When you switch to "audyssey" it will apply additional equalization filters which may or may not improve your sound quality. What I do is take the Audessey results and apply them to where it starts adding boost/cut near 400-600hz, and simply use "preset EQ". Preset EQ also incorporates the sub, although I don't believe any filters can be applied to that channel (though my receiver is older).
In source direct, the sub will still receiver discrete LFE content, but not matrixed content from two-channel. Otherwise, yes, the auto setup of Audyssey XT will set speaker trims and subwoofer output levels.
Audyssey will also determine a crossover point for you, but this may not be optimal. The tests Audyssey takes are at 75-85db or so, but real content can have dynamic peaks as high as 105-115db. So while audyssey optimizes for regular volumes, it doesn't take into account that distortion rises at higher volumes. So you may find that if audyssey sets your crossover point as 60hz for example, that 80hz or 100hz is more optimal for your particlar main speakers so that they do not have to dig as deep at high SPLs. a two way bookshelf for example can hit 60hz, but it isn't likely going to do that at 100db! Crossing it at 100hz will give you a cleaner sound at higher SPLs.
Never go LOWER than what audyssey recommends, only go up. Also as you go higher the crossover effects (especially localization) become more and more audible. So finding the balance between mains and subwoofer is still YOUR job, although audyssey makes it easier. The filters it applies to the sub channel may also improve sound quality, as the resolution of these FIR boosts and cuts is extremely high. Play around with it, it won't be too hard unless you can't navigate through menus.
At your subwoofer level, make sure to defeat any onboard crossover settings or set it to the highest possible value.
Also, I'm sorry to hear you spent so much for that receiver... Accessories4less has the same receiver for $650 brand new, from an authorized dealer:
http://www.accessories4less.com/make-a-store/item/MARSR6005A/MARANTZ/SR6005-110w-X-7ch-3-D-Home-Theater-Surround-Receiver/1.html
And even less for a Factory Refurb:
http://www.accessories4less.com/make-a-store/item/MARSR6005/MARANTZ/SR6005-110w-X-7ch-3-D-Home-Theater-Surround-Receiver/1.html