Damn Shimano has replaced their older technical documents with somewhat relatively useless general instructions, can't find the tooth capacity on the front derailleur but I suspect a change to a 24t could be problematic. Was looking at some of my saved documents for some front derailleurs I've used and the capacity was 22t (diff between largest and smallest rings' tooth count). So maybe a 26t would be the smallest you could go unless you used a smaller big ring....
Looking at the differences between changing to say a 34t which your rear derailleur could accomodate vs changing the front ring, in gear inches (lower is easier to climb with), using
Sheldon Brown's calculator
28 x 32 like you have now is 23.6 gear inches
28 x 34 with a cassette change is 22.2 gear inches
26 x 32 with a front chainring change is 21.9 gear inches
26 x 34 changing both would yield 20.6 gear inches.