Should I put a larger screw in the existing grommets? Or start from scratch and re-install moving the brackets?
Grommets are the silver rings that the curtain rod goes through.
I'm trying to imagine what is happening and guessing that the framing of the window is similar to the image below. The prefab window assembly gets inserted into the framed hole for the window. I'd guess that your first mounting attached to the solid wood header above the window and you second mount went above it so you are only attaching to sheet rock unless you got lucky and hit one of the vertical studs..
With the right hardware, sheet rock attachment can be sturdy enough for curtains:
These are good for 80 pounds in 5/8" sheetrock, but don't do pull-ups with them! If you think about your rod bracket and if you pull down where the rod sits, you have a leverage situation not to far from pulling out a nail with a hammer! Using typical dimensions, If it is 2" from the upper wall screw to the bottom of the bracket and 4" from the wall out to where the rod sits, twice the force of the weight of the curtains and rod is applied at that upper wall screw! So the 80 pound strength of the anchor is met with 40 pounds worth of weight of curtain and rod. Unless you are using weighted curtains, you should be good! (however, if you go to HD or Lowes, you will see several anchors rated above 80 pounds).
Pop toggles work with various diameter sheet metal screws. They are ideal for anchoring into hollow base materials such as drywall. They are ideal for hanging electrical fixtures, drapery supports, bathroom/kitchen
www.homedepot.com