I found that my monitor would do as you proclaim 1280 x 768 was fine though and 1920 x 1080 was working but in between had sync problems. My big screen TV will not sync at all, it momentarily goes blank then says no signal, so it is close to working but doesn't. Plugging in another monitor and it is fine on 1920 x 1080. You might be able to try 1920 x 1080 on your monitor?
If your monitor is not syncing at higher resolutions with a nVidia card, then go into nVidia display panel menu to the nView Display Settings; Device Settings; Device Adjustments; and select "Display adapter scaling", before adjusting up the resolution. If it is already selected then try selecting "Monitor scaling", but I have to select "Display adapter scaling" every time I install a nVidia card.
If anyone has got a new AGP AMD video card, any later than X850, needed the reverse side AGP interface "ralo" (not sure of its name now) chip heatsinked. This is necessary to prevent motherboard damage and of course the "ralo" chip from destroying itself. I have glued a strip of 3 x 25 mm copper on mine, onto some plastic packers to keep the correct parallelity, full length of board, painted black.