I do not have an external graphics card to test your theory. The integrated graphics circuits I have can't be removed from the system, because they are a subcircuit of the chipset or the processor.
If the vga.sys driver is still blocking the system from going into sleep because it's still running in the background, is it still possible somehow to remove the blockade?
Edit: I think what I need to do, is basicly to find a save and clean way to unload vga.sys and make sure it gets loaded again if my driver gets unloaded.
Well, thank you @jumper. I think you gave me the crucial information I was missing.