You may attempt to force layer acceleration with New Moon. You need to set the property "layers.acceleration.force-enabled" to "true" on the about:config page. For me, I also had to set either "layers.prefer-d3d9" or "layers.prefer-opengl" to "true" for it to work for some reason. Maybe it tries D3D10 or higher and just fails when it's not detected, no automatic fallback? Anyway, OpenGL gave me display glitches, so I went with D3D9.
With that, videos also play fine in YouTubes' fullscreen mode, which is otherwise unusable. GPU is a GTX Titan Black in my case, with the last XP x64 driver, 368.81. Not sure how stable it really is though, I've only played with this for a while, then switched it off again, as I don't really need it. Just wanted to see how I can get fullscreen video to render smoothly, because somebody asked me to find a way to do this with New Moon.