Don't go over, I think the limit is 512megs of ram, for Windows 98SE. Virtualbox emulates a lower spec video card I believe, but I wouldn't be surprised if you run into problems with Oracle dropping Windows 98se support from Virtualbox. Windows98se expects a max of I believe 512megs of video ram. If there are still virtualbox extensions for Windows 98se, you will want to install them.
Another tip, Windows 98se typically doesn't play well with ACPI. Install with setup /p i I believe to disable ACPI detection. Also, the bios should not support ACPI (Think virtualbox bios).
Some NVIDIA cards have known bugs and known workarounds for them.
I believe that 98se can be patched by an unofficial patch to allow it to handle 2 gigs of ram.