I've recently downloaded all the hotfixes I could get from MS for SP2. I then tried using nLite to integrate them but then it starts trying to do stuff with drivers.cab which fails badly.

nLite says that the drivers.cab file is corrupt but here comes the weird part: If I ignore these errors, Windows XP won't install due to missing drivers and such but if I remove all hotfixes from the list and just use the normal stuff nLite is capable of doing (such as removing things I don't need), then there are absolutely no errors when installing and when using the system once installed.

Anyone who knows what causes this? If it was drivers.cab, I would assume Windows would fail without hotfixes as well.

Have you tried re-copying your source files and trying again? Are you sure that you integrate the right hotfixes and that there are no wrong or obsolete ones? See Tomcat76's hotfix lists for instance via the HFSLIP section of this forum.