HFSLIP documentation

[berntie]

Hi all!

First off, be warned: Long post ahead. I thought it would be helpful to give you some details about my motivation to make sure everybody understands what I'm talking about. That's why it has gotten so long.

Second, Hfslip rocks! It does what has to be done, but no one else does.

However, I have a psychological problem with it: I do not understand what it does internally and that makes me somewhat reluctant to use it.

I Why does lack of understanding make me reluctant to use it?

Because hfslip sometimes does things I don't want it to do, like for example not installing the joystick pictures in the DX package. And it probably does other such things.

II How come I don't understand what it does?

Well, there is plenty of documentation on how to use hfslip, but seemingly none on how the wonderful thing of truly slipstreaming the hotfixes, IE, etc. is acomplished. When I first tried to investigate that matter, I thought "Hey, it's open source, simple thing to see what it does", took a look at the source code ... and fell of my seat: 3000 lines of code! My limited knowledge of batch programming and INF files combined with the size of the program led to immediate despair on my side.

It took some time until I regained enough courage to look at the code again. I had been thinking "Darn, it must be possible to figure out what that script does" and after hours and hours of studying and googling I came to the conclusion that I could at least manage to understand what the code does (i.e., improve my knowledge on MS-DOS batch programming). I also managed to learn a little more on INF files, though I still don't understand everything you need to understand in order to manipulate windows setup.

With some beefed up understanding of batch programming and INF files, I then also understood on a very high level what a few parts of the script do.

Ex.: What I know the IE6 slipstreamer does:

1. Extract CABS
   
2. Create some INFs
   
3. Create and fill HFOE.txt
   
4. Delete a file that's in the Rollup
   
5. Delete some files
   
6. Move all cats to svcpack
   
7. Move what's left to i386e
   
8. Presumably, the IEAK stuff isn't for me
   

So far, so good/bad: Having a glimpse of what's done there still leaves a lot of questions unanswered.

Ex.: Some questions that come to my mind but are left unanswered:

• What's wrong with the cabs in the IE package that are ignored by hfslip?
  
• What are the INFs created in step 2 and HFOE.txt good for?
  
• Why is the file that's part of the rollup deleted? Because it's in a more recent version there? Is that probably the reason that one must include the rollup if IE6 is slipstreamed?
  
• What's wrong with the deleted files?
  
• What's that i386e folder?
  
• ...
  

Similar questions also arise about many other issues.

Ex.: Why is it that a CD containing a hfslipped DX9 "cannot be installed from DOS", as can be read on FDV's page? Does that statement even hold when using recent releases of hfslip? I couldn't find a note on that issue on hfslip.org. And please note that there is a difference between saying "DOS installs are not supported." and "DOS installs won't work."

I've come to the conclusion that even if a totally understand what the code does (not that I'm close to that, though), I will never be able to answer such questions (dealing mainly with why something is done) by myself. Or to put it in an other way: I've given up on trying to gain a sufficient understanding of the script.

III What do I want then?

Well, I guess, what I'm asking for is documentation on the source code level of the program. Please note that I'm definately not demanding anything, I just say it would be nice to have some more detailed information available.

I'm aware that providing in-depth documentation is a huge amount of work, but maybe it is worth the effort, since that would probably not only satisfy my need for understanding but even enable me (and other people?) to contribute to the project. Also, good and detailed documentation would surely benefit the hfslip project in the long term. As of now, I wonder how many people are out there who completely understand the script ...

The following is only relevant if somebody can be found who has the time and knowledge to improve documentation.

-------------------------------------------------------------------------------------------------------------------------------------

I've also thought about ways to do that, in case someone wants to do it. I see the following possibilities:

• Splitting the script into different versions for 2k, XP, and 2003. That would probably reduce the number of lines of code.
  Disadvantages: The users must be smart enough to download the correct version for their OS and there would probably be some (a lot?) of duplicated code.
  
• Splitting the script into independent(!) subscripts responsible for slipstreaming the hotfixes, IE, DX, WMP, etc.
  Disadvantage: Duplicated code.
  
• Making older (smaller) versions of hfslip available for download.
  Disadvantage: It would be vital to explain in detail what is supported by the respective version and what not. Ex.: Is the bug in halmacpi.dll resolved or not?
  
• Thoroughly comment the script. Ex.: HFOE.txt is used for ... The registry entries added by this INF file are needed for/do the following ...
  Disadvantage: A lot of work.
  
• Creating a guide "How to slipstream hotfixes and other updates into a Windows source", just concentrating on how to replace the original files in i386 with the ones from the hotfixes, DX etc. and leaving out hfslip alltogether.
  Disadvantage: Probably also a lot of work?
  

---------------------------------------------------------------

Finally, it may be a good idea to tell you that I am only interested in using hfslip on W2kSP4. I've sworn myself to never use XP or Vista and I'm not willing to pay 500+ bucks for a copy of 2003.

So, having written down my thoughts, I surely feel a lot better now. Thanks for reading and no offense to anyone.

Cheers and good night to the people in Europe

Edited November 29, 2006 by berntie

[Super-Magician]

Hello, berntie! Thanks for your well thought-out post!! Unfortunately, I have not had time to read your entire message before writing this reply, and I apologize. I do however want to make a few notes.

Regarding your suggestions, all of this has in some way been directly or indirectly discussed in the past. At one point a few months ago, the authors (and a few particular users) decided that the script was becoming too much bloatware. After a relatively long discussion, they cut down on certain sections of the script.

Although I understand that having different versions for each OS or component to slipstream would in same cases be good (and I obviously appreciate your suggestions!), I feel that the cons outweigh the benefits. For one thing, like you said, large parts of these subscripts would be duplicated. In addition, it would be very hard to maintain. We made such a suggestion to Tomcat76, but it seems obvious that it would be a lot of work for him (if not already ).

I will probably have more thoughts after I read more of your post, but for now, toodles!

[fdv]

I'll offer my thoughts on the difficulty presented here. I don't want the following to be read as being offensive or dictatorial. I say all of this while being friendly, in the spirit of the forum.

It's inevitable that you'll be asked why you want to keep things like the pictures of joysticks.

To be clear, HFSLIP is geared toward replacing old files with new ones. Application add-ons are a happy side effect of what HFSLIP can do. I pick this statement out because it brings up a larger point: you're hesitant to use it based on a lack of understanding. That lack of understanding is a concern because HFSLIP "sometimes does things I don't want it to do." Do you trust the devs to make good decisions? I don't want to be insulting, but there has been no rush to document code because the devs listened to forum input on all sorts of issues.

I would agree that understanding the code and the INF manipulation is a steep learning curve, but there are those here who do understand by reading source. Source commenting is something the principal players have been too busy to do, and HFSLIP has changed radically in a short time. That would take a lot of effort to keep up with. Tom certainly doesn't have the time, and Tomcat's quite busy.

Back in the day, when HFSLIP was getting started, it made leaps and bounds as folks experimented with pushing boundaries and adding new features (before this forum existed). As an example, driver CAB merging was something that no one person knew completely... me, Oleg and Tommy studied system files, played with experimental builds that Tom issued hourly, and after about 6 or 7 tries we'd nailed the feature over a weekend. On Friday, we knew it was a feature that would be great to add, like nLite had. By Monday we knew how it worked so well we could probably explain it in our sleep. These days, we'd have to go back to our notes

You're doing what needs to be done in order to learn. I mean, it's what we did. Examine source material on the Internet and do lots of digging. Even now, I know more about INF files and Tommy knows more about CMD files and if I want to examine a feature in HFSLIP I have to ask Tommy or TC. There is a lot to be said for asking specific questions. But you're trying to "learn it all," it seems, and I'm going to tell you what you know but maybe hate to hear: it's really tough and we were at the same point you are but it takes time. Ultimately there is more value in you proceeding as you are than in the devs adding comments to code (other than a section header here or there). I admit, commented source might get everyone here more knowledgeable a lot quicker, but no one really has the time, and this brings up a second problem: code won't give you all of your answers (read on below about this point).

The specific questions you have (for example) about IE slipstreaming seem to be more related to why rather than how. A key concept here is that MS is about bloat. They make very sloppy code, and in order to implement something, they just throw more files into the mix. TommyP's original goal was to slipstream IE6, and that's about it. He spent a LOT of time writing code when he decided to go beyond that and slipstream hotfixes too. In any case, though I can't speak for him, I think TommyP would simply say, "trust me." Integrating IE 6 simply replaces IE 5 and to put it bluntly, there is a whole lot of detail that most people just don't need to know or bother with. Maybe it's nice to know that web channels worked differently in IE 5 than in 6 but it's one example of component-specific knowledge that probably isn't very useful. And a lot of components are like that.

Another example: when the Post SP4 Rollup came out, we noticed the HAL ACPI problem. There was also an ATAPI file problem. MS reissued the hotfix fixing the ATAPI issue but they neglected the ACPI problem. The HFSLIP devs know this, but even with examining the source, you probably wouldn't realize what's going on there unless you know the backstory. This is what I mean when I say that there are things that are just trivia or background -- it's component-specific knowledge that probably isn't very useful.

Yet another example: DOS-based installs. A DOS install uses DOSNET.INF. CD-based installs don't. In order to support DOS installs, HFSLIP would have to make edits to DOSNET. Since there was zero demand for DOS installs and TommyP had no interest in the feature, it wasn't implemented. Supporting DOS installs would require a lot of testing and a whole new layer of complexity. Code won't tell you this but the HFSLIP "senior citizens" like me know the backstory. Comments in code would have limited usefulness since if DOS installs were one day added as a feature, HFSLIP wouldn't just have a section that says "parse and edit DOSNET here." Large parts would have to be analyzed and rewritten a bit in several places. HFSLIP has a lot of elegant and very clever tricks it uses, and adding DOS support would again be work few are willing to implement, preferring to devote time to other features.

HFOE is Outlook stuff. The best way to learn about Outlook is to expand and examine MSOE50.IN_. It's just copying files and adding a lot of entries to the registry. Oh, and a LOT of files and registry entries for stationery. Which I believe most users would not find very useful... (Gee, thanks Microsoft, for providing nice graphics that make my e-mails bigger and come through as attachments to non-Outlook users.)

Different versions: wouldn't reduce code all that much.

Older versions: I'm sure Tommy would oblige.

Slipstreaming without HFSLIP: extract a hotfix with the /X switch. Take the new files and drag them into i386, replacing the old ones. The challenge there would be any new files not already in i386, and any new registry entries in the hotfix.

Anyway, I hope this is a helpful start on some of these questions and issues. Apologies in advance if you knew much of this stuff.

[berntie]

It's inevitable that you'll be asked why you want to keep things like the pictures of joysticks.

To be clear, HFSLIP is geared toward replacing old files with new ones. Application add-ons are a happy side effect of what HFSLIP can do. I pick this statement out because it brings up a larger point: you're hesitant to use it based on a lack of understanding. That lack of understanding is a concern because HFSLIP "sometimes does things I don't want it to do." Do you trust the devs to make good decisions? I don't want to be insulting, but there has been no rush to document code because the devs listened to forum input on all sorts of issues.

Well, I'm certainly not mistrusting anybody (like thinking "Hey, they're probably messing up my installation"). It is just that some things are a matter of taste. Maybe the joystick pic thingy is a bad example, but having these pics certainly won't harm system security and I am willing to sacrifice a few MB of hard disk space for the sake of completeness.

I would agree that understanding the code and the INF manipulation is a steep learning curve, but there are those here who do understand by reading source. Source commenting is something the principal players have been too busy to do, and HFSLIP has changed radically in a short time. That would take a lot of effort to keep up with. Tom certainly doesn't have the time, and Tomcat's quite busy.

I'm aware of the fact that it would require a lot of effort.

Code won't tell you this but the HFSLIP "senior citizens" like me know the backstory.

That is exactly the point. A lot of valuable knowledge is locked up in your heads. What I was ultimately asking for, was that knowledge being written down. If your top priority is adding features to hfslip instead of doing that, that's fine. I would prefer it the other way round, but I'm not in the position to tell anybody what to do

These days, we'd have to go back to our notes

I think any notes you have on how to do something could be helpful for other people to learn. Why not make them available for download?

I admit, commented source might get everyone here more knowledgeable a lot quicker, but no one really has the time

Nobody has the time? That's what I expected .

Slipstreaming without HFSLIP: extract a hotfix with the /X switch. Take the new files and drag them into i386, replacing the old ones. The challenge there would be any new files not already in i386, and any new registry entries in the hotfix.

Obviously, the challenging part is, well, just challenging. There, a guide would be quite helpful.

[tommyp]

If it matters, 98% of HFSLIP is making sure setup files are made and altered correctly, 2% is to be sure that the latest binaries are packed in the sourcess folder. If you want to know the nitty gritty, you probably want to read this:

http://support.microsoft.com/kb/828930/

[Tomcat76]

DOS-based and network-based installs should be supported for a while, now...

[Kiki Burgh]

quite a post ... the "big boys" have their say about this ... just trying to insert some comic relief ...

indeed comments would be really helpful though but seriously, HFSLIP makes it a bliss to slipstream &/or integrate all those updates & hotfixes ... what made me fall in love with HFSLIP (apart from very supportive folks of course ) is doing away with the long wait for completion after manually applying those regular updates/hotfixes while waiting for MS to release SP or at best having my OS patched already while guessing if MS would do otherwise ... reading through the script truly entails patience & extra knowhow to at least have a bare understanding of its execution ... this could be a great source too: http://www.microsoft.com/technet/prodtechn...t/winupdte.mspx ... thanks!

Edited November 29, 2006 by Kiki Burgh

[whitehorses]

Hey FDV!

I am amazed as always from your style and the knowledge behind it. It's a pitty that - as you said - this knowledge is "not very usful". I mean it's amazing for someone to study something such deeply just for hobby, without getting paid for it, but in the end it turns out that every one thinks you are crazy to invest that much in it. I had this feeling, when my father told me this summer to stop "tweaking". What a shame...

Last year I was digging deeply inside too, rewritten the fdv fileset, trying to make it to my taste, but in the end I had a screwed up install I even tried to make an alternative fileset which works with nLite. Wasn't succesfull though.

Someone with half this knowledge in the linux camp is called a guru... that's a shame too.

Edited November 30, 2006 by whitehorses