circumflatulator

Just a quick update: Everything still working fine. Enjoying the media server immensely. Has worked perfectly without a single glitch since last year when this thread was current. All 2 TB filled and it is a very rare day indeed if at least 2 hours of video is not streamed off those disks. Most days are much more. That probably indicates the RAID hack for XP is very reliable and capable of flawless operation over a wide range of hard drive and disk controller configurations, as in my setup. I still don’t know what will happen when the first hardware failure occurs. I try not to think about that. Will post here when it does happen.

I'm no expert, but it sounds like you have it set up right. On my home network, unchecking the box "Allow network users to change my files" does just what you want it to. Maybe try checking the box, then hit "Apply", then "Ok", then going back and unchecking it (hitting Apply and OK again) and seeing if XP shows the message "Changing folder permissions". I know it works for me if I do it for the entire disk drive. I have not tried it on individual folders, though. Maybe one of the gurus here will give you some pointers.

jaclaz, Don’t feel bad about pointing me to those web sites. I owe a great debt of gratitude to you and others on this forum who pointed me to the truth! The whole truth may not be in any single site that questions the validity of trusting RAID or other redundant disk setup to store precious data. In fact, for me, a very important thing I learned in this whole endeavor is of a more philosophical versus a technical nature. Computers are a great tool for many things. We play games on them, surf the web, watch videos, etc. etc. etc. A few of us may even write a program or two! For those tasks, a few tens to hundreds of GB is enough disk space and when a drive crashes, sure it’s a pain in the a$$ but not really a catastrophe. A new drive and a couple hours of reloading the data back and away you go, the memory of the crash quickly fading. But as disk space gets cheaper, we add more and more. Then we start finding lots of cool things to do with that new-found storage space. For me, it was music, pictures, and movies/tv shows. Before long, you got a ton of stuff that, while most of it might be “technically replaceable”, the effort to reload/rebuild it when a drive fails becomes a massive endeavor. And if you start putting thing on those disks that may not be easily reloaded (like all those scanned family pictures, where the originals were borrowed, scanned, and then returned to various relatives all over the USA), the surety of a disk failure sometime in the future starts to scare the crap out of ya. That’s where I was when I started this RAID-5 effort. On the surface, it sounded so perfect. Redundancy and backup with very little extra money and no time-consuming weekly removable-media backup sessions. But that’s an illusion. RAID-5 arrays can fail too, with a total loss of data. So my advice to all (and virtually every one of you will eventually find yourselves in my predicament, say when disk storage gets down to <10 cents/GB) is to plan an affordable, workable and secure redundancy/backup configuration BEFORE you “build a monster” like I did. RAID-5 isn’t bad. It’s better than no redundant drives/backup at all. But it’s not enough on its own when the quantity and/or value of your data gets beyond a certain point. And I’m going fishing tomorrow morning with a friend. A few beers and one of them “sneaky swimming bastards” on my line should make everything all-right again!

Oh, Crap. After reading more info on RAID-5, I’m not so sure I have the reliability/backup issue covered as well as I originally thought. I’m even beginning to wonder if I’ve created a monster. My only purpose in creating this RAID array was to achieve a “cheap automatically-backed-up data storage and serving” device that is very VERY unlikely to suffer any lost data because of a disk drive failure. That was it. Period. The data stored on that computer is not technically irreplaceable. But for all intents and purposes, it is **** close. The scanned family photographs (going back to the late 1800s and consisting of over 4,000 flat-photograph scans and over 6,000 35mm hi-res color slide scans) could technically be re-scanned. It only took 4 years to gather/borrow/scan/return those pictures/slides, but it is theoretically possible to re-do it all if the array were to crash. And I still have all the original music CDs and movie DVDs, so I could re-rip (and re-encode the videos) them all - maybe 300 hours or so should do it. And all the episodes of Leave It To Beaver and The Andy Griffith Show (now you know my age) and other such stuff from my childhood, that I have digitally recorded off the satellite, could be re-recorded - over time. MUCH much time… So maybe I should classify my data as “virtually irreplaceable”. It would really be a personal tragedy to lose it. When it was just stored on individual drives, if one fried – only some of my data disappeared. If the RAID-5 array goes belly-up, it’s ALL gone. And there it all sits- existing in only ONE place - on a single 2TB RAID-5 array running on an un-impressive XP machine in my basement. Oh yeah, I’ve done a few things to help out my chances of keeping that data intact. I’m the only one who has write/modify privilege on the server. So other than my own stupidity or fat-finger mistakes there isn’t any way the users of that data (wife/kids/grandkids playing movies on the networked dedicated-player-computers around the house) can delete or modify it. The server is not connected to the internet, so viruses and hackers are unable to compromise the data. I absolutely refuse to load/install/run any 3-rd party software on that server for fear of getting it infected or having a crappy program wipe out a drive or two. Hell, I won’t even put M$ updates on it – it’s working fine as it is. The rack-mount case has redundant power supplies, oversize triple-redundant cooling fans with 4 temperature sensors and alarms out the wazoo. A true HEPA air filter on the front of the case. A dedicated 3KVA APC UPS with 3 hours of battery-backup run time. And if the power mains to my home go out for over 3 minutes, there is an outside 15KW propane generator that automatically starts and runs most everything in the house (that was already installed for other reasons, like the un-reliable power we have out here in the sticks where I live). Even those hard drives were installed with particular care to assure excellent air flow around all 6 sides of each one. And the ambient temperature in my basement (here in Utah, almost every home is built with an underground basement. Some folks finish theirs up for additional living space, but mine is only for the HVAC room, water heater, storage area, and shop area) never varies more than 5 degrees from 65F year round. Perfect environment for computer stuff. And the rack-mount case itself is installed in the top 6U position (in case of water leaks in basement) of a 6-foot tall, 19-inch industrial grade rack cabinet – which itself is bolted to the concrete floor in case of earthquake (we rarely have them out here but they say a big one is overdue). Jeez, you’d think I’d be happy with this setup and able to sleep at night knowing that my precious data is virtually indestructible. WRONG!!!! Starting by following the links jaclaz provided in his last 2 posts, I began finding disturbing information on the unreliability of RAID-5 arrays (the worst is: <http://www.baarf.com/>). Agreed, this information is somewhat dated since its relevance changes as the technology improves, and most of the folks who posted there are database users where the write performance of RAID-5 severely affects their machines. But I still came away with serious concern that this “monster” I’ve created is really not as safe as I had hoped. Everywhere I read things like “the only truly safe backup method for important data is to do magnetic tape or disk, or optical-media backups on a regular basis”. How in the hell am I supposed to “back up” 2TB of data that way? Figure the number of CDs or DVDs or tapes to do that! Plus the time commitment. Over time, just the blank media costs would go a long way to pay for a mirrored RAID array (which is supposed to be much safer than RAID-5). Sorry for my rantings. It’s just that I feel lost – unsure that after all this, my data is really safe enough. I know there is not a perfect answer to this problem. SysAdmins and others in the IT industry have known this for a long time. I guess if cost were not an issue, RAID-1 looks like the way to go. But I’ve shot my financial wad on the disks I already have virtually filled with data. Not much option for me but to hope this existing array holds out until either I win the Idaho Lottery or the price of disk storage drops a bunch. But then what? Would I build a 4TB RAID-1 array and then toss and turn all night worrying about that cheap disk power plug splitter I installed last month catching fire inside the server, or a mega-lightning-bolt hitting the power line that blows the whole thing to smitherines? I think I’m getting psychotic over this. Time to go fishing.

jaclaz, Real good info. I kind of understood the reasons for the write speed penalty, but that article explained it in detail. When I read the author’s reasons why he doesn’t like RAID-5, it really concerned me. But on further research I found a work-around for the worst of his issues: the no-checksum calculation for a RAID-5 read operation, which may cause bad data to be propagated throughout the array over time. What I found is that (at least for Western Digital drives) a lot has changed in the 2 years since Mr. Kagel wrote his article. Specifically, WD has implemented what they call “Data Lifeguard” functions into their drives which does keep track of (among many other things) failed read parity and how many spare sectors are available should used sectors go flakey. When that number drops below 50, the drive is supposed to alert the OS and display (I’m assuming in the Disk Management window on XP) an error message. However, WD also provides a downloadable application (also called Data Lifeguard) which you run manually and it shows you all kinds of diagnostic data, including the number of “Grown-Defects" since the disk was originally manufactured. Presumable, you should run that utility every week or so if you’re not sure your OS supports the error display mentioned above. The other issue that bothered me, having a second drive fail before you can replace the first failed drive, seems highly improbable in my situation, as all the drives in my array have been in use 24/7 for over a year. I’m assuming that this issue would only be a problem if someone built an array out of a bunch of disks purchased at the same time and from the same manufacturing lot. Manufacturing defects usually show up in the first few days/weeks of use, and are sometime called “infant-mortality” failures. Also, the MTBF (mean-time-before-failure) numbers for hard drives have gone up considerably in the last couple of years. Another way a user could get hit with a second drive failing before the first is replaced is if he procrastinated too long before replacing the drive. I have a spare 250GB drive that I am using for video-editing (meaning it only will have temporary files on it anyway) that can be used as a “hot spare” when I need to replace a failed drive from the array. I fully agree with Mr. Kagel’s logic: mirroring is the better way to have reliable failed-disk-data-recovery. But I would need to buy and install 7 more 250GB drives, and besides the cost, there just isn’t room in that rack-mount case (nor cooling or power supply capacity) to do that. I guess that is a tradeoff I'll have to make. But when all is said and done, the true test of how reliable this array actually is will only be know after I’ve had my first failure and have tried to recover from it. When that happens, I’ll post the results here.

I've had a few problems with the ATI software that came with my All In Wonder 9600 and HDTV Wonder boards. And I too had to remove everything and re-install to get it to work right. I finally learned to not install the Remote Wonder (RW) software as it seemed to cause most of the problems I was having (that means having to live without the cool remote control, but oh well...). Another thing I learned is that there is a program called DAO that is part of the ATI software, and it too needs to be removed to get the entire ATI software out of your machine. If that one is still showing up under 'Add/Remove Programs', remove it too. And when you get everything removed, you have to reboot the computer to get back to a pre-installed state. Hope that helps.

Ok, sorry for the long delay in posting my experience with this. But in case anyone else is interested in doing this, here is what happened: A friend of mine knows a fellow who works for M$ and he indicated XP is a 32-bit OS and as such can only keep track of 4,294,967,296 (FFFFFFFF base 16) sectors, which at 512 bytes per sector, equals 2,199,023,255,042 (2 TB) of storage space in any single logical drive. Thus the 2TB limit indicated in previous posts. Using 250GB drives, that means the largest RAID-5 array consists of 9 250GB drives (8X250GB for the data, and 1X250GB for the checksums). The checksum storage does not figure in the 32-bit limit calculations. These limitations do not apply with Server 2003 or other 64-bit operating systems, but that doesn’t apply to me as I want to stick with the XP system I have. Another thing learned from him: You cannot add drives to the RAID-5 array without deleting the volume, adding the additional drive(s), and then reformatting the volume. This means all the data previously on that array would be lost trying to add an additional drive. With the above in mind, I worked with a friend who needed to buy some more hard drive storage for his computer and he bought 9 new 250GB drives, which he kindly lent to me so I could temporarily copy all my existing data to them. Then I reformatted 9 of my existing 250GB drives into the new RAID-5 array (that took 5 hours) and then copied my data back from the borrowed drives. That whole process took a couple of days. Observations: The XP RAID-5 hack works great. It’s been up and in use for a month now and is perfectly stable. It does have a write-speed issue: Writing to the array is about 3 times slower than to a single drive. Read speed appears faster than from a single drive. This is fine for my use (file server for music/video/pictures etc) since I only write a file once and then read it many times when it’s played. I haven’t done the benchmark tests yet, but will post them here when I can get to it. The CPU overhead (doing the XOR checksum calculations during writing) is about 4%, nothing to worry about. During read operations, CPU is at less than 1%. Memory page utilization and disk que length is less that using a single drive. All good! The 9 drives that make up the 2TB array are all 250GB Western Digital, but some are IDE, and others are SATA. They are spread over 2 separate PCI controller boards and the built in controllers on the mobo. I was concerned about timing issues because of this setup, but it is working perfectly. Another nice benefit: I used to have my movies spread over numerous drives. Quite a hassle when you are trying to find a particular film but can’t remember which drive it is on. With one huge logical drive, cataloging all media in self-descriptive categories is a snap and makes searching very intuitive. I would also recommend that you try to build the largest RAID-5 array you can financially swing because the larger the array, the less “overhead” (seen in Disk Management) you have to tolerate for full redundant backup. With my 9 drives, 8 are holding actual data which shows as only 11% overhead in the Disk Management window. With only 3 drives, that figure would be 33% overhead (2 drives hold data, 1 for checksums). All in all, I am VERY happy with the performance of XP with this RAID hack, and would highly recommend it to anyone needing large disk storage for media (or other ‘static’ data) files who wants online fault-tolerance without the hassle or expense of tape/CD/DVD backup. Now when one of my disk drives fail (notice I said WHEN, not if) , I WON’T LOSE ANY DATA! You just replace the failed drive with a new one and XP “rebuilds” the data that used to be there. NOTE: If you have a drive fail, be careful you remove and replace ONLY the defective drive. If you accidentally replace the wrong one, when you boot up the system will detect 2 “empty” drives in the array and it can’t recover from that much “lost” data. You then will likely lose ALL your data! And again, thank you to everyone on this forum who helped me get the courage to try this!

djackson, Thanks for the info on your experience with the XP RAID-5 mod. Its comforting to know that you were able to transplant the intact array to a different machine. And like you, I am not after any of the read-speed benefits of RAID-5. I’m after data storage with the ability to recover from a hard drive failure without losing any data. And with HD space becoming so cheap (250GB=$90.00US) a large RAID-5 media file server made out of a standard PC is the budget way to go. No more having to store/index/catalog/find CDs or DVDs or load them into players. Just a couple of mouse clicks and the media is playing right on the big screen. A very convenient setup the wife, grandkids and I have been enjoying.

jcarle: EXCELLENT information. I believe you have answered the last remaining question I had: How big do I dare try to build any single LUN in the RAID array before running into a limitation = 2TB. Then I can just build another 2TB LUN, and so on. This is important information as once I get all my data into the RAID array, the original disks that held that data will have themselves been integrated into the array. Then if the RAID LUN were to crash when I tried to exceed the 2TB limit, ALL my data would be lost (Aggghhh!) That is my worst fear. And I agree that going the Server 2003 route would be a superior choice, but I cringe at the thought of wiping my system drive and installing a new OS (I know, an irrational fear – but that scares me for some reason. Maybe its because for the first time ever, I actually have come to trust a piece of software coming out of Micros***, as this XP SP-2 I’m running has never glitched, blue-screened, or given me the slightest hiccup. Maybe that’s because it has never been connected to the Internet and I have NO plans to implement any Micro$oft upgrades/updates in the future.) However if, for some insane reason, I decide to TRY and add more than 2TB to a single LUN, I’ll definitely post the results here and on shaggy’s site. Thanks, big-time.

shaggy8675: Your website is the only source that was found (BIG thanks to TheBlackMan) on the Internet where real-world people talk about their personal experiences using the XP RAID hack. Made me feel much better knowing others are successfully using it. And thank you for the additional information! If everything goes well, this coming weekend I’ll have it up and running. I will report my findings/experience to you and you can add that info to the RAID0 and RAID1 data you already have posted there. However I don’t yet know how to do the performance benchmark data gathering, but I’ll figure it out. What I am planning on doing is adding a new 4-port SATA controller board (PROMISE SATAII150 TX4) and 4 new 250GB SATA hard drives (WD 2500JD) to the computer and setting that up as a single 750GB RAID-5 volume. Then I will copy all the data from my 250GB music backup drive to the new RAID volume. Then I’ll reformat that music backup drive and try adding it to the RAID volume and make sure the music files are still there and usable. That way if there is a problem and I lose the data while adding a disk to the RAID volume, I’ll still have the mirror-copy of my music files on a separate drive – nothing will be lost. If that much works, then I’ll start copying all the files from my video disks, and then add those disks one by one to the RAID-5 volume. If I don’t run into any size or number of disks limitations, when all is done I should have a single 3TB RAID-5 volume! (Note: All the pre-existing drives are WD 2500XX drives, with the XX being either JD or PB or JB. All their specs are identical except some are IDE and the others are SATA. My understanding is that they can all be included in the same RAID-5 array using the XP hack.) The only way I can see that I would lose my data is if I go to add another drive to the RAID volume and I run into some unpublished limit (trying to exceed a maximum-size-of-volume or maximum-number-of-drives-per-volume/array) and have the whole RAID array go into neverland. I guess that is the risk of using this hack as I’m sure Micro$oft won’t publish any specifications on this hidden feature built into XP. On a side note, is there any reason to suspect that this hack is possible because XP uses core elements of Micro$oft Server (or visa-versa) and those unknown size limitations are the same as in SERVER? If that was the case, then some research on SERVER (2000 or 2003) should produce information on those unknown limits. AGAIN – THANK YOU TO EVERYONE WHO POSTED TO HELP ME OUT!!!! I’ll try to reciprocate, but my knowledge is quite limited compared to most here on this site.

Thank you for the ideas, jaclaz And don’t worry about hurting my feelings: I know enough to know I know very little about this stuff! (what a weird sentence) Checked out the reByte hardware – but it has two limitations my present setup doesn’t have: 1- Only 4 drives are supported. 2- Only IDE drives supported. Since I already have nine 250GB drives in my machine and need more space, the 4-drive limitation is a killer. Also, I learned early on that having a bunch of IDE drives in one box made cable layout a nightmare. So the last 4 hard drives are all SATA and they are a joy to work with. The reByte seems to be a good answer if it wasn’t for those limitations. The NASLite product is also limited to a maximum of 4 drives. And they also indicate the drives must be IDE. However this product seems to be targeted at using multiple computers, each with up to 4 IDE drives, all networked together to create a NAS server farm. I guess that is an option if my hopes of keeping all the drives inside one large fileserver turns out to be an unworkable idea. I realize that these limitations may be overcome in these companies’ next releases, but I am out of disk space now and need to do something soon. That’s why the XP hack looked so appealing to me. Quick and cheap. Just wish I could find out what its limitations are. Again, thanks for the links. All ideas and options welcome!

Thanks BlackMan. It took about 30 minutes to check out every part of that site but it was worth it. I think I'll have to email some of the folks in that sites guestbook who have tried the RAID-5 setup to get some real-world feedback, as the benchmarks and comments in the rest of the site only seemed to refer to RAID-0 and RAID-1. Thanks again. p.s. I would have sworn I googled every site on the internet with "XP RAID-5" in the searchbar. How this one escaped my search is a real mystery.

Hello. First "forum" post ever. Medium PC savvy. Retired hardware/software engineer. Built rack-mount P4-3GHz Intel-mobo machine with XP-Pro 1 year ago to hold my CD collection. Ripped over 300 music CDs in .wav format onto 1 250GB HD. Worried about drive crash, so added another 250GB and copied all files for manual backup. Worked great. Since then have got into video. Bought more drives and ripped some of my DVDs. Started recording off satellite too. That computer is now just a file server for the other computers that actually play the media around the house. Now over 2TB but no backup for video stuff if HD fails. Much google research made me decide to go RAID-5. Researched RAID controllers. My mobo only has PCI slots but all the large capacity (6 SATA ports and above) RAID boards require PCI-X. Ran into hack http://www.tomshardware.com/storage/20041119/ to activate RAID5 in XP. Sounds great. But I can't seem to find answers to the following nagging questions: 1. Is ANYONE actually using this hack? Does it work good? I don’t care too much about write speed, but needs to read at least as fast as the individual drives/volumes do now. Is it reliable? 2. How many 250GB drives is the max this hack will allow? 3. Does it have a GB or number-of-drives limit to a single large RAID-5 volume? 4. Are there any negative aspects to very large RAID5 volumes using this hack? 5. Since I’d like to keep adding drives to the RAID-5 volume(s) as my library grows, can that be done easily? Or does the array need to be created to its maximum size before copying any files to it? The answers to all these questions are available for the hardware RAID controllers, but can’t find diddlysquat about that XP hack. Since the risk of losing all my data is great if there were to be a problem adding drives later on, I’m really can’t move ahead until more is learned. Any feedback would be appreciated. Thank you!