Refreshing data on the disk

[jaclaz]

Maybe there is some misunderstanding going on, I only have seen and read (via google translate) the single German article you posted a link to:

https://www.computerwoche.de/a/der-langsame-tod-von-festplatten-und-ssds,3549906

And it was written to inform users *like you*, not *like me*, as you can read German and I can't (I actually can but only a little bit - ein bisschen ).

BTW that article first says:

Quote

Festplatten haben das Potenzial, mit intakten Daten jahrzehntelang zu funktionieren, selbst wenn sie über längere Zeiträume nicht genutzt werden.

and later states:

Quote

Bei HDDs genügt es, sie einmal pro Jahr oder alle zwei Jahre hochzufahren, um zu verhindern, dass sich die mechanischen Teile der Festplatte festsetzen. Sie sollten die Daten aber auch "auffrischen", indem Sie sie neu kopieren oder dazu ein Drittanbieter-Programm wie Disk Fresh verwenden.

powering up the hard disk once a year or every two years is recommended to (supposedly) prevent stiction (which may only happen on some given types of motors/bearings), the refreshing is recommended..

If you want to follow the recommendations (opinions) in that article and use Diskfresh, then you should use it 3-4 times a year, as recommended by the manufacturer of the tool.

Surely the answer to the ultimate question about Physics and Magnetism (and Life, the Universe, and Everything) is 42.

jaclaz

 

[AstroSkipper]

29 minutes ago, jaclaz said:

Maybe there is some misunderstanding going on, I only have seen and read (via google translate) the single German article you posted a link to:

@UCyborg posted further sources in his initial post. And the language in which articles were written doesn't matter nowadays. We all can use Google Translator or much better DeepL And, you will find more scientific or technical information by using Google & Co, of course.

29 minutes ago, jaclaz said:

Surely the answer to the ultimate question about Physics and Magnetism (and Life, the Universe, and Everything) is 42.

Very funny!  But to be honest, I actually prefer rather scientific answers.  And I say that as an outspoken film lover. 

AstroSkipper 

Edited April 18, 2023 by AstroSkipper

[jaclaz]

Yep, I would also like to have scientific answers (possibly backed up by real  world data), with all due respect to the respective Authors, a thread on Vogons or a page on io.bikegremlin.com are only (baseless) opinions.

The whole point revolves around the lack of proper data about real world "bit rot" on hard disks.

Many people believe that the built in ECC in modern hard disks is enough to prevent them and that the reasons why you lose your data are largely not related to bit rot (i.e. the hard disk will fail for a number of other reasons long before magnetic bit rot comes into play).

Some other people believe in the magic of data refreshing (since we have nothing to use for comparison, we have no way to know if it is effective, how much it is effective, for all we know it could even be worse than doing nothing[1]) .

Both kind of people should actually have redundant backups for their important data, as it is the only surely working mitigation policy.

jaclaz

 

[1] only as a purely fictional example, you take a hard disk that has been kept unpowered for several months (if you follow the advice of one of the two software makers in the world of a program for this) or years (if you follow the advice of some random internet user publishing a blog or technical journalist writing an article on an online computer magazine) and you start reading and writing data to it for several hours continuously, would this stress accelerate its failure (for other reasons[2])?

[2] the "other reasons" are relevant as if any of these "other reasons" happen, you have lost your data anyway, and it makes no real difference, once you have lost your data, whether they were lost due to bit rot/lack of refreshes or for any of the "other reasons".

Edited April 18, 2023 by jaclaz

[AstroSkipper]

1 hour ago, jaclaz said:

Yep, I would also like to have scientific answers

Good to hear!

1 hour ago, jaclaz said:

The whole point revolves around the lack of proper data about real world "bit rot" on hard disks.

1 hour ago, jaclaz said:

Some other people believe in the magic of data refreshing (since we have nothing to use for comparison, we have no way to know if it is effective, how much it is effective, for all we know it could even be worse than doing nothing[1]) .

But we have the theoretical knowledge to understand the instability of magnetic data. In such a case, one can certainly do without such data, even if it would be very welcome. And all this is not a question of belief. It is rather a question of reason.  BTW, no one has claimed that data refreshing is a panacea for preserving magnetic information on an HDD. But it is one reasonable strategy, and that is what this thread is about. 


AstroSkipper

Edited April 18, 2023 by AstroSkipper

[AstroSkipper]

All that said does not mean I refresh my HDDs on a regular base. Sorry, I'm afraid I am too lazy to do that with all my HDDs.  But from a physical or technical point of view, refreshing of disks makes perfect sense and can make its contribution as a data preservation strategy. Additionally, checking an HDD in form of data refreshing triggers the controller of such a disk to renew its S.M.A.R.T. values and to decide for example whether pending sectors have to be rated as defective or not. I already did that successfully by using Spinrite in the past.  After performing such checks or refreshes, there is a positive side-effect. You will definitely know much more about the health of your disks and the condition of the data on it, including up-to-date S.M.A.R.T. values. 

AstroSkipper 

[jaclaz]

A strategy based on a theory without any real world, practical, reliable data supporting it is not "reasonable".

Again, magnetic bit rot exists (in theory), but it may also exist (in practice) so that hard disks have built-in ECC and many modern hard disks have other additional self-check routines, of which we know nothing or next to nothing and that may vary greatly with the several different technologies used in different generations of hard disks and different makes/models of them.

In theory, some of this bit rot may (somehow) escape the ECC and/or the other provisions.

In practice, there is no real world data supporting (or denying) this.

Refreshing the data on disk would logically prevent this if done "often enough", but noone knows how often is "often enough", if performed not often enough it serves no purpose whatsoever (as the data is already lost).

And we don't know if this (rewriting a same value over itself) actually makes the sector "weaker"[1] or affects this (or that) other aspects of the hard disk functionalities.

About S.M.A.R.T., we actually have real world data supporting the finding that it is mostly meaningless as a predictor of hard disk failures:

For all we know you don't need to refresh data to have the disk do whatever it does (that we don't know), powering it on periodically[2] and leaving it idle for some time [3] will make it do whatever it is supposed to do (that we don't know) and this might (or it might not, we don't know) update some relevant S.M.A.R.T. attributes that are anyway largely meaningless.

Now the real questions are:

Should non powered hard disks be stored horizontally or vertically?[4]

Should they be protected from cosmic rays? [5]

jaclaz

 

 

[1] there are also theories about "weak" sectors that can be "revived" by writing to them different patterns

[2] but we don't know how often

[3] "time enough", that we don't know how much it is

[4] which is loosely linked to the question whether they should be mounted horizontally or vertically in our PC's

[5] Heck, if we have muon tomography and it works, what will muons do to our bits (and does parallel recording affect the way we should store our disks)? https://en.wikipedia.org/wiki/Muon_tomography

[AstroSkipper]

23 hours ago, jaclaz said:

A strategy based on a theory without any real world, practical, reliable data supporting it is not "reasonable"

It almost seems like you're struggling a bit with the real world and its data.   This term appears in many of your sentences. Just a side note! Physics describes the real world perfectly, and I would not close my mind to it.  And at the risk of repeating myself here, the strategy of data refreshing is reasonable in any case. Simply using the reason of a healthy, scientifically thinking mind! 
Anyway! I have explained this clearly enough in the previous posts, so no need here to do that again.

23 hours ago, jaclaz said:

About S.M.A.R.T., we actually have real world data supporting the finding that it is mostly meaningless as a predictor of hard disk failures ...
... and this might (or it might not, we don't know) update some relevant S.M.A.R.T. attributes that are anyway largely meaningless.

S.M.A.R.T. values of HDDs are very interesting, helpful and definitely not meaningless. Of course, nothing can be really perfect, but I observe these values continously on my Windows XP computer using Hard Disk Sentinel Professional which is always running in the background.  Reporting or rather alerting important changes of the S.M.A.R.T. values by Hard Disk Sentinel has saved me from data loss several times. And happened in the real world.  
Observing and evaluating these values led to technical improvements on my computer, which I made myself years ago. 

23 hours ago, jaclaz said:

Should non powered hard disks be stored horizontally or vertically?

And the winner is: the horizontal storage.  
HDDs should be stored and used horizontally. This also has something to do with physics.  In any case, that is what I always did and will do.  And I have some real world data confirming this. My HDDs are very old, most of them more than 15 years. And they are working fine and are in a good state of health. 

23 hours ago, jaclaz said:

Should they be protected from cosmic rays?

On planet earth, no need to protect them from cosmic rays at the moment. But maybe, one day!  Whether we (humanity) will still be alive then, or this kind of technology? Who knows!   

AstroSkipper 

PS: By the way, I do not believe anything. I am an atheist. 

Edited April 19, 2023 by AstroSkipper

[jaclaz]

It is not like you have thousands of disks stored horizontally and thousands of disks stored vertically and you found a substantial difference in their lifetime that can be connected to their storage spacial orientation (i.e. independent from the various drives make, model and year of manufacture), for all we know (say) Seagate's from 2007 are better stored vertical whilst WD's from 2006 are better stored horizontally.

What you offer (like any of us can only do) is some anecdotal, very limited, data, I am very happy that you have very old hard disks still working (horizontally) and that you keep an eye on their S.M.A.R.T. values, but for all we know this may have nothing connected to the disks longevity, it could well be that they are properly cooled, that you have good air where they are kept/used, that your PSU's are very good, we cannot know for sure.

jaclaz

 

[AstroSkipper]

1 hour ago, jaclaz said:

What you offer (like any of us can only do) is some anecdotal, very limited, data

For me, very important data, though. I have been dealing with hard drives since the late 1980s. Therefore, I have a lot of experience.

1 hour ago, jaclaz said:

I am very happy that you have very old hard disks still working (horizontally)

Thank you! I am happy, too. 

1 hour ago, jaclaz said:

you keep an eye on their S.M.A.R.T. values,

Yep! And I will do so as long as my HDDs exist. It helped me a lot in the past as I already mentioned. 

 

[jaclaz]

OT (but not much) I sleep on the left side of the bed and always put down the left feet first when I get up.

I started this in the early 1960's and so far has proved very effective.

For a small period of time in the late 1980's I had to sleep on the right side (and thus had to put down my right foot first) and my laptop hard disk suddenly started developing bad sectors.

I am not saying the two are connected, still as soon as I was able to re-arrange the bedroom and got back to good ol' left foot first those disk errors stopped.

To be fair, maybe it was due to the new laptop I bought at the time, though since then I had many laptops and their disks never started developing bad sectors again[1] .

jaclaz

[1] there was an exception in late 1993 (or maybe it was early 1994?) when I found 3 (three) bad sectors on a Compaq laptop hard disk (if I recall correctly it was a Seagate 120 Mb in size)

[AstroSkipper]

6 hours ago, jaclaz said:

OT (but not much) I sleep on the left side of the bed and always put down the left feet first when I get up.

I started this in the early 1960's and so far has proved very effective.

For a small period of time in the late 1980's I had to sleep on the right side (and thus had to put down my right foot first) and my laptop hard disk suddenly started developing bad sectors.

I am not saying the two are connected, still as soon as I was able to re-arrange the bedroom and got back to good ol' left foot first those disk errors stopped.

To be fair, maybe it was due to the new laptop I bought at the time, though since then I had many laptops and their disks never started developing bad sectors again[1] .

jaclaz

[1] there was an exception in late 1993 (or maybe it was early 1994?) when I found 3 (three) bad sectors on a Compaq laptop hard disk (if I recall correctly it was a Seagate 120 Mb in size)

OT: Maybe, you perform a statistical investigation to get data in terms of correlation.  That would be very interesting for me as a mathematician. 

AstroSkipper 

Edited April 19, 2023 by AstroSkipper
Update of content

[UCyborg]

I ran DiskFresh on the disk with oldest stuff on it, it's a WD WD5000AADS, which was bought in July 2010. The program is simple and should run on every Windows from XP to 10 at least. It also reports bad sectors, have two of those, which I knew about for quite some time now (just that they exist, I haven't found the time to investigate when I saw for the first time that they exist in the S.M.A.R.T. data, which must have been months ago), but only encountered one file several weeks back that could not be read. I've got the file back from backup copy and ran 7-Zip's CRC32 check through remaining files just to see if they're readable. They are, so I guess should run some other program that would write zeroes to unallocated clusters to get the drive to remap bad sectors.

Whatever processes might be occurring, they should be very slow, it doesn't seem like leaving some files on the drive for 10 years will surely change them. So one might just wear out the drive faster by refreshing it regularly.

Spinrite...beware of that program, especially if the disk is going wonky.

[AstroSkipper]

2 hours ago, UCyborg said:

Spinrite...beware of that program, especially if the disk is going wonky.

As I already wrote in previous posts, I used Spinrite 6.0 in the past, and it helped me to refresh old HDDs and to check a few pending sectors on them which were then remapped as good.  This program has served me well. And I wouldn't hesitate to use it again if there were problems on my old disks. HDDs that don't survive this tool wouldn't last much longer anyway. A backup of their data is mandatory, only if possible, of course. Just to be on the sunny side of the street! 

Edited April 19, 2023 by AstroSkipper

[jaclaz]

Spinrite may (or may not) be the best program around, but in any cases it has now become very, very outdated.

The current version, 6.0 is now around 18 years old, and there are many reports (even on its Wikipedia page):

https://en.wikipedia.org/wiki/SpinRite

about issues with some newer BIOS/Firmware and "large" disks (which may even mean larger than 128 GB ones in some cases).

In any case, whatever "proprietary methods" it uses (maybe) it is improbable that those methods are fully compatible/suitable to the new hard disk technologies that in the meantime have been used in the manufacturing of hard disks, so it either reverts to using "normal methods" when it finds something new or it risks to do more damage than good.

All in all, using it (today) on a disk manufactured in the last 15-18 years, even if one manages to access large disks with it, is risky, while its use could still be attempted as a last resort (say in attempting to recover data that any more recent tool cannot) it isn't (IMHO) advisable to use it for "maintenance" tasks (such as this disk refreshing)

Diskfresh (as well not particularly current, last version is seemingly 2013) on the other hand seems like a "normal" program (i.e. not using any particular proprietary method) so it should work just fine on more modern hard disks as it likely only uses conventional read/write procedures.

It remains a mistery (to me) why, if this disk refreshing is so useful/advised by "experts", only two dedicated programs (for Dos/Windows) exist that perform this disk refreshing.

The badblocks program (Linux) in the read-write test mode makes *something more* (from what I can understand):

1) it reads the contents of a given amount of blocks from hard disks and caches them
2) it writes to the blocks a "random" pattern
3) it verifies that the pattern has been written correctly
4) it writes back the cached contents
5) it verifies that the cached contents have been written correctly

so it seems like it will be much slower (three reads/two writes), though, if we follow the line of reasoning about the need of bytes on hard disk needing to be "exercised" (which personally I consider being a form of voodoo) it would actually make sense to test the random pattern.

Besides, like many (Linux but not only) command line tools there is the risk to issue by mistake a destructive command.

Once re-said that (in my opinion) it makes no sense to "refresh" data on disk, a hypothetical suitable procedure is more like:
1) procure a (new) disk bigger than the original
2) make a RAW (dd-like) copy of the original on the larger disk
3) calculate the checksums of both the original and of the dd-copy and verify that the checksums are the same
4) dd back the image on the original
5) re-calculate the checksum of newly written  "original" and verify it is still the same
6) keep the RAW image on the new disk as a further backup

This will take LOTS of time on today's large hard disks and put a lot of stress on the two hard disks involved, so you'd better have a setup where the hard disks are properly cooled (fan pointing to them directly).

jaclaz

 

[AstroSkipper]

7 hours ago, jaclaz said:

Spinrite may (or may not) be the best program around, but in any cases it has now become very, very outdated.

The current version, 6.0 is now around 18 years old, and there are many reports (even on its Wikipedia page):

https://en.wikipedia.org/wiki/SpinRite

about issues with some newer BIOS/Firmware and "large" disks (which may even mean larger than 128 GB ones in some cases).

In any case, whatever "proprietary methods" it uses (maybe) it is improbable that those methods are fully compatible/suitable to the new hard disk technologies that in the meantime have been used in the manufacturing of hard disks, so it either reverts to using "normal methods" when it finds something new or it risks to do more damage than good.

All in all, using it (today) on a disk manufactured in the last 15-18 years, even if one manages to access large disks with it, is risky, while its use could still be attempted as a last resort (say in attempting to recover data that any more recent tool cannot) it isn't (IMHO) advisable to use it for "maintenance" tasks (such as this disk refreshing)

Just for clarification! I use Spinrite 6.0 only on old PATA HDDs. I would never think of using such old hardware-related software on current disks.  Such considerations would be simply absurd.  All has to be sensible or in some cases even time-period correct and has to match the necessary system requirements. However, the HDDs in my anchient Windows XP computer (year of manufacture 2000 and BIOS from 2002) are all very old and match the system requirements for using Spinrite.  Newer hardware means logically more recent tools for such measures. And I already stated:

On 4/18/2023 at 6:24 PM, AstroSkipper said:

All that said does not mean I refresh my HDDs on a regular base. Sorry, I'm afraid I am too lazy to do that with all my HDDs.  But from a physical or technical point of view, refreshing of disks makes perfect sense and can make its contribution as a data preservation strategy. Additionally, checking an HDD in form of data refreshing triggers the controller of such a disk to renew its S.M.A.R.T. values and to decide for example whether pending sectors have to be rated as defective or not. I already did that successfully by using Spinrite in the past.  After performing such checks or refreshes, there is a positive side-effect. You will definitely know much more about the health of your disks and the condition of the data on it, including up-to-date S.M.A.R.T. values. 

AstroSkipper 

AstroSkipper 

Edited April 20, 2023 by AstroSkipper
Update of content