Question test HDD files

[the54thvoid]

yes

1 year without use loses magnetization, mechanical and oxidation problems?

Do HDDs also use flash memory to retain firmware and BIOS? Is this a problem since flash memory is not recommended for archiving?

Is it necessary to test annually or every two months to check for any type of corruption?

Is a 2.5" HDD more likely to die when stored or in constant operation?

This post.

Then 10 replies follow.

Then you repost it again, Verbatim (minus the 'yes'.)

1 year without use loses magnetization, mechanical and oxidation problems?

Do HDDs also use flash memory to retain firmware and BIOS? Is this a problem since flash memory is not recommended for archiving?

Is it necessary to test annually or every two months to check for any type of corruption?

Is a 2.5" HDD more likely to die when stored or in constant operation?

You did this in a previous thread. And you keep reformulating your questions.

Stop thread spamming with the same posts.

 

[Vincero]

As far as I'm aware, RAR is the only format where you can specify a parity recovery amount so it can recover from XX bytes of data corruption instead of just detecting it.

It is also one of the only mainstream file archiving versions that also allows file versioning of archived files.

It's worth paying for if being used for it's extra features, otherwise any Zip/7zip tool will do basic CRC checks - 7zip GUI I think even gives you the option to generate SHA also.

 

[A Computer Guy]

As far as I'm aware, RAR is the only format where you can specify a parity recovery amount so it can recover from XX bytes of data corruption instead of just detecting it.
It is also one of the only mainstream file archiving versions that also allows file versioning of archived files.

I didn't know that about RAR. Pretty useful thanks!

 

[Vincero]

I didn't know that about RAR. Pretty useful thanks!

They don't call it that. But here it is in the docs

How Does the Recovery Record Feature Work?

WinRAR - the data compression, encryption and archiving tool for Windows that opens RAR and ZIP files. Compatible with many other file formats.

www.win-rar.com

Edit: to be clear I would still lean towards using PAR files alongside the archive instead of using the RAR recovery record - PAR allows a bit more flexibility

 

[nandobadam]

Winrar is installed on my PC. Does the corruption test with Winrar work if I open it and then select the zip, rar and 7z files that are on the external hard drive connected to the PC and click "test" in winrar?

 

[Onasi]

@nandobadam
Yes. The disk is mounted and for the OS it’s essentially no different from an internal HDD. The test, from my understanding (and RTFM), will perform a dummy extraction (without actually extracting the files) to validate the output. It doesn’t write anything. It will still take time proportional to the archive size since it needs to read it.

Honestly, you could have just, y’know, just tried it instead of asking. Or read the documentation for WinRAR pertaining to the command:

"Test archived files" command - WinRAR Documentation

documentation.help

Command T - test archive files - WinRAR Documentation

documentation.help

 

[nandobadam]

@nandobadam
Yes. The disk is mounted and for the OS it’s essentially no different from an internal HDD. The test, from my understanding (and RTFM), will perform a dummy extraction (without actually extracting the files) to validate the output. It doesn’t write anything. It will still take time proportional to the archive size since it needs to read it.

Honestly, you could have just, y’know, just tried it instead of asking. Or read the documentation for WinRAR pertaining to the command:

"Test archived files" command - WinRAR Documentation

documentation.help

Command T - test archive files - WinRAR Documentation

documentation.help

Yes, I tested it as I mentioned but I didn't know it worked correctly, this type of test puts a lot of stress on the HDD. Does this test also prevent the plates from demagnetizing?

 

[Onasi]

this type of test puts a lot of stress on the HDD

It does not.

Does this test also prevent the plates from demagnetizing?

Any meaningful activity should do. Best theoretical way is a full re-write of the drive, but it might be excessive.

 

[Shrek]

The contacts from the hard drives board to the heads can corrode, especially in a humid environment.

Here’s a Surprising Trick Repairing Hard Disk (HDD) | Electronics Repair And Technology News

 

[A Computer Guy]

Does this test also prevent the plates from demagnetizing?

I would be 1000% more worried about accidently dropping your USB hard drive than demagnetizing. If you are really worried about demagnetizing then work into your backup routine to reformat one of your drives in rotation every few months and make fresh copy.

 

[Wirko]

Hashes are enough, and more practical.

Yes, but hashes are only valid the moment they are calculated. If you compare stored hashes of two files but don't read the entire files, that's fast and practical - and you still don't know if any one of the files is corrupted.

If someone's uneasy with just CRC32, then MD5, SHA1, or others.

CRC-32 has a small advantage: it can correct single-bit errors (or a few bits in the best case) exactly because it's not a cryptographic hash. Not sure if this is of any value in real life, probably not.

 

[A Computer Guy]

Yes, but hashes are only valid the moment they are calculated. If you compare stored hashes of two files but don't read the entire files, that's fast and practical - and you still don't know if any one of the files is corrupted.

CRC-32 has a small advantage: it can correct single-bit errors (or a few bits in the best case) exactly because it's not a cryptographic hash. Not sure if this is of any value in real life, probably not.

One advantage of the hash or CRC is you only need to completely read the file you are validating to compare against the hash or CRC vs. the binary compare you are reading both files in their entirety.

 

[Wirko]

As far as I'm aware, RAR is the only format where you can specify a parity recovery amount so it can recover from XX bytes of data corruption instead of just detecting it.

That's probably useful if you transmit files over unreliable networks. Then you can set a percentage based on experience and expectations (how reliable is the link and what's the acceptable rate of unrecoverable errors, requiring re-transmission). But what about storage? How could one calculate the right (whatever this means) percentage?

One advantage of the hash or CRC is you only need to completely read the file you are validating to compare against the hash or CRC vs. the binary compare you are reading both files in their entirety.

You're right. It depends on the purpose. If you can declare one of the files as "known good, doesn't need revalidation" then yes. In my backup procedure, that's not the case.

 

[Vincero]

That's probably useful if you transmit files over unreliable networks. Then you can set a percentage based on experience and expectations (how reliable is the link and what's the acceptable rate of unrecoverable errors, requiring re-transmission). But what about storage? How could one calculate the right (whatever this means) percentage?

There isn't a set 'right' amount.
It's like saying what's the right amount of sugar in a coffee...

How many parity disks in a RAID set is the right amount? 1, 2, 3, ...?

Sure there are guidelines if the amount of data disks is >5, >20, etc., but there isn't a 'right' answer... Entirely depends on the value of the data to the individual.

(And my reference to RAID is not implying it is an alternative to a backup/archive).

The alternative options the OP could consider is a cheap NAS enclosure and using something like ZFS or BTRFS with parity functions enabled to effectively transparently manage this all.

 

[nageme]

If you can declare one of the files as "known good, doesn't need revalidation" then yes. In my backup procedure, that's not the case.

By definition, the original / source file is "known good". You hash it, and get a "known good" hash. That's your basis for any future verification.

 

[eidairaman1]

This post.

Then 10 replies follow.

Then you repost it again, Verbatim (minus the 'yes'.)



You did this in a previous thread. And you keep reformulating your questions.

Stop thread spamming with the same posts.

This 1 has several different threads relating to this it would appear

 

[Blue4130]

This 1 has several different threads relating to this it would appear

He also has it posted in three different threads on the servethehome forum and at least 3 threads on ltt forums... And his critical data that he is backing up.... ROMs and console emulators.

 

[Dr. Dro]

He also has it posted in three different threads on the servethehome forum and at least 3 threads on ltt forums... And his critical data that he is backing up.... ROMs and console emulators.

I mean, what he stores in the drives is his business, but the fact remains this dude bought some random used drives with more than a decade of use on their back from AliExpress and was told that they are potentially unreliable and could have false SMART data, so instead of doing the logical thing and buying a replacement from a reputable source, with warranty, etc. he keeps posting these in multiple forums, with minimal rewording just to reassure himself. I actually vaguely remember some pretty old posts about this years ago, I wonder if it's the same guy.

People here at TPU have been very constructive in the threads but at this point, he was told everything that he needs or would like to know and more. Past a point it just becomes obnoxious, really. It seems he started at least 9 threads on this exact subject by now, at least from what I can see with regular user permissions, who knows if the mods deleted or merged some.

I don't like to backseat moderate and love to read constructive posts (which the community has provided), but looks like the OP is not exactly interested in knowledge, feedback, exchange of ideas or even reassurance at this point. I just question why continue to ask the same question over and over again, if they are going to get the same answer, you know that quote about the meaning of madness or whatever.

 

[eidairaman1]

He also has it posted in three different threads on the servethehome forum and at least 3 threads on ltt forums... And his critical data that he is backing up.... ROMs and console emulators.

It's not rocket surgery this 1 claims it to be. @nandobadam have local usb, optical drive, and induction/flux based storage, also encrypted "cloud". Keep 1 backup that doesn't get touched but to read from it once in a blue moon and another that is updated, put the media in a dry dark cool place use pelican cases.

 

[Shrek]

Of course there is also the 1000-year M disk

 

[AusWolf]

Again, you cannot test for corruption without anything to compare WITH. You need either a copy of the files or a checksum to compare against. Nothing will tell you if the files are corrupted or not in sheer vacuum.

Well, it's a compressed file, so if trying to decompress it results in errors, that might be a sign that the file is corrupted. Unfortunately, there's nothing left to do then.

I don't know why OP is still hell bent on compressing files. Quite a few of us have already said why it's a bad idea.

 

[nageme]

@AusWolf
Though, there's the option of using RAR + recovery record, or PARs, or plain redundancy.

some random used drives with more than a decade of use

Old models doesn't necessarily mean they were in use the whole time. Might even be NOS.

I'm not at all a fan of used drives, from unknown sources or otherwise, but multiple copies on multiple used drives mitigates the risk.
Especially if the drives aren't the same brand and model.

CRC-32 has a small advantage: it can correct single-bit errors (or a few bits in the best case) exactly because it's not a cryptographic hash. Not sure if this is of any value in real life, probably not.

Interesting. Any software that implements it?
I couldn't find concrete details, but I somehow doubt it could fix any 1-bit error in large files. Large being, more than extremely small.

 

[Dr. Dro]

Old models doesn't necessarily mean they were in use the whole time. Might even be NOS.

Known not to be the case with dirt cheap enclosures from Aliexpress. One of these enclosures OP got actually came with an Apple HDD inside which is a dead giveaway it's salvage from a recycled MacBook

 

[A Computer Guy]

I don't know why OP is still hell bent on compressing files. Quite a few of us have already said why it's a bad idea.

It's less file system overhead to copy around a large compressed file especially if you have thousands of small files otherwise. I think in 7zip you can also do storage mode without compression so essentially making one big file from whatever folder your working on.

 

[AusWolf]

It's less file system overhead to copy around a large compressed file especially if you have thousands of small files otherwise. I think in 7zip you can also do storage mode without compression so essentially making one big file from whatever folder your working on.

A fair point. That advantage is still outweighed by the drawbacks, imo.