RAID spontaneously degrading (Intel RST)

[Violet_Shift]

Had a look at GSmartControl's output on that particular disk.

Here are the results, but I think I can interpret this myself.

The rest of the drives except this one have nothing listed as suspect. (No errors, etc). The most recent of these errors, according to the time logged, was 6 hours ago - probably while I've been copying hundreds of gigabytes of stuff on to other drives to save it.

Could this drive's behaviour also have been responsible for the "failure" six months ago that showed identical symptoms? I replaced that disk but maybe it's actually OK and this is the one that I have to worry about.

 

[Aquinus]

Had a look at GSmartControl's output on that particular disk.

Here are the results, but I think I can interpret this myself.

The rest of the drives except this one have nothing listed as suspect. (No errors, etc). The most recent of these errors, according to the time logged, was 6 hours ago - probably while I've been copying hundreds of gigabytes of stuff on to other drives to save it.

Could this drive's behaviour also have been responsible for the "failure" six months ago that showed identical symptoms? I replaced that disk but maybe it's actually OK and this is the one that I have to worry about.View attachment 68305 View attachment 68306

All of the errors being within the same hour is interesting. I recommend running an extended SMART diagnostic test. If it generates an error, you probably have a drive that's on its way out.

 

[FordGT90Concept]

Still odd that the drive with SMART errors isn't the one that fell out of the RAID.

 

[Aquinus]

Still odd that the drive with SMART errors isn't the one that fell out of the RAID.

Then run extended tests on all the drives. That behavior sounds like no TLER and patrol read enabled. It's probably timing out when trying to recover a block and takes too long. Also keep in mind this drive is only throwing errors because blocks are non-correctable. The other drive might be correcting the blocks but, it may be timing out because of no TLER.

The OP should check SMART attribute IDs 5, 197, and 198 on the other drives as well.

 

[Violet_Shift]

Right, running extended self tests on all four drives.

The one I just showed the errors from failed immediately and is telling me "Completed with read failure". I'm removing this one from the RAID as soon as I can and tossing in the old "failed" one to see if that one is an innocent victim in all of this once all the self-tests are done, assuming there's nothing wrong with the other three.

All I know at this point is that I'm never touching another Seagate product again. I've used WD all the time before I built this RAID (sadly in the aftermath of the whole Thailand thing) and I never had a problem.

 

[Slizzo]

Yeah, I have a hard time looking at replacement drives for my old 1.5TB WD green. The cheap drive is always the Seagate. May go with a Toshiba or HGST eventually. WD Blacks are just too expensive now IMO.

 

[FordGT90Concept]

It depends on the model more than the brand. All hard drives are not created equal.

 

[Violet_Shift]

Yeah, it seems that the 4 tera Seagate drives have better failure rates. Sadly I don't have the $ to replace the whole array at this point so I'm praying that the one that "failed" 6 months ago is okay. I'll test it once all this is done. At 50% on the extended self-tests.

 

[FordGT90Concept]

I'm pretty sure Intel RST doesn't consider "caution" a failed state. It runs the drives until SMART switches to "failed." Don't get me wrong, the drive should be replaced but I'm not confident it will fix the RAID degrading.

 

[Violet_Shift]

Next step is the firmware update then (maybe that's why they're dropping?). Still waiting on the extended surface scans to finish.

I had a look at the SMART status as the old drive that fell out of RAID six months ago and it has the same problems on it.

 

[Violet_Shift]

I decided to upgrade the firmware on the drives now, but I've ran into (I think) a minor problem. I executed the upgrade, and it seems to be done, but the program isn't prompting me to do anything.

The guide online told me to turn the computer off when prompted but it's been a few hours with the screen never changing and I've received no such prompt:


Already probably lost my data from doing this (fortunately backed up, still the frustration factor) but I really don't want to wreck the 3 good drives. Anyone know what to do next? Am I good to turn the machine off?

 

[FordGT90Concept]

It looks done to me.

Z1F10KGX CC4B -> CC4H
Z1F10BAH CC4B -> CC4H
W1F4TWMN CC29 -> skipped
Z1F0YM7D CC4B -> CC4H

The three that the firmware applied to were updated.

 

[Violet_Shift]

Thanks. I was just a bit paranoid. Always am when it comes to firmware.

Anyway, after a few days I've not had any problems, so I presume the firmware update corrected the problems.

Now there's still the question of the suspect hard drive which I do want to replace before it causes any problems, but I'm very hesitant to buy another ST3000DM001.

If I progressively replace them with the WD Black WD3003FZEX drives (one at a time with a rebuild after each of course), will I be ok? They have the same capacity, interface, and spin rate, but I am concerned about mixing drives regardless.

 

[Solaris17]

CrystalDiskInfo does a piss poor job at reading SMART information. Could you use GSmartControl and let us know what it reads for those two fields? If the raw data is simply a hex value, you have 84 uncorrectable sectors, GSmartControl should confirm that. That could be the reason for the RAID becoming degraded.

You can get GSmartControl here.

I highly recommend this tool because it can detect through most RAID setups, properly detects raw values for SMART attributes, and allows you to view and run SMART diagnostic tests.

I personally think that this is the best GUI tool for this task.

I dont gsmart fucks up SMART reads on SSDs all the time at work I recommend CDI

 

[FordGT90Concept]

Personally, I'd just buy a hard drive with a 3+ year warranty and put it on the shelf until Intel RST kicks the drive out of the RAID. Then I'd swap it.

Yeah, of course you'll be okay. RAID has the best performance using identical drives but we're talking a very tiny difference in performance. It's actually better to have a variety of drives because then you're less likely to lose two drives simultaneously which, on RAID5, is fatal.

 

[Aquinus]

I dont gsmart fucks up SMART reads on SSDs all the time at work I recommend CDI

Could you explain that in a little more detail? Do you mean it gets the attribute names wrong or that it screws with actually reads from SSDs? CDI sucks because it doesn't tend to read values properly which leaves you to trying to interpret the raw data. I've also had no issues using smartmontools (the library under gsmartcontrol,) on several SSDs.

 

[Solaris17]

Could you explain that in a little more detail? Do you mean it gets the attribute names wrong or that it screws with actually reads from SSDs? CDI sucks because it doesn't tend to read values properly which leaves you to trying to interpret the raw data. I've also had no issues using smartmontools (the library under gsmartcontrol,) on several SSDs.

I've never had issues with CDIs attribute reads. I have however had GSMART tell me drives are fine when they are clearly failing no boots, system disk missing lock up on data access R/W but CDI would display the issues clear as day.

 

[FordGT90Concept]

For the record, SeaTools is going to be the best way to test Seagate drives for failures. If you're thinking RMA, you need the number it generates anyway. They have to be removed from the RAID for SeaTools to detect it though (at least last time I tried).

 

[Aquinus]

I have however had GSMART tell me drives are fine when they are clearly failing no boots, system disk missing lock up on data access R/W but CDI would display the issues clear as day.

What in particular would that be that CDI saw that GSC didn't? I find that GSC gives more information than CDI, not less. It's nice being able to check the SMART error log and the results of SMART diagnostic runs though, I don't recall CDI being able to access much beyond the attributes themselves.

Also consider what the OP saw with it. It's clear as a sunny day that he has at least 1 drive with issues. All of those errors happened within hours of the OP sending the screenshot.

 

[Solaris17]

What in particular would that be that CDI saw that GSC didn't? I find that GSC gives more information than CDI, not less. It's nice being able to check the SMART error log and the results of SMART diagnostic runs though, I don't recall CDI being able to access much beyond the attributes themselves.

Also consider what the OP saw with it. It's clear as a sunny day that he has at least 1 drive with issues. All of those errors happened within hours of the OP sending the screenshot.

I don't really need to consider anything about what the OP said. I guess i'm not really understanding why you are attempting to argue? Gsmart hasn't worked correctly on hundreds of drives (I literally analyze thousands a year) Attributes specifically wrong on gsmart are the wear attribute indicator and bad/pending sector count.

What in particular would that be that CDI saw that GSC didn't?

huh?

I have however had GSMART tell me drives are fine when they are clearly failing no boots, system disk missing lock up on data access R/W but CDI would display the issues clear as day.

just going to nip that before it becomes a problem. I suppose im just going to attempt to make it clear that im after the CORRECT information. I could care less if CDI was missing something like the short test etc.

 

[brandonwh64]

This is what mine looks like but the its still fully functional and only real issue I see is that speeds are around 550-600MB instead of 1000+ as they where when I first set it up.

 

[FordGT90Concept]

When you click on the drive that has the exclamation point, what does it say? Since it is an SSD, I suspect that ! drive went into read-only mode which is why it is 550 MB/s now but the RAID0 is still functional (the non-failed drive is accepting all the writes).

 

[Aquinus]

I don't really need to consider anything about what the OP said. I guess i'm not really understanding why you are attempting to argue? Gsmart hasn't worked correctly on hundreds of drives (I literally analyze thousands a year) Attributes specifically wrong on gsmart are the wear attribute indicator and bad/pending sector count.

huh?

just going to nip that before it becomes a problem. I suppose im just going to attempt to make it clear that im after the CORRECT information. I could care less if CDI was missing something like the short test etc.

I've found issues with drives causing occasional issues by using the extended test, I wouldn't call that worthless. Also considering these tests can be run independently of the OS is a plus. The error log also is incredibly useful and the usefulness is shown in the OP's screenshots. I agree that the GSmartControl isn't ideal for reading SSDs, but what is the OP diagnosing again? The problem is that SMART isn't consistent across vendors but tends to be more consistent with HDDs as opposed to SSDs. According to GSmartControl, my SSDs are in read only, have no reserved blocks left, and is ready for the graveyard. However retired block rate is zero, there have been zero erase failures, no program failure counts, and no reallocated sectors. So sure, an attribute or two might read inaccurately thanks for a vendor being dumb but, I think that it comes down weighing out what looks right and wrong. Not all manufacturers use SMART the same way depending on the attribute. It's a pain in the ass as you very well know but, it happens. My point is that GSmartControl is a good tool to have available should you need it and at least for spinning disks, tends to work well.

I'll agree that with SSDs, it might not be the best option but, with spinning disks, I think that's a matter that is open for debate because my experience with HDDs with this tool has been very different. The library GSmartControl uses under the hood (smartmontools) is practically the thing to use if you're administering linux servers and want to check SMART information from the command line. I've found it capable for my own machine and for the servers at work so, I'm just speaking from my own experience which has been mainly with spinning disks.

I long for the day that vendors actually become consistent with this kind of thing across all kinds of drives. The problem is SMART is like SAML, people implement that parts they need and kind of munge it to work the way they want it to. It sucks half of the time as a result but, that doesn't mean it isn't a useful tool to use.

Either way, lets agree that GSmart isn't ideal for SSDs and leave it at that.

 

[Violet_Shift]

Well, I thought my problems are over. I replaced the drive that was having errors with a WD drive, and things were fine for a few days.

Then, less than half an hour ago, while I was about to play a game (Dragon Age: Awakening for the record, I know that that doesn't matter, but I'm in a mood to post completely irrelevant things), and then as the game hit main menu, the music started looping, and after I'd killed the process I decided to restart the machine (at this point I checked RST just to be sure nothing was wrong).

I restarted the machine, and my pre-boot RAID settings tell me that the array has failed, and that one of the drives has fallen out. Not just degraded, but failed. I started the computer, and sure enough, the RAID isn't accessible, and GSmartControl tells me that the drive in slot 4 has had an "end to end" error, while RST can't even see it.

I turn it off and remove the suspect drive, and it's switched to "degraded" and the computer can see the array again, and data -seems- okay (I'm not sure if I should trust it).

It was the drive that was seeming healthy but was being thrown out of the array. And now it seems completely bricked - the computer won't even boot when it's connected.


So I know my course of action now is to replace this one, but my question is, if it once showed as failed, and the machine couldn't even see it, and now it's showing as degraded, is everything OK? Can I rebuild again and proceed as normal, while staring with suspicion at the remaining 2 ST300DM001's? (one of them is a bit newer so I might trust that a slight amount more).

Boy has this been an expensive string of computer problems.

 

[FordGT90Concept]

I'd rebuild... you have to in order to protect the data. I've never seen this kind of behavior so I'm scratching my head on it.


Definitely run an extended test on that Seagate drive with SeaTools. If it doesn't generate an ID, the drive is perfectly perfect--the RAID controller is suspect.