RAID transfer SLOOOW

[taz420nj]

Ok, so I got my new array set up on Plywood, and I started transferring media over from the old one - but the transfer rate is horrendous. For the first few minutes it was screaming along at 350MB/s, basically maxing out the SATA2 interfaces on the old RAID card, then it dropped down to 45MB/s. Now I know RAID 5/6 have a parity penalty, but the H700 is a decent card and it shouldn't drop that low when the buffer fills up (and the drop occurred after dozens of GB had been copied, the card has a 512MB cache).

I read about the stripe size and cluster sizes causing performance issues when large files are concerned, so I have tried several different combinations (including ReFS) and now ALL transfers are topping out at 45-50MB/s - it doesn't even start fast and slow down anymore. WTF???

HD Tune Pro shows benchmark performance on the H700 to be over 800MB/s read and write and on the LSI to be roughly 300MB/s read and write on an 8GB file (the average size of my media). I'll post screenshots in a bit.

Here's the setup..

SuperMicro X7DBE motherboard
2x Xeon E5430 Quad Core
24GB RAM

Original Array:
3Ware/LSI 9550-8 PCI-X (SATA2) w/BBU (in a 133MHz PCI-X slot)
4x Hitachi Deskstar SATA3 2TB 5900 RPM

New Array:
Dell PERC H700 (SAS2/SATA3) Integrated PCIe x8 (in a x8 PCIe 2.0 slot)
512MB Cache, BBU installed, Write-Back enabled (Dirty Cache light comes on during transfers so cache is functioning properly)
8x Hitachi Ultrastar SATA3 2TB 7200RPM
Connected via 2x SFF-8087 SAS to 4xSATA3 fanout cables

Any ideas?

What stripe size/cluster size/file system should I be using? The video files are large, but each one has associated screenshots/posters/XML files for library purposes, and then there's my music library and programs (mostly ISOs of install media).

 

[blobster21]

If the "PCIE Max Payload Size" setting is available in your motherboard's bios, how is it configured right now ? 128B ? 256B ?

i have had some issues with my adaptec RAID adapters in the past with anything but the default value (128B)

 

[taz420nj]

I'll have to check but I do remember a PCIe setting with 128 and 256B as the options.. "Coalesce"? It was set at 128B.

Edit: Yeah PCI-E I/O performance is set to Coalesce/128B

 

[blobster21]

As for your question about stripe size, i'm going with the default value of 256K on both RAID0 and RAID5EE.

The cluster size is a different matter, as my overlaying file system is ext4, and my setting probably doesn't apply to you, though i am basically storing/accessing the same kind of data as you, size wize (you are on NTFS, isn't it ?)

 

[taz420nj]

The default stripe/chunk on the H700 is actually 64K.. It was originally at the default (and NTFS formatted as default size, which is I believe 4K) when I was getting the fast then slow down.. I have tried 32K, 128K, and 256K..

And yes it is a Windows server, so I have NTFS and ReFS available.

 

[blobster21]

so both cards bench pretty well when they are tested separately, and then the speed is stuck around 50MB/s when it comes to move data from one array to the other....

Could you plug a spare SSD on an empty motherboard sata plug, and transfer some data from/to the largest array ? At this point it's worth verifying if the problem occurs only when both arrays are involved...

Edit : just another guess : there could be some bottleneck because both RAID adapters shares the same ressources, you know how the slots sometimes have their lanes closely intertwined...do you have some spare slots to move the cards apart from each others ?

 

[newtekie1]

I almost wonder if the Windows write cache isn't responsible for the high transfer speeds at first, then the slow down after several GB has been "written". If I'm not mistaken, when you format a volume with ReFS, it automatically turns off the Windows Write caching because it is supposed to be reliable. That is why your transfers are always slow when you use ReFS, but fast at first when you use NTFS.

I bet if you formatted it to NTFS, but turned off the Windows Write caching for that volume, it would also always be slow.

 

[eidairaman1]

@cdawall

 

[cdawall]

Is the card/chipset over-temping? That'll do it as well especially on the older ones.

 

[Aquinus]

And yes it is a Windows server

Write buffering. Thanks, Windows. What you're seeing is the read speed from the drives you're copying off of until the buffer is filled (which could be as much as 4GB.) Then it drops down to what you're seeing for write speed at the destination because you're no longer writing directly to memory and are then bottlenecked by writing to the target.

Stress test the new array first before putting it to use.

 

[taz420nj]

so both cards bench pretty well when they are tested separately, and then the speed is stuck around 50MB/s when it comes to move data from one array to the other....

Could you plug a spare SSD on an empty motherboard sata plug, and transfer some data from/to the largest array ? At this point it's worth verifying if the problem occurs only when both arrays are involved...

Edit : just another guess : there could be some bottleneck because both RAID adapters shares the same ressources, you know how the slots sometimes have their lanes closely intertwined...do you have some spare slots to move the cards apart from each others ?

I don't have a spare SSD, just the one in my laptop and I don't feel like pulling it apart right now.. I tried copying from a 1TB hard drive and it maxed at 70MB/s.. Still way slower than it should be.

It can't be shared resources, the one card is PCI-X and the other is PCIe.

I almost wonder if the Windows write cache isn't responsible for the high transfer speeds at first, then the slow down after several GB has been "written". If I'm not mistaken, when you format a volume with ReFS, it automatically turns off the Windows Write caching because it is supposed to be reliable. That is why your transfers are always slow when you use ReFS, but fast at first when you use NTFS.

I bet if you formatted it to NTFS, but turned off the Windows Write caching for that volume, it would also always be slow.

I only tried ReFS last.. It was topping at 45MB/s on every combination of factors after the original. Even after going back to the original configuration (64K stripe, NTFS default cluster size), it will not go back to the "fast burst" that I was getting before.

Is the card/chipset over-temping? That'll do it as well especially on the older ones.

I doubt it, there's a fan blowing right past it.

Write buffering. Thanks, Windows. What you're seeing is the read speed from the drives you're copying off of until the buffer is filled (which could be as much as 4GB.) Then it drops down to what you're seeing for write speed at the destination because you're no longer writing directly to memory and are then bottlenecked by writing to the target.

Stress test the new array first before putting it to use.

If this array were on another really old slow card like the 9550, I might buy that. But I don't believe for a second that the H700 is only capable of calculating parity at 45MB/s. It's an enterprise card that's only one generation old.

 

[Aquinus]

If this array were on another really old slow card like the 9550, I might buy that. But I don't believe for a second that the H700 is only capable of calculating parity at 45MB/s. It's an enterprise card that's only one generation old.

In RAID, the entire array is only as fast as the weakest link. It only takes one drive to slow the entire thing down with any RAID level. If the slow drive gets kicked out and the array becomes degraded, it's not unrealistic for the degraded array to be faster than before the bad disk was kicked out but until then, the slowest drive will hinder the array.

Edit: Could you get a dump of the SMART stats for all of the drives in the 8-disk array? Also, is that RAID 5 or 6?

Edit 2: GSmartControl can read through most RAID controllers. I would suggest using this for either Windows or Linux (if you're in a GUI environment.)

 

[taz420nj]

None of the drives are reporting any issues. If there were any problem it would benchmark around the same rate but it's going at full expected speed.

The new array is RAID 6.

Heres screenshots... In the last one it started at like 100 then dropped almost instantly to 30, then it settled in right around 45-46...

 

[Toothless]

@FordGT90Concept

 

[blobster21]

i have seen this behaviour with regular non-raid disk-to-disk copy, and also through samba shares.

The set of icons in the quicklaunch toolbar makes me think you are running 2012 (R2 ?), and if my memory serves, i was on 2012R2 when this occured.

If you want to rule out the write caching influence during your file copy, you could give a try to this advice :

The server most likely has a high System cache usage, monitor the "system cache resident bytes" counter, its located under the "Memory" object in perfmon. as workaround you can use Xcopy with the J parameter to use un-buffered IO

Try Xcopy /J -- it uses un-buffered IO and should not have an overhead on the system cache

I would also suggest you to enable hardware write caching for the disks

https://social.technet.microsoft.co...formance-on-2012-standard?forum=winserver8gen

 

[taz420nj]

i have seen this behaviour with regular non-raid disk-to-disk copy, and also through samba shares.

The set of icons in the quicklaunch toolbar makes me think you are running 2012 (R2 ?), and if my memory serves, i was on 2012R2 when this occured.

If you want to rule out the write caching influence during your file copy, you could give a try to this advice :

Yeah it's R2.. I'll give that a try when I get home from work.

Sorry for the delay, off for the weekend so I have some time to mess with this..

I am seeing the same transfer rate using xcopy /j...

 

[blobster21]

Ouch

Speaking of file copy performances, how fast would be each RAID array if you decided to copy and paste a large folder internally ? (aka from D: to D:, and then from G: to G

I have the feeling that the system bus is the culprit....

 

[Steevo]

Did you perform a long format on the array once it was initiallized to make sure that everything was set up? also you have to remember a raid five it has to calculate parity and stripe for the spanning stripe and if the disk format wasn't a full one and it's also trying to perform format at the same time as his writing data it will be horrendously slow.

 

[taz420nj]

The plot thickens... I tried a couple different folders (nothing special, all have the same file types - .mkv video file, a few .jpgs and an xml), and some do seem to transfer at full speed.... This one transferred 8GB in about 15 seconds..

 

[Aquinus]

Big files are going to write very quickly because sequential write speeds is where RAID shines. If you're copying a ton of small files, it's going to slow right down because you're doing more writes that are smaller and you're touching things like the file system more often for every files being written when you have small files versus big ones. As a result, transfer rate for small files will always be less than a single large file. SSDs mitigate this but, this is most definitely a thing for rotational media drives. So what you're copying actually has a lot to do with how quickly it will copy.

 

[taz420nj]

Did you perform a long format on the array once it was initiallized to make sure that everything was set up? also you have to remember a raid five it has to calculate parity and stripe for the spanning stripe and if the disk format wasn't a full one and it's also trying to perform format at the same time as his writing data it will be horrendously slow.

No I quick formatted. Do you have any idea how long a 12TB long format would take lol?

Big files are going to write very quickly because sequential write speeds is where RAID shines. If you're copying a ton of small files, it's going to slow right down because you're doing more writes that are smaller and you're touching things like the file system more often for every files being written when you have small files versus big ones. As a result, transfer rate for small files will always be less than a single large file. SSDs mitigate this but, this is most definitely a thing for rotational media drives. So what you're copying actually has a lot to do with how quickly it will copy.

There's only a few small files in each folder, maybe up to 15. And it always seems to transfer the video file first, so that's what it's measuring. I've also tried just transferring the actual video file without the small files.

 

[Aquinus]

Edit: Could you get a dump of the SMART stats for all of the drives in the 8-disk array?

@taz420nj I'm still standing by this one. The attributes will tell us the state of the drives or if there have been any problems, even if it's not currently having one. Something could stand out and SMART is literally the first thing you should check with respect to any kind of storage issue.

 

[taz420nj]

@taz420nj I'm still standing by this one. The attributes will tell us the state of the drives or if there have been any problems, even if it's not currently having one. Something could stand out and SMART is literally the first thing you should check with respect to any kind of storage issue.

Like I said, not showing any issues. And yes I know about the hours on them, it's already been taken care of with the place I bought them from.

It's disks 3-10.

 

[blobster21]

After reviewing the photos of your media server, it seems possible to move the perc H700 on the second PCI-E 8x slot, it would be interesting to test it in this configuration.

You could also do it the other way around and move the LSI9550 on another PCI-X 133Mhz, since you've got one on spare ?

Correct me if i'm wrong, but your LSI adapter is plugged in a PCI-X 100Mhz slot, according to the user's manual, page 10 (X7DBE rev 2.0 has a slight color scheme modification for the PCI-X 100Mhz slot, the one on the bottom is no longer blue but it's still 100Mhz nonetheless)

Interestingly, on page 69, i read you can also "enable a selected device as the PCI bus master", in the PCI-X/E slots pool ( Slot1 PCI-X 100 MHz ZCR, Slot2 PCI-X 133MHz, Slot3 PCI-X 133MHz, Slot4 PCI-Exp x4, Slot5 PCI-Exp x8, and Slot6 PCI-Exp 8x )

Yes indeed, the disk report shows no problems on the raid6 array nor the LSI one .

 

[Steevo]

No I quick formatted. Do you have any idea how long a 12TB long format would take lol?



There's only a few small files in each folder, maybe up to 15. And it always seems to transfer the video file first, so that's what it's measuring. I've also tried just transferring the actual video file without the small files.

Yes, yes I do. I have done over 20TB in RAID5 and many controllers and Windows will only do a small amount of large volumes to speed up setup processing at the expense of performance later unless you tell the Raid card to perform a full initialization and on cheap RAID cards the only way to give this is with a full format or a drive free space wipe.