Huge SSD NVEm + SAS Setup Plans

[JAB Creations]

My future is looking bright financially and I'll be upgrading a lot of my system in the coming months. Before I get in to specifics this is what I use my rig for (my rig's specs are under my profile on the left):

• Web Development primarily, about ~60% of the time at least.
• Reading, research, ~10%.
• Entertainment besides gaming, basically eating my meals and enjoying something, about ~10% of my time.
• Gaming, about ~5% of my time, I'll go weeks without gaming sometimes but money pays the bills.
• I don't encode audio/video often, I do though it's negligible.

Storage is the topic here and here are my goals:

• C:\ can be a single NVMe SSD, likely a 4TB drive.
• D:\ will be two NVMe SSDs in RAID 1, likely a 4TB drive.
• E:\ will be a mass storage of at least 8TB though I'd prefer 16TB.
• The C:\ and D:\ drives will be the same so I can interchange them if necessary.
• I will have two cold spares for the C:\ and D:\.
• I will image one of the the C:\ and D:\ every couple of months for cold storage.
• I will have two cold spares for the E:\.
• I will image one of the E:\ spares every couple of months for cold storage.

Whatever drives I work with I plan on making clear how the speed, RAID and other details are clearly understood so others can benefit. As someone who solves problems one of the things I loathe the most is seeing "worked, thanks!" replies with zero details that negates any value to anyone else reading someone else's threads. In a few months once I have this going if I haven't updated this first post someone please politely remind me.

Because higher capacity SSDs are generally available in the enterprise segment I imagine that Threadripper motherboards don't have SAS. When the 4950X comes out that is what I'll upgrade to, I don't need Threadripper as I don't run much in the way of audio/video encoding (I do, but not with all the money in the world to justify it). I just came across a 16 port SAS controller for $471 on Newegg. Here is what I'd like performance wise:

• C:\, D:\ will be NVMe without a doubt and that controller will definitely be running in RAID mode. I'd like to get the transfer rates up and I favor Samsung as I've had trouble in the past with Crucial setting a couple SSD drives in RAID 1, no problems with Samsung and I figure that they'll have TLC 4TB NVMe drives out in a couple months tops by now.
• E:\ will likely be an addon PCI-Express card and I only need two drives. SAS bus allows a faster transfer rate, I think ~1 gigabyte a second? I'm not sure as it'll be my first experience with SAS. I'm not sure how much difference one SAS controller is from another though I'm always willing to spend more for quality. I can live with a ~5X improvement over my current mechanical RAID 1 setup.

I'm mostly interested in reasonably high (but not blind / RAID 0 non-sense) performance blended with reasonably high capacity. I always keep games, applications and the operating system separate from my work files and anything mass storage like video editing, ISOs, game installers from old CDROMs/DVDs, etc on the mass storage drives.

I'm looking forward to Samsung's TLC 4TB NVMe drives coming out soon for RAID. I'm not sure about the SAS cards though there are 16TB SAS enterprise SSD drives out.

For the past few years when (not if) Windows takes a dump I just plug in a spare SSD like a floppy disc, recover what I need, clone over and I'm back up and running sans a few updates in less than an hour. My main goal is to generally avoid having to mess with my main rig once the new parts become available because as I much as I enjoy tinkering I enjoy making money a hundred times more.

This will be going on to my 3800X / X570 rig and I expect to be doing this sometime roughly around mid-October. Suggestions please?

 

[Solaris17]

Why are you going with SAS? Do you have a specific need? SAS is generally aging and while it has some clear advantages over SATA your use cases are going to be moving big data work loads to SAN appliances not some random PC with a bunch of drives in it. I'm sorry but photoshop and 4k video workloads aren't going to be enough justification.

Most people I have seen in the DC space if not going for an appliance like nimble usually have supermicro or dell chassis or even QNAPS and higher end synologys with RAID 10 for performance and redundancy and depending on burst workload an SSD cache tier.

You might even be better off using an NVMe card as long as your mobo can use pci-e bi-purification.

To be certain, I sit around PB of data several times a week and while expensive drives and RAID controllers are the norm, I think I can count in only double digits the amount of SAS drives and appliances in use. Maybe its a monetary flex?

In either case, depending on need, you might want to explore other options, USB-C or 10G NAS with the ability to RAID 10 with storage tiering? Maybe atleast then the ability for hotspares and auto rebuilding may be apparent and behind an easy button.

Not sure how many $400 RAID cards you have used personally, but pretty much every LSI dell or HP card I have ever used requires proper setup before It can be put into production and the interfaces are less than friendly.

Food for thought maybe.

 

[JAB Creations]

Great insight, hence why I posted. I'm interested in SAS because going beyond 4TB (this year) will mean going beyond TLC. I'd prefer to simply have a couple of 16TB NVMe drives for my D:\ though the capacities simply aren't there yet and therefore I will continue to segregate my data. I could opt to get an additional PCI-Express NVMe addon card and use that along with a couple of 8TB NVMe drives (Sabrent Rocket being the only option currently that I'm aware of) though I'd be at roughly 5TB of used space once the dust is settled, maybe a little less if I migrate some of that data to the D:\.

Again, I'm not working with gobs of data, I just want to make a lasting investment in storage so I can concentrate on my software business instead. I still have plenty of time to determine what I want to do so for me this is a great time to get input from others.

I'm not familiar with "pci-e bi-purification", could you please clarify what that is?

 

[theonek]

interesting choice of a vast ssd capacity and amount, though consumer drives are not so durable as server class ones. Although a spinners remains the leader in capacity and price which you can consider for storage the essential info and all other left on ssd's. Nas is the solution for on/off-line stotrage and ssd's on you main pc for fast access and work.

 

[dgianstefani]

Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.

 

[sepheronx]

Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.

But there is a lot of benefits to RAID especially if that 16TB SSD fails.

 

[dgianstefani]

The likelihood of that happening on server grade ssds is incredibly low.

 

[Chrispy_]

C for OS, D for production data? I think you'd be better served with a higher-performance 1-2TB OS drive, ideally with a large SLC buffer so that you're never hitting raw writes on the TLC.

As for the production drive, RAID1 is not backup; You're better served by having a second SSD for production data and using volume snapshots at an interval. The failure modes of NAND storage are way less independent compared to mechanical storage so the chances of one NVMe drive failing and it's neighbour in an array surviving are much lower than the same scenario with SAS/SATA mechanical drives.

I would recommend getting a 4TB TLC drive with solid raw-NAND performance and then using VSS to take incremental rolling snapshots at 30 minute intervals to an EXTERNAL device that can be cold-swapped for the failed production drive in the event it dies. With PCIe storage, a RAID1 array in the same physical device protects you from so few of the potential failure modes that's simply not advisable. USB 3.2 20Gbit is plenty of bandwidth to an external drive for volume snapshots and that gives you so many more recovery options and it also gives you a small window to recover from mistakes, something that RAID1 will not.

Not sure how many $400 RAID cards you have used personally, but pretty much every LSI dell or HP card I have ever used requires proper setup before It can be put into production and the interfaces are less than friendly.

QFT.

For a single worstation stick to onboard SATA ports and not softRAID cards which is all you'll get at the sub-$400 mark. I use HPE Nimble all-flash and hybrid flash SANs for production and NEXSAN for tier-2 storage and the single best thing about them is that I never have to deal with another shitty LSI or Adaptec PERC controller ever again.

Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.

Well that's great if he has $20K to spend on an SSD. They're not cheap, even if you buy the lower-endurance variants (Enterprise sales jargon for MLC instead of SLC)

 

[JAB Creations]

Nas is the solution for on/off-line stotrage and ssd's on you main pc for fast access and work.

Are there any respectable NAS that doesn't incure a 10% or greater penalty for...being a NAS? I like my drives in the rig they're being accessed from any way. I currently have seven drives because the need for capacity about a decade ago was being outdone by the available capacity. These days the capacity has been catching up though!

Just buy a 16TB server class SSD and be done with it.

Hence SAS. 8TB is the most one can get on an NVMe and that it something that I'll have to mess around with in a year or two. With SAS I won't get NVMe speeds but I won't be limited to SATA bug speeds either.

But there is a lot of benefits to RAID especially if that 16TB SSD fails.

Only my C:\ won't be running in RAID 1. There sure as hell wouldn't be a 4TB drive in my rig spinning solo so sure as hell wouldn't be a 16TB running solo! I love RAID and drive cloning.

C for OS, D for production data? I think you'd be better served with a higher-performance 1-2TB OS drive, ideally with a large SLC buffer so that you're never hitting raw writes on the TLC.

My work data on D:\ has only been in jepordy when I discovered that AMD decided that their RAID setup from 990FX needed to be changed at some point when I got my X570. Having to manually clone to an intermediary drive was not fun! Especially for the 4TB mech drives, both separate RAID 1s!

As for the production drive, RAID1 is not backup; You're better served by having a second SSD for production data and using volume snapshots at an interval. The failure modes of NAND storage are way less independent compared to mechanical storage so the chances of one NVMe drive failing and it's neighbour in an array surviving are much lower than the same scenario with SAS/SATA mechanical drives.

I know it's not backup, it's redundancy! If a drive is plugged in to any system then it is not a backup. I actually intend to have double-clones at some point to protect against that really low (though still possible) chance of something going wrong during cloning. I'm also moving out west so with dryer air I'm expecting higher chances of static electricity. Everything has been based on the best I can do with my budget at a given point.

Well that's great if he has $20K to spend on an SSD. They're not cheap, even if you buy the lower-endurance variants (Enterprise sales jargon for MLC instead of SLC)

In a month or two I can shell out $3,000 per 16TB drive though it's not the most tempting price obviously. I'd have to wait another month to buy a couple of backup drives though in the long term I wouldn't have to worry about running out of space.

I am tempted by the idea of running an external RAID 1 with 8TB NVMe drives. I'd be more than happy to post some comparative benchmark results for folks. The USB 2 ports on my old AMD laptop max out at 30 megabytes, I get 200+ on my desktop on USB 3.0 so it'd be interesting. The original idea was to use an external NVMe USB converter/cord for cloning purposes. I think it'd be a really interesting read, video, etc as it'll give others insight in to what will eventually become more affordable. I just don't have the time to tinker with my rig like I used to. I can afford to experiment modestly (I'm not going to be rich sooner, just modestly well off) though there are only so much more that one can get and frankly I suspect I wouldn't fill up even an 8TB RAID 1 for a couple of years at least.

My motherboard doesn't support USB 20 gigabits, it does support 10 gigabits. Are the front ports compatible with 20 or is there a physical requirement? If so I'd have to check with the chassis specifications page. Though doing basic math even 20 gigabytes / 8 bits per byte would get 2.5 gigabytes at best so I think I'd stick to interior usage and external cloning.

I've also read that getting a PCI-Express NVMe addon card improves performance over motherboard NVMe slots so there is that to be considered. I won't have a garage filled with parts to tinker with though I've had more than my share of hours wasted by not having this-or-that part spare and on-hand. When I do a report I'm sure someone will appreciate knowing what they'll really need and can actually do without.