• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Huge SSD NVEm + SAS Setup Plans

Joined
May 13, 2015
Messages
285 (0.12/day)
Processor AMD Ryzen 3800X / AMD 8350
Motherboard ASRock X570 Phantom Gaming X / Gigabyte 990FXA-UD5 Revision 3.0
Cooling Stock / Corsair H100
Memory 32GB / 24GB
Video Card(s) AMD Radeon 290X (Toggling until 6950XT)
Storage C:\ 1TB SSD, D:\ RAID-1 1TB SSD, 2x4TB-RAID-1
Display(s) Samsung U32E850R
Case be quiet! Dark Base Pro 900 Black rev. 2 / Fractal Design
Power Supply EVGA Supernova 1300G2 / EVGA Supernova 850G+
Mouse Logitech M-U0007
Keyboard Logitech G110 / Logitech G110
My future is looking bright financially and I'll be upgrading a lot of my system in the coming months. Before I get in to specifics this is what I use my rig for (my rig's specs are under my profile on the left):
  • Web Development primarily, about ~60% of the time at least.
  • Reading, research, ~10%.
  • Entertainment besides gaming, basically eating my meals and enjoying something, about ~10% of my time.
  • Gaming, about ~5% of my time, I'll go weeks without gaming sometimes but money pays the bills.
  • I don't encode audio/video often, I do though it's negligible.
Storage is the topic here and here are my goals:
  • C:\ can be a single NVMe SSD, likely a 4TB drive.
  • D:\ will be two NVMe SSDs in RAID 1, likely a 4TB drive.
  • E:\ will be a mass storage of at least 8TB though I'd prefer 16TB.
  • The C:\ and D:\ drives will be the same so I can interchange them if necessary.
  • I will have two cold spares for the C:\ and D:\.
  • I will image one of the the C:\ and D:\ every couple of months for cold storage.
  • I will have two cold spares for the E:\.
  • I will image one of the E:\ spares every couple of months for cold storage.
Whatever drives I work with I plan on making clear how the speed, RAID and other details are clearly understood so others can benefit. As someone who solves problems one of the things I loathe the most is seeing "worked, thanks!" replies with zero details that negates any value to anyone else reading someone else's threads. In a few months once I have this going if I haven't updated this first post someone please politely remind me.

Because higher capacity SSDs are generally available in the enterprise segment I imagine that Threadripper motherboards don't have SAS. When the 4950X comes out that is what I'll upgrade to, I don't need Threadripper as I don't run much in the way of audio/video encoding (I do, but not with all the money in the world to justify it). I just came across a 16 port SAS controller for $471 on Newegg. Here is what I'd like performance wise:
  • C:\, D:\ will be NVMe without a doubt and that controller will definitely be running in RAID mode. I'd like to get the transfer rates up and I favor Samsung as I've had trouble in the past with Crucial setting a couple SSD drives in RAID 1, no problems with Samsung and I figure that they'll have TLC 4TB NVMe drives out in a couple months tops by now.
  • E:\ will likely be an addon PCI-Express card and I only need two drives. SAS bus allows a faster transfer rate, I think ~1 gigabyte a second? I'm not sure as it'll be my first experience with SAS. I'm not sure how much difference one SAS controller is from another though I'm always willing to spend more for quality. I can live with a ~5X improvement over my current mechanical RAID 1 setup.
I'm mostly interested in reasonably high (but not blind / RAID 0 non-sense) performance blended with reasonably high capacity. I always keep games, applications and the operating system separate from my work files and anything mass storage like video editing, ISOs, game installers from old CDROMs/DVDs, etc on the mass storage drives.

I'm looking forward to Samsung's TLC 4TB NVMe drives coming out soon for RAID. I'm not sure about the SAS cards though there are 16TB SAS enterprise SSD drives out.

For the past few years when (not if) Windows takes a dump I just plug in a spare SSD like a floppy disc, recover what I need, clone over and I'm back up and running sans a few updates in less than an hour. My main goal is to generally avoid having to mess with my main rig once the new parts become available because as I much as I enjoy tinkering I enjoy making money a hundred times more.

This will be going on to my 3800X / X570 rig and I expect to be doing this sometime roughly around mid-October. Suggestions please?
 

Solaris17

Dainty Moderator
Staff member
Joined
Aug 16, 2005
Messages
22,453 (3.82/day)
Location
Florida
System Name Venslar
Processor I9 7980XE
Motherboard MSI x299 Tomahawk Arctic
Cooling EK Custom
Memory 32GB Corsair DDR4 3000mhz
Video Card(s) Nvidia Titan RTX
Storage 1x 250GB 960 EVO | 1x 500gb Intel 720p | 32TB SAN
Display(s) 3x AOC Q2577PWQ (2k IPS)
Case Inwin 303 White (Thermaltake Ring 120mm Purple accent)
Audio Device(s) Schiit Fulla 3 on Beyerdynamic DT 990 Pros
Power Supply Seasonic 1050W Snow
Mouse M55 RGB PRO (White)
Keyboard Ducky Shine 6 Snow White
VR HMD HTC VIVE
Software Windows 10 x64 Pro
Why are you going with SAS? Do you have a specific need? SAS is generally aging and while it has some clear advantages over SATA your use cases are going to be moving big data work loads to SAN appliances not some random PC with a bunch of drives in it. I'm sorry but photoshop and 4k video workloads aren't going to be enough justification.

Most people I have seen in the DC space if not going for an appliance like nimble usually have supermicro or dell chassis or even QNAPS and higher end synologys with RAID 10 for performance and redundancy and depending on burst workload an SSD cache tier.

You might even be better off using an NVMe card as long as your mobo can use pci-e bi-purification.

To be certain, I sit around PB of data several times a week and while expensive drives and RAID controllers are the norm, I think I can count in only double digits the amount of SAS drives and appliances in use. Maybe its a monetary flex?

In either case, depending on need, you might want to explore other options, USB-C or 10G NAS with the ability to RAID 10 with storage tiering? Maybe atleast then the ability for hotspares and auto rebuilding may be apparent and behind an easy button.

Not sure how many $400 RAID cards you have used personally, but pretty much every LSI dell or HP card I have ever used requires proper setup before It can be put into production and the interfaces are less than friendly.

Food for thought maybe.
 
Joined
May 13, 2015
Messages
285 (0.12/day)
Processor AMD Ryzen 3800X / AMD 8350
Motherboard ASRock X570 Phantom Gaming X / Gigabyte 990FXA-UD5 Revision 3.0
Cooling Stock / Corsair H100
Memory 32GB / 24GB
Video Card(s) AMD Radeon 290X (Toggling until 6950XT)
Storage C:\ 1TB SSD, D:\ RAID-1 1TB SSD, 2x4TB-RAID-1
Display(s) Samsung U32E850R
Case be quiet! Dark Base Pro 900 Black rev. 2 / Fractal Design
Power Supply EVGA Supernova 1300G2 / EVGA Supernova 850G+
Mouse Logitech M-U0007
Keyboard Logitech G110 / Logitech G110
Great insight, hence why I posted. I'm interested in SAS because going beyond 4TB (this year) will mean going beyond TLC. I'd prefer to simply have a couple of 16TB NVMe drives for my D:\ though the capacities simply aren't there yet and therefore I will continue to segregate my data. I could opt to get an additional PCI-Express NVMe addon card and use that along with a couple of 8TB NVMe drives (Sabrent Rocket being the only option currently that I'm aware of) though I'd be at roughly 5TB of used space once the dust is settled, maybe a little less if I migrate some of that data to the D:\.

Again, I'm not working with gobs of data, I just want to make a lasting investment in storage so I can concentrate on my software business instead. I still have plenty of time to determine what I want to do so for me this is a great time to get input from others.

I'm not familiar with "pci-e bi-purification", could you please clarify what that is?
 
Joined
Aug 27, 2011
Messages
296 (0.08/day)
System Name Gaming PC/ EDU PC/ HFS PC
Processor Intel i9-9900KF/ Dual Ryzen 7 2700X
Motherboard Asrock Z390 Taichi Ultimate/ Dual Asrock X370 Proffesional Gaming
Cooling Noctua NH-C14S/ Arctic Xtreme Freezer/ Ryzen Wraith Prysm RGB
Memory 64GB Corsair Vengeance PRO RGB 3200/ 32GB Corsair Dominator 3000/ 16GB Corsair Vengeance Pro 3200
Video Card(s) EVGA RTX 3080 FTW3 Ultra / MSI RTX 2080Ti Ventus / EVGA GTX 1060 SC Gaming
Storage Dual 970 EVO Plus 1TB + 6Tb 860 EVO/ 960 EVO 500GB + 18Tb R0/ 840EVO 250Gb + 16Tb R0
Display(s) Samsung 32" U32R590 Curved 3480x2160 / Samsung 32" LC32H711 Curved 2560x1440 Freesync
Case Cooler Master Stacker 830 NV Edition/ Dual Cooler Master 690 Advance II
Audio Device(s) Creative X-Fi Surround 5.1 SBX/ Creative X-Fi Titanium Pci-E/ On-board Realtek
Power Supply Triple Corsair Platinum HX850i
Mouse Logitech G7 WL / Logitech G903 Lightspeed / MS BT 8000
Keyboard Dual Logitech G19s
Software Win10 Pro
interesting choice of a vast ssd capacity and amount, though consumer drives are not so durable as server class ones. Although a spinners remains the leader in capacity and price which you can consider for storage the essential info and all other left on ssd's. Nas is the solution for on/off-line stotrage and ssd's on you main pc for fast access and work.
 
Joined
Dec 29, 2017
Messages
1,002 (0.74/day)
System Name Silent
Processor R9 5950x @5.2 ST PBO2/ 4.7 MT Curve Optimiser
Motherboard Asus Strix X570i Gaming
Cooling Optimus AMD Raw Copper/Plexi, EK Copper 240/40mm, Utilitube D5, 4x NF-A12x25, Mayhems XTR Nano White
Memory 16GB G-Skill Trident Z Neo 4000 14-16-14-28 w/2x NF-A4x10 fan
Video Card(s) Rtx 3070 Founders
Storage Optane 800p 118gb, Samsung 980 Pro 1tb
Display(s) 32" 240hz 1440p Samsung G7 , 32" 1440p 60hz
Case Sliger SM570 Custom 13-Litre
Audio Device(s) Schiit Hel, Vmoda Crossfade M100 Master, Razer Nommo Pro
Power Supply Corsair SF750 Platinum
Mouse Razer Viper Ultimate
Keyboard Razer Huntsman Elite w/Mercury PBT keycaps
Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.
 
Joined
Sep 28, 2005
Messages
2,304 (0.39/day)
Location
Calgary Alberta, Canada
System Name PCGR
Processor Intel 10500 ES
Motherboard MSI Z490 Gaming Plus
Cooling Snowman MT4 Dual Fan
Memory 2x8GB DDR4 3600 Corsair Vengeance
Video Card(s) Gigabyte RX 6800XT Gaming OC
Storage 1x 1tb ADATA XPG 8200 Pro NVME | 1x 1TB WD Blue HDD
Display(s) BenQ 24" 1080P
Case MSI MPG 110M
Audio Device(s) Onboard
Power Supply Cooler Master 550W MWE 80+ Gold
Software Windows 10 64bit Pro
Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.

But there is a lot of benefits to RAID especially if that 16TB SSD fails.
 
Joined
Dec 29, 2017
Messages
1,002 (0.74/day)
System Name Silent
Processor R9 5950x @5.2 ST PBO2/ 4.7 MT Curve Optimiser
Motherboard Asus Strix X570i Gaming
Cooling Optimus AMD Raw Copper/Plexi, EK Copper 240/40mm, Utilitube D5, 4x NF-A12x25, Mayhems XTR Nano White
Memory 16GB G-Skill Trident Z Neo 4000 14-16-14-28 w/2x NF-A4x10 fan
Video Card(s) Rtx 3070 Founders
Storage Optane 800p 118gb, Samsung 980 Pro 1tb
Display(s) 32" 240hz 1440p Samsung G7 , 32" 1440p 60hz
Case Sliger SM570 Custom 13-Litre
Audio Device(s) Schiit Hel, Vmoda Crossfade M100 Master, Razer Nommo Pro
Power Supply Corsair SF750 Platinum
Mouse Razer Viper Ultimate
Keyboard Razer Huntsman Elite w/Mercury PBT keycaps
The likelihood of that happening on server grade ssds is incredibly low.
 
Joined
Feb 20, 2019
Messages
3,067 (3.25/day)
System Name Bragging Rights
Processor Atom Z3735F 1.33GHz
Motherboard It has no markings but it's green
Cooling No, it's a 2.2W processor
Memory 2GB DDR3L-1333
Video Card(s) Gen7 Intel HD (4EU @ 311MHz)
Storage 32GB eMMC and 128GB Sandisk Extreme U3
Display(s) 10" IPS 1280x800 60Hz
Case Veddha T2
Audio Device(s) Apparently, yes
Power Supply Samsung 18W 5V fast-charger
Mouse MX Anywhere 2
Keyboard Logitech MX Keys (not Cherry MX at all)
VR HMD Samsung Oddyssey, not that I'd plug it into this though....
Software W10 21H1, barely
Benchmark Scores I once clocked a Celeron-300A to 564MHz on an Abit BE6 and it scored over 9000.
C for OS, D for production data? I think you'd be better served with a higher-performance 1-2TB OS drive, ideally with a large SLC buffer so that you're never hitting raw writes on the TLC.

As for the production drive, RAID1 is not backup; You're better served by having a second SSD for production data and using volume snapshots at an interval. The failure modes of NAND storage are way less independent compared to mechanical storage so the chances of one NVMe drive failing and it's neighbour in an array surviving are much lower than the same scenario with SAS/SATA mechanical drives.

I would recommend getting a 4TB TLC drive with solid raw-NAND performance and then using VSS to take incremental rolling snapshots at 30 minute intervals to an EXTERNAL device that can be cold-swapped for the failed production drive in the event it dies. With PCIe storage, a RAID1 array in the same physical device protects you from so few of the potential failure modes that's simply not advisable. USB 3.2 20Gbit is plenty of bandwidth to an external drive for volume snapshots and that gives you so many more recovery options and it also gives you a small window to recover from mistakes, something that RAID1 will not.

Not sure how many $400 RAID cards you have used personally, but pretty much every LSI dell or HP card I have ever used requires proper setup before It can be put into production and the interfaces are less than friendly.
QFT.

For a single worstation stick to onboard SATA ports and not softRAID cards which is all you'll get at the sub-$400 mark. I use HPE Nimble all-flash and hybrid flash SANs for production and NEXSAN for tier-2 storage and the single best thing about them is that I never have to deal with another shitty LSI or Adaptec PERC controller ever again.

Just buy a 16TB server class SSD and be done with it.

No need to fuck around with a separate drive enclosure.
Well that's great if he has $20K to spend on an SSD. They're not cheap, even if you buy the lower-endurance variants (Enterprise sales jargon for MLC instead of SLC)
 
Joined
May 13, 2015
Messages
285 (0.12/day)
Processor AMD Ryzen 3800X / AMD 8350
Motherboard ASRock X570 Phantom Gaming X / Gigabyte 990FXA-UD5 Revision 3.0
Cooling Stock / Corsair H100
Memory 32GB / 24GB
Video Card(s) AMD Radeon 290X (Toggling until 6950XT)
Storage C:\ 1TB SSD, D:\ RAID-1 1TB SSD, 2x4TB-RAID-1
Display(s) Samsung U32E850R
Case be quiet! Dark Base Pro 900 Black rev. 2 / Fractal Design
Power Supply EVGA Supernova 1300G2 / EVGA Supernova 850G+
Mouse Logitech M-U0007
Keyboard Logitech G110 / Logitech G110
Nas is the solution for on/off-line stotrage and ssd's on you main pc for fast access and work.

Are there any respectable NAS that doesn't incure a 10% or greater penalty for...being a NAS? I like my drives in the rig they're being accessed from any way. I currently have seven drives because the need for capacity about a decade ago was being outdone by the available capacity. These days the capacity has been catching up though!

Just buy a 16TB server class SSD and be done with it.

Hence SAS. 8TB is the most one can get on an NVMe and that it something that I'll have to mess around with in a year or two. With SAS I won't get NVMe speeds but I won't be limited to SATA bug speeds either.

But there is a lot of benefits to RAID especially if that 16TB SSD fails.

Only my C:\ won't be running in RAID 1. There sure as hell wouldn't be a 4TB drive in my rig spinning solo so sure as hell wouldn't be a 16TB running solo! I love RAID and drive cloning.

C for OS, D for production data? I think you'd be better served with a higher-performance 1-2TB OS drive, ideally with a large SLC buffer so that you're never hitting raw writes on the TLC.

My work data on D:\ has only been in jepordy when I discovered that AMD decided that their RAID setup from 990FX needed to be changed at some point when I got my X570. Having to manually clone to an intermediary drive was not fun! Especially for the 4TB mech drives, both separate RAID 1s!

As for the production drive, RAID1 is not backup; You're better served by having a second SSD for production data and using volume snapshots at an interval. The failure modes of NAND storage are way less independent compared to mechanical storage so the chances of one NVMe drive failing and it's neighbour in an array surviving are much lower than the same scenario with SAS/SATA mechanical drives.

I know it's not backup, it's redundancy! If a drive is plugged in to any system then it is not a backup. I actually intend to have double-clones at some point to protect against that really low (though still possible) chance of something going wrong during cloning. I'm also moving out west so with dryer air I'm expecting higher chances of static electricity. Everything has been based on the best I can do with my budget at a given point.

Well that's great if he has $20K to spend on an SSD. They're not cheap, even if you buy the lower-endurance variants (Enterprise sales jargon for MLC instead of SLC)

In a month or two I can shell out $3,000 per 16TB drive though it's not the most tempting price obviously. I'd have to wait another month to buy a couple of backup drives though in the long term I wouldn't have to worry about running out of space.

I am tempted by the idea of running an external RAID 1 with 8TB NVMe drives. I'd be more than happy to post some comparative benchmark results for folks. The USB 2 ports on my old AMD laptop max out at 30 megabytes, I get 200+ on my desktop on USB 3.0 so it'd be interesting. The original idea was to use an external NVMe USB converter/cord for cloning purposes. I think it'd be a really interesting read, video, etc as it'll give others insight in to what will eventually become more affordable. I just don't have the time to tinker with my rig like I used to. I can afford to experiment modestly (I'm not going to be rich sooner, just modestly well off) though there are only so much more that one can get and frankly I suspect I wouldn't fill up even an 8TB RAID 1 for a couple of years at least.

My motherboard doesn't support USB 20 gigabits, it does support 10 gigabits. Are the front ports compatible with 20 or is there a physical requirement? If so I'd have to check with the chassis specifications page. Though doing basic math even 20 gigabytes / 8 bits per byte would get 2.5 gigabytes at best so I think I'd stick to interior usage and external cloning.

I've also read that getting a PCI-Express NVMe addon card improves performance over motherboard NVMe slots so there is that to be considered. I won't have a garage filled with parts to tinker with though I've had more than my share of hours wasted by not having this-or-that part spare and on-hand. When I do a report I'm sure someone will appreciate knowing what they'll really need and can actually do without.
 
Top