$7500~$8000 Server Building Challenge

[xkm1948]

This build is for a fellow researcher who is setting up his new lab. As the title implies, this will be for the main data processing hub for his lab.

What he does:

1. Genomic assembly. Heavy on CPU based computing, requires tons of RAM and storage space.
2. Protein modeling/docking analysis. Heavy on GPU.
3. Data storage. TB level storage for all of his lab data.
4. Virtual Reality for VR protein modeling/molecular interaction analyze.

He has some quotes from his new research institute's computer supplier. However he does not like the particular brand(lenovo).

Let the fun begins.

 

[dorsetknob]

3. Data storage. TB level storage for all of his lab data.

Probably want to separate the storage to a NAS ( with Raid ) from Main PC Your still need Local Disk

 

[bpgt64]

This build is for a fellow researcher who is setting up his new lab. As the title implies, this will be for the main data processing hub for his lab.

What he does:

1. Genomic assembly. Heavy on CPU based computing, requires tons of RAM and storage space.
2. Protein modeling/docking analysis. Heavy on GPU.
3. Data storage. TB level storage for all of his lab data.
4. Virtual Reality for VR protein modeling/molecular interaction analyze.

He has some quotes from his new research institute's computer supplier. However he does not like the particular brand(lenovo).

Let the fun begins.

This sounds more like, do my work for me, than here's what I was thinking, what do you think....

 

[Frick]

u by 4 of these and cluster them over this

 

[xkm1948]

This sounds more like, do my work for me, than here's what I was thinking, what do you think....

I am not very familiar with Server building. If it has to have some starting point, I will try:

Option A:

AMD Napless based 16core 32 threads
Corresponding Motherboard that supports Napless.
512GB Registered DDR4 RAM
AMD MI25 for GPU OpenCL computing
2TB PCI-E SSD for OS/APPs
4x8TB WD Gold for storage
Monitor
HTC Vive
...


Option B:

Intel 2697AV4 16 core 32 threads
Corresponding motherboard
512GB RAM
Nvidia tesla M60 or K80
2TB PCE-E SSD
4x8TB WD Gold for storage
Monitor
HTC VIve
...

 

[Norton]

If it has to have some starting point

What does he run now?

What are the pros/cons of his current hardware? Does he need bleeding edge level tech?

Sotware used, OS, etc..?

 

[silentbogo]

With all the things that's happening now, I'd wait for new Opterons.
You may be able to get something, like in AMD showcase from last month.

https://www.theregister.co.uk/2017/03/08/the_italian_job_amd_goes_to_naples/
http://www.anandtech.com/show/11183...aples-cpus-for-1p-and-2p-servers-coming-in-q2

... and khm-khm...
http://wccftech.com/amd-zen-naples-32-core-cpu-benchmarks-leaked/

AMD MI25 for GPU OpenCL computing

I'm not even sure it's available yet. Announcement was in December, but after that no news, no reviews... Just the same "Q1 2017", as with consumer Vega.
Regarding the second build, you may be better off investing into Tesla P-series instead. On paper it is about ~20% slower than MI25, it is already available and the PCIe version is priced about the same as brand new Maxwell or Kepler counterparts that you chose. BTW, K80 is kinda old, and can probably be outperformed by a pair of RX480s or maybe even a single 1080Ti for less than a fifth of money.
http://www.thinkmate.com/hardware/co-processors

Basically, what I was trying to say, is that your friend has to wait until june-july to see what AMD has to offer in both HPC segments.

The only thing that concerns me, is that P100, M60 and K80 are all in a $5k ballpark (MI25 ain't going to be cheaper)... this only leaves ~$2500-$3000 for the rest of the rig....
Have you considered using consumer-grade hardware for OpenCL workloads? Betcha a new revision of Titan XP (or two) will do just fine, or two buckets and a pocketfull of Polaris cards

 

[xkm1948]

It would be awesome if consumer grade GPU can be used for CUDA/OpenCL based workloads. One TitanXp with full GP100 should be good for a whole lot of things.

He will be waiting for Napless to drop for sure. It is just I have never done any server builds and I don't even know where to buy server CPUs and etc.(Newegg still good for that?)

What does he run now?

What are the pros/cons of his current hardware? Does he need bleeding edge level tech?

Sotware used, OS, etc..?

He is renting out HPC on campus right now. Bad thing is there are so many people using HPC that a lot of his jobs got scheduled two~three weeks down the road. He wanted to have machine that rival the performance of 2~3 nodes on a HPC and all for his disposal whenever he needs to run something.


He will be using Windows10Pro and dual boot with Ubuntu. The largest RAM size Win10Pro can handle is 512GB I believe. Most of the GPU heavy molecular work will be done on Windows10 side, while CPU heavy genome stuff will be done on Linux side.

 

[FordGT90Concept]

I would wait for and go with Naples. You can expect a lot more bang for the buck with Zen-based processors, especially in the multithreading department.

Vega is going to release at about the same time so...

MI25 is a deep learning card so I wouldn't consider that.

 

[silentbogo]

It would be awesome if consumer grade GPU can be used for CUDA/OpenCL based workloads. One TitanXp with full GP100 should be good for a whole lot of things.

He will be waiting for Napless to drop for sure. It is just I have never done any server builds and I don't even know where to buy server CPUs and etc.(Newegg still good for that?)

Newegg - not likely...
Not for cutting-edge stuff. Not since they went full-blown aliexpress on us
Naples may be there upon release, but compute accelerators definitely not. Also, it may be very-very-very expensive, even with Naples on the way...
I'd say, if all he needs is a raw compute power, wait for naples and fill in the blanks with consumer-grade hardware (e.g. regular non-ECC DDR4, GPUs instead of accelerators, non-enterprise SSDs etc.)
8 grand is a lot of money, but with workstations like that it will disappear faster than I can say "whereismykidney?!"

 

[xkm1948]

Newegg - not likely...
Not for cutting-edge stuff. Not since they went full-blown aliexpress on us
Naples may be there upon release, but compute accelerators definitely not. Also, it may be very-very-very expensive, even with Naples on the way...
I'd say, if all he needs is a raw compute power, wait for naples and fill in the blanks with consumer-grade hardware (e.g. regular non-ECC DDR4, GPUs instead of accelerators, non-enterprise SSDs etc.)
8 grand is a lot of money, but with workstations like that it will disappear faster than I can say "whereismykidney?!"

I agree. I was looking at the P100 price, Holy shit that was expensive.

The biggest question is whether it is possible to utilize consumer grade GPU for protein work. From what I learned from Fold@home team, it is possible. While at the same time it also depends on the driver support from GPU developer. I have limited knowledge on GPU accelerated computing though. I definitely need to learn more. Nvidia/AMD sell the same chip for way higher price for a reason I believe.

 

[FordGT90Concept]

You need to look at the system requirements for all the software he's going to be using. It might have bias especially for NVIDIA or AMD.

 

[xkm1948]

You need to look at the system requirements for all the software he's going to be using. It might have bias especially for NVIDIA or AMD.

Most requires OpenCL. Only a few paid applications require CUDA.

I am trying to use my FuryX to do some protein modeling. So far it seems the GPU just won't engage OpenCL at all. I believe it may need some special driver.

 

[FordGT90Concept]

OpenCL is something most cards support but something that explicitly requires CUDA is going to need an NVIDIA card.

Fury X supports OpenCL 2.0. OpenCL 2.1 with 2.2 in the works.


Edit: Pascal may not support OpenCL at all.

Edit: If this is still accurate, Pascal only supports OpenCL 1.2

Looks like NVIDIA is going to prefer CUDA over OpenCL where AMD obviously will prefer OpenCL.

 

[silentbogo]

https://www.microway.com/knowledge-...of-nvidia-geforce-gpus-and-nvidia-tesla-gpus/

Basically more FP64 performance, adjustable Boost, more reliable parts and that's it. Consumer-grade pascal cards also support half-precision math, so beyond slight performance disadvantage you don't really lose anything.

For your FuryX try this:
http://support.amd.com/en-us/kb-articles/Pages/OpenCL2-Driver.aspx

 

[FordGT90Concept]

A list of software he expects to use would be really useful.

 

[Easy Rhino]

Sounds like he should just purchase an appliance and support for the appliance/software.

 

[xkm1948]

Thanks a lot guys! I will inform him on the consumer grade Pascal versus specific Tesla cards. He is not into machine learning so I assume the FP64 and etc. are not really applicable for him.

Titan 3840SP Pascal seems to be a way better options now.

 

[xkm1948]

A list of software he expects to use would be really useful.

I would be asking him for a list.

 

[FordGT90Concept]

FP64 is going to depend entirely on what software he is running. Double precision performance varies wildly between cards.

 

[xkm1948]

Welp after some serious discussion with my friend he will be getting the TitanXp instead of a M60. He is not into artificial intelligence so FP64 is something he would never use.

I offered to help him build the server. Getting to play with some flagship GPU would be nice.
( ͡° ͜ʖ ͡°)

 

[xkm1948]

Alright here is a preliminary list I just put up:

CPU: Intel Xeon E5-2699V4, Broadwell-E 22core 44 threads
Cooling: Noctua D15
MoBo: ASUS X99E-WS/10G
RAM: Samsung 256GB Registered DRAM(32GB Load Reduced DDR4 * 8)
SSD: Intel 600P 1TB
Storage: WD Gold 8TB *2 in RAID1
GPU: Nvidia TitanXp
Monitor: ASUS PA328Q Open for suggestions
PSU: Corsair HX1000i Open for suggestions
Case: Fractal Design Define XL R2 Open for suggestions
Keyboard+Mouse: Good mechanical NON RGB back-lit keyboard. Good mouse
VR: HTC Vive



My reason of going Broadwell-EP. This is a pretty mature system already. It would be hard to tell how long it will take before Napless iron out most of little bugs.

 

[silentbogo]

Alright here is a preliminary list I just put up:

CPU: Intel Xeon E5-2699V4, Broadwell-E 22core 44 threads
Cooling: Noctua D15
MoBo: ASUS X99E-WS/10G
RAM: Samsung 256GB Registered DRAM(32GB Load Reduced DDR4 * 8)
SSD: Intel 600P 1TB
Storage: WD Gold 8TB *2 in RAID1
GPU: Nvidia TitanXp
Monitor: ASUS PA328Q Open for suggestions
PSU: Corsair HX1000i Open for suggestions
Case: Fractal Design Define XL R2 Open for suggestions
Keyboard+Mouse: Good mechanical NON RGB back-lit keyboard. Good mouse
VR: HTC Vive



My reason of going Broadwell-EP. This is a pretty mature system already. It would be hard to tell how long it will take before Napless iron out most of little bugs.

Looks almost like my dream machine...
All I can afford at this time is a micro-cluster of OrangePi's and two pairs of socks

 

[Solaris17]

Dell R810 with 4 E7-8870s

 

[blobster21]

redundant PSUs, you'd hate to lose all thise live data in case of power failure