Server Project

TopHatProductions115 · Aug 14, 2020

Hello there! I've hinted toward this project in various forum posts, but I'll cut to the chase. I'm in the final phase of a long-awaited server project, which began back in mid-2018. However, I never got to actually acquire any of the parts until 2019/2020. With the massive gap in time, I had to change things up a bit, due to market forces and parts availability. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. This is what I've ended up with as of summer 2023:

HPE ProLiant DL580 G7

Code:

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, ISOs) + 
                7x HGST HUSMM1680ASS204 MLC 800GB SSDs (guest datastores)
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable + 
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                                6x HITACHI HUH728080AL4200 8TB HDDs +
                                1x HITACHI HUH72808CLAR8000 8TB HDD +
                                1x IBM Storewise XIV v7000 98Y3241 4TB HDD
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Inateck KU8212 (USB 3.2) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

The planned software configuration has been moved back to the LTT post, and will be changing often for the foreseeable future.

Previous product links and details can be found here.

ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the ~~Titan Z~~ FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though). The reason for why I’m doing this.

If you want more details about the project posted here, please let me know.

There are three other mirrors for this project, all of which get updated when new developments get made:

TopHatProductions115 · Aug 15, 2020

Replaced the Rosewill RASA-11001 with a Kingwin MKS-435TL, due to the fact that it doesn't need molex and will also have a cleaner look to it. Forgot how to edit posts here, sorry if I could have added the info to the OP above. Also planning out a long-term Tesla K80 upgrade...

TopHatProductions115 · Aug 16, 2020

Grabbed the wired mouse. Now back to the waiting game, to see when everything will arrive in the mail...

phill · Aug 17, 2020

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

TopHatProductions115 · Aug 21, 2020

The new drive cage arrived via Amazon. There's only one item left that hasn't arrived in the mail yet - my new mouse...

TopHatProductions115 · Aug 21, 2020

Virtual Flash looks like fun. I might actually look into it more when the time comes...

TopHatProductions115 · Aug 21, 2020

phill said:
Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

phill · Aug 22, 2020

TopHatProductions115 said:
Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

I know that feeling!!! I have a few servers here, boy they are power hungry!!

Looking forward to seeing another picture filled update!!

TopHatProductions115 · Aug 22, 2020

So, here's the current plan:

Have ESXi run 24/7, in a low-power/on-demand state
Have Windows Server 2016 (VPN+AD) setup for 24/7 low-power operation as well
Have Artix Linux (Nextcloud) setup for 24/7 low-power operation too
Have MacOS setup for Hybrid Sleep after 20 minutes' inactivity and setup for Wake-on-LAN
Have Windows 10 setup for ^^^ as well, since it's not running anything mission critical
Don't power down ESXi unless I want to lose access to over half my infrastructure :3

Notes:

https://helgeklein.com/blog/2012/03/dead-simple-wake-on-lan-for-your-windows-server-at-home/

phill · Aug 24, 2020

I look forward to seeing this all up and running

TopHatProductions115 · Aug 27, 2020

So close, yet so far:
https://communities.vmware.com/message/2980320#2980320

phill · Aug 27, 2020

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

TopHatProductions115 · Aug 27, 2020

phill said:
Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Yes. And I'm getting ready to run a memory diagnostic, to see which sticks may be failing.

TopHatProductions115 · Aug 28, 2020

phill said:
Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Crisis averted. From the looks of it, the issue was caused by me running a slightly older version of ESXi 6.5 (U2). VMware's servers have been a bit unreliable for downloads recently, so I had to dig for a copy of ESXi 6.5 U3 on the internet. Now troubleshooting the next issue:

https://communities.vmware.com/message/2980646#2980646

Also have to look into this at some point (even though the server will be mostly hiding behind a VPN):

https://kb.vmware.com/s/article/55636

TopHatProductions115 · Aug 28, 2020

I'm getting ready to reinstall ESXi 6.5 U3, since HPE's Smart Array assumed I was gonna 'RAID all da things' and lose over half my storage to sub-optimally configured RAID arrays that it made (one of which was in perpetual recovery and included an unsupported SSD). Had to manually configure each drive as a logical volume with RAID 0 fault tolerance, treating each drive as its own virtual array of one disk. I ended up RAIDing all the things anyway, by that logic.

TopHatProductions115 · Aug 28, 2020

Now to address the SSD issue in the background, while I look at security patches and initial VM setup.

TopHatProductions115 · Aug 29, 2020

Currently trying to make a new datastore via SSH. More info here:

https://communities.vmware.com/message/2980777#2980777

TopHatProductions115 · Aug 30, 2020

So close to the final stage:

Failed to create VMFS datastore - Cannot change the host configuration.

TopHatProductions115 · Aug 31, 2020

While I'm waiting on comments for the previous issue, and a few ISOs to upload to my server, I can start working on investigating this:

Gotta search for the patches through this page, by entering the details mentioned in the last 3 kb pages:

https://my.vmware.com/group/vmware/patch#search

On a side note, also ran into this when setting up my first VM:

http://www.yellow-bricks.com/2018/0...when-powering-on-vm-on-vsphere/#comment-77449

Reserve Memory beforehand, I guess

TopHatProductions115 · Aug 31, 2020

Here to update the Reddit mirror's link - those expire every 6 months

https://www.reddit.com/user/TopHatProductions115/comments/ijoeg0/project_personal_datacentre/

phill · Aug 31, 2020

Someone has been having fun!! Gotta love trouble shooting don't you??

I hope it all settles out soon for you!!

TopHatProductions115 · Sep 1, 2020

phill said:
Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

phill · Sep 1, 2020

TopHatProductions115 said:
Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes :laugh:

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

TopHatProductions115 · Sep 1, 2020

Oh, I will :3

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

communities.vmware.com

TopHatProductions115 · Sep 6, 2020

phill said:
Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

One step closer

System Name	Serious Series - Serious Server (99.99%)
Processor	4x Intel Xeon E7-8870's
Motherboard	HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling	HP ProLiant OEM cooling fans(s) + heatsinks
Memory	256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s)	AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage	1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s)	Samsung ViewFinity S70A UHD 32" (S32A700)
Case	HP ProLiant DL580 G7 chassis
Audio Device(s)	1x Creative Sound Blaster Audigy Rx
Power Supply	4x HP 441830-001/438203-001's (1200W PSU's)
Mouse	Dell MS819
Keyboard	Logitech K845 (Cherry MX Blue)
VR HMD	N/a
Software	VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores	3DMark won't let me post my scores publicly at this time...

System Name	Serious Series - Serious Server (99.99%)
Processor	4x Intel Xeon E7-8870's
Motherboard	HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling	HP ProLiant OEM cooling fans(s) + heatsinks
Memory	256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s)	AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage	1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s)	Samsung ViewFinity S70A UHD 32" (S32A700)
Case	HP ProLiant DL580 G7 chassis
Audio Device(s)	1x Creative Sound Blaster Audigy Rx
Power Supply	4x HP 441830-001/438203-001's (1200W PSU's)
Mouse	Dell MS819
Keyboard	Logitech K845 (Cherry MX Blue)
VR HMD	N/a
Software	VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores	3DMark won't let me post my scores publicly at this time...

System Name	Serious Series - Serious Server (99.99%)
Processor	4x Intel Xeon E7-8870's
Motherboard	HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling	HP ProLiant OEM cooling fans(s) + heatsinks
Memory	256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s)	AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage	1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s)	Samsung ViewFinity S70A UHD 32" (S32A700)
Case	HP ProLiant DL580 G7 chassis
Audio Device(s)	1x Creative Sound Blaster Audigy Rx
Power Supply	4x HP 441830-001/438203-001's (1200W PSU's)
Mouse	Dell MS819
Keyboard	Logitech K845 (Cherry MX Blue)
VR HMD	N/a
Software	VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores	3DMark won't let me post my scores publicly at this time...

System Name	Not so complete or overkill - There are others!! Just no room to put! :D
Processor	Ryzen Threadripper 3970X
Motherboard	Asus Zenith 2 Extreme Alpha
Cooling	Lots!! Dual GTX 560 rads with D5 pumps for each rad. One rad for each component
Memory	Viper Steel 4 x 16GB DDR4 3600MHz not sure on the timings... Probably still at 2667!! :(
Video Card(s)	Asus Strix 3090 with front and rear active full cover water blocks
Storage	I'm bound to forget something here - 250GB OS, 2 x 1TB NVME, 2 x 1TB SSD, 4TB SSD, 2 x 8TB HD etc...
Display(s)	3 x Dell 27" S2721DGFA @ 7680 x 1440P @ 144Hz or 165Hz - working on it!!
Case	The big Thermaltake that looks like a Case Mods
Audio Device(s)	Onboard
Power Supply	EVGA 1600W T2
Mouse	Corsair thingy
Keyboard	Razer something or other....
VR HMD	No headset yet
Software	Windows 11 OS... Not a fan!!
Benchmark Scores	I've actually never benched it!! Too busy with WCG and FAH and not gaming! :( :( Not OC'd it!! :(

System Name	Serious Series - Serious Server (99.99%)
Processor	4x Intel Xeon E7-8870's
Motherboard	HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling	HP ProLiant OEM cooling fans(s) + heatsinks
Memory	256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s)	AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage	1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s)	Samsung ViewFinity S70A UHD 32" (S32A700)
Case	HP ProLiant DL580 G7 chassis
Audio Device(s)	1x Creative Sound Blaster Audigy Rx
Power Supply	4x HP 441830-001/438203-001's (1200W PSU's)
Mouse	Dell MS819
Keyboard	Logitech K845 (Cherry MX Blue)
VR HMD	N/a
Software	VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores	3DMark won't let me post my scores publicly at this time...

Server Project

TopHatProductions115

TopHatProductions115

TopHatProductions115

phill

Moderator

TopHatProductions115

TopHatProductions115

TopHatProductions115

phill

Moderator

TopHatProductions115

phill

Moderator

TopHatProductions115

phill

Moderator

TopHatProductions115

TopHatProductions115

TopHatProductions115

TopHatProductions115

TopHatProductions115

TopHatProductions115

TopHatProductions115

TopHatProductions115

phill

Moderator

TopHatProductions115

phill

Moderator

TopHatProductions115

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

TopHatProductions115