Server Project

[TopHatProductions115]

Hello there! I've hinted toward this project in various forum posts, but I'll cut to the chase. I'm currently in the final phase(s) of a long-awaited server build project. I began the original planning for it back in 2017/2018, but never got to actually acquire any parts until 2019/2020. Due to the massive gap in time, I had to change things up a bit, due to market forces and parts availability. The server is meant to replace (and extend) my current workstation - a Dell Precision T7500. This is what I've ended up with as of summer 2020:

CSE :: HPE ProLiant DL580 G7
CPU :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
RAM :: 64GB (16x4GB) DDR3-1333 PC3-10600R ECC
STR :: 1x HP 518216-002 146GB HDD (ESXi, VMware Linux Appliance, System ISOs) +

1x Western Digital WD Blue 3D NAND 500GB SATA SSD (vSphere Flash Read Cache) +​

4x HP 507127-B21 300GB HDDs +​

1x 500GB Seagate Video ST500VT003 HDD (Remote Development VM)​

1x LSI SAS 9201-16e HBA SAS card (4-HDD DAS) +​

1x Mini-SAS SFF-8088 to SATA Forward Breakout x4 cable +​

4x HITACHI HUA722020ALA330 HDDs +​

1x Rosewill RASA-11001 (4x 3.5in HDD cage)​

PCIe :: 1x HP 512843-001/591196-001 System I/O board +

1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board​

GPU :: 1x nVIDIA GeForce GTX 1060 6GB +

1x nVIDIA Tesla K10​

SFX :: 1x Creative Sound Blaster Audigy Rx
NIC :: 1x SolarFlare SFN5322F +

1x HP 491838-001 (NC375i)​

FAN :: 4x Arctic F9 PWM 92mm case fans **
PSU :: 4x 1200W Server PSU's (HP 441830-001/438203-001)
PRP :: 1x Dell MS819 Wired Mouse *

Parts marked with * have not been purchased/sourced yet. Those marked with ** are already in-house, but require further planning/modification before they can be added to the server.

As of now, the fans aren't really required for functionality. They were meant to help quiet the server down a bit, but they require some modification to work. This part can wait.


Here is the current software configuration plan for the server:

• VMware Linux Appliance (for managing/monitoring ESXi 6.5)
• Temporary/Test VM (for when I want to host a temporary role/application, without impacting mission-critical infrastructure)
• Windows Server Datacentre (Active Directory, SoftEther+, Technitium^, Server JRE, YaCy, hMailServer+, AOMEI PXE Boot^, etc.)
• MacOS Mojave (Google Drive/MEGA sync*, PleX Server**, Handbrake, SVP4 Pro, iTunes, DaVinci Resolve, F@H, etc.)
• Artix Linux (NextCloud+, OpenSSH, strongSwan+, Technitium^, ejabberd+, FreePBX+SMS+, CUPS^, sNTP^, etc.)
• Windows 10 LTSB (front-end tasks - OBS Studio, Moonlight**, DesignWorks, Console Game Emulators, modpack dev., etc.)
• Remote Development (my build environment - Netbeans, MinGW/64, cmake, GNUWin, CUDA SDK, Mosh, nginx+SQLite+Php, etc.) ^

* Temporary task that will be replaced by a permanent, self-hosted solution
** Can benefit from port forwarding, but will be primarily tunnel-bound
^ Tunnel-bound (VPN/SSH) role - not port forwarded/exposed to the Internet
+ Active Directory enabled - Single Sign On (SSO)


If you want more details about the project posted here, please let me know.


There are four other mirrors for this project, all of which get updated when new developments get made:

• LTT Forum
• L1 Forum
• Reddit post
• VMware Forum

 

[TopHatProductions115]

Replaced the Rosewill RASA-11001 with a Kingwin MKS-435TL, due to the fact that it doesn't need molex and will also have a cleaner look to it. Forgot how to edit posts here, sorry if I could have added the info to the OP above. Also planning out a long-term Tesla K80 upgrade...

 

[TopHatProductions115]

Grabbed the wired mouse. Now back to the waiting game, to see when everything will arrive in the mail...

 

[phill]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

 

[TopHatProductions115]

The new drive cage arrived via Amazon. There's only one item left that hasn't arrived in the mail yet - my new mouse...

 

[TopHatProductions115]

Virtual Flash looks like fun. I might actually look into it more when the time comes...

 

[TopHatProductions115]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

 

[phill]

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

I know that feeling!!! I have a few servers here, boy they are power hungry!! Looking forward to seeing another picture filled update!!

 

[TopHatProductions115]

So, here's the current plan:

• Have ESXi run 24/7, in a low-power/on-demand state
• Have Windows Server 2016 (VPN+AD) setup for 24/7 low-power operation as well
• Have Artix Linux (Nextcloud) setup for 24/7 low-power operation too
• Have MacOS setup for Hybrid Sleep after 20 minutes' inactivity and setup for Wake-on-LAN
• Have Windows 10 setup for ^^^ as well, since it's not running anything mission critical
• Don't power down ESXi unless I want to lose access to over half my infrastructure :3

Notes:

• https://helgeklein.com/blog/2012/03/dead-simple-wake-on-lan-for-your-windows-server-at-home/

 

[phill]

I look forward to seeing this all up and running

 

[TopHatProductions115]

So close, yet so far:
https://communities.vmware.com/message/2980320#2980320

 

[phill]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Yes. And I'm getting ready to run a memory diagnostic, to see which sticks may be failing.

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Crisis averted. From the looks of it, the issue was caused by me running a slightly older version of ESXi 6.5 (U2). VMware's servers have been a bit unreliable for downloads recently, so I had to dig for a copy of ESXi 6.5 U3 on the internet. Now troubleshooting the next issue:

• https://communities.vmware.com/message/2980646#2980646

Also have to look into this at some point (even though the server will be mostly hiding behind a VPN):

• https://kb.vmware.com/s/article/55636

 

[TopHatProductions115]

I'm getting ready to reinstall ESXi 6.5 U3, since HPE's Smart Array assumed I was gonna 'RAID all da things' and lose over half my storage to sub-optimally configured RAID arrays that it made (one of which was in perpetual recovery and included an unsupported SSD). Had to manually configure each drive as a logical volume with RAID 0 fault tolerance, treating each drive as its own virtual array of one disk. I ended up RAIDing all the things anyway, by that logic.

 

[TopHatProductions115]

Now to address the SSD issue in the background, while I look at security patches and initial VM setup.

 

[TopHatProductions115]

Currently trying to make a new datastore via SSH. More info here:

• https://communities.vmware.com/message/2980777#2980777

 

[TopHatProductions115]

So close to the final stage:

• Failed to create VMFS datastore - Cannot change the host configuration.

 

[TopHatProductions115]

While I'm waiting on comments for the previous issue, and a few ISOs to upload to my server, I can start working on investigating this:

• https://kb.vmware.com/s/article/55636
• https://kb.vmware.com/s/article/55806
• https://www.vmware.com/security/advisories/VMSA-2018-0020.html
  • https://kb.vmware.com/s/article/56547
  • https://kb.vmware.com/s/article/56563
  • https://kb.vmware.com/s/article/56896

Gotta search for the patches through this page, by entering the details mentioned in the last 3 kb pages:

• https://my.vmware.com/group/vmware/patch#search

On a side note, also ran into this when setting up my first VM:

• http://www.yellow-bricks.com/2018/06/12/module-monitorloop-power-on-error-failed-when-powering-on-vm-on-vsphere/#comment-77449

Reserve Memory beforehand, I guess

 

[TopHatProductions115]

Here to update the Reddit mirror's link - those expire every 6 months

• https://www.reddit.com/user/TopHatProductions115/comments/ijoeg0/project_personal_datacentre/

 

[phill]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

 

[TopHatProductions115]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

 

[phill]

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

 

[TopHatProductions115]

Oh, I will :3

Failed to create VMFS datastore - Cannot change... |VMware Communities

Coming hot off the heels of the last two issues, I now have only one more to address before I start applying security patches (last step before moving

communities.vmware.com

 

[TopHatProductions115]

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

One step closer