Server Project

[TopHatProductions115]

Hello there! I've hinted toward this project in various forum posts, but I'll cut to the chase. I'm in the final phase of a long-awaited server project, which began back in 2017/2018. However, I never got to actually acquire any of the parts until 2019/2020. With the massive gap in time, I had to change things up a bit, due to market forces and parts availability. The server is meant to replace (and extend) my current workstation - a Dell Precision T7500. This is what I've ended up with as of the 2020/2021 transition:


CSE :: HPE ProLiant DL580 G7
CPU :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
RAM :: 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
PCIe :: 1x HP 512843-001/591196-001 System I/O board +

1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board​

GPU :: 1x nVIDIA GeForce GTX Titan Xp +

1x nVIDIA GeForce GTX Titan Z +​

SFX :: 1x Creative Sound Blaster Audigy Rx
NIC :: 1x HPE NC524SFP (489892-B21)
I/O :: 1x HPE PCIe ioDuo MLC I/O Accelerator (641255-001)

1x SanDisk Fusion ioScale MLC 3.2TB Flash Accelerator (F11-002-3T20-CS-0001)​

STR :: 1x HP Smart Array P410i Controller (integrated) +

1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) +​

4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +​

1x Western Digital WD Blue 3D NAND 500GB SSD (Virtual Flash) +​

1x Intel 320 Series SSDSA2CW600G3 600GB SSD (VFF)​

1x Seagate Video ST500VT003 500GB HDD​

1x LSI SAS 9201-16e HBA SAS card +​

1x Mini-SAS SFF-8088 cable +​

1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) +​

4x HITACHI (HGST) Ultrastar HUH728080AL4205 8TB HDDs​

4x IBM Storewise XIV v7000 98Y3241 4TB HDDs​

1x Sony Optiarc BluRay drive​

PRP :: 1x AOC U2879VF (4K)

1x Inateck KU5211 (USB 3.2) +​

1x Logitech K845 (Cherry MX Blue) +​

1x Dell MS819 Wired Mouse​

PSU :: 4x HP 1200W Server PSU's (441830-001/438203-001)


Here is the current software configuration plan for the server:

• vCenter Server Appliance (v6.7 - for managing and monitoring ESXi 6.5)
• Windows Server Datacentre (Active Directory^, AD CS, Azure AD Connect, MariaDB^, Meinberg NTP^, hMailServer+, SoftEther+, etc.)
• Artix Linux (Samba, Docker + Compose, Portainer, NGINX Proxy Manager, NextCloud^+, OnlyOffice EE, xBrowserSync, RustDesk, YaCy Grid, etc.)
• macOS Mojave (PleX Media Server**, iTunes Sync, Handbrake, Deluge (seeding), Time Machine server, Remotix, SVP4 Pro, etc.)
• Windows 10 LTSC (Moonlight Gamestream**, DaVinci Resolve, Console Game Emulators, modpack dev., etc.)
  • Remote Dev. (merged with above - Netbeans, MinGW/64, cmake, GNUWin, Mosh, nginx+SQLite/MariaDB+Php, etc.) ^
• Temporary/Test VM (for when I want to host a temporary role/application, without impacting mission-critical infrastructure)

* Temporary task that will be replaced by a permanent, self-hosted solution
** Can benefit from port forwarding, but will be primarily tunnel-bound
^ Tunnel-bound (VPN/SSH) role - not port forwarded/exposed to the Internet
+ Active Directory enabled - Single Sign On (SSO)


If you want more details about the project posted here, please let me know.


There are four other mirrors for this project, all of which get updated when new developments get made:

• LTT Forum
• L1 Forum
• VMware Forum

 

[TopHatProductions115]

Replaced the Rosewill RASA-11001 with a Kingwin MKS-435TL, due to the fact that it doesn't need molex and will also have a cleaner look to it. Forgot how to edit posts here, sorry if I could have added the info to the OP above. Also planning out a long-term Tesla K80 upgrade...

 

[TopHatProductions115]

Grabbed the wired mouse. Now back to the waiting game, to see when everything will arrive in the mail...

 

[phill]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

 

[TopHatProductions115]

The new drive cage arrived via Amazon. There's only one item left that hasn't arrived in the mail yet - my new mouse...

 

[TopHatProductions115]

Virtual Flash looks like fun. I might actually look into it more when the time comes...

 

[TopHatProductions115]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

 

[phill]

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

I know that feeling!!! I have a few servers here, boy they are power hungry!! Looking forward to seeing another picture filled update!!

 

[TopHatProductions115]

So, here's the current plan:

• Have ESXi run 24/7, in a low-power/on-demand state
• Have Windows Server 2016 (VPN+AD) setup for 24/7 low-power operation as well
• Have Artix Linux (Nextcloud) setup for 24/7 low-power operation too
• Have MacOS setup for Hybrid Sleep after 20 minutes' inactivity and setup for Wake-on-LAN
• Have Windows 10 setup for ^^^ as well, since it's not running anything mission critical
• Don't power down ESXi unless I want to lose access to over half my infrastructure :3

Notes:

• https://helgeklein.com/blog/2012/03/dead-simple-wake-on-lan-for-your-windows-server-at-home/

 

[phill]

I look forward to seeing this all up and running

 

[TopHatProductions115]

So close, yet so far:
https://communities.vmware.com/message/2980320#2980320

 

[phill]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Yes. And I'm getting ready to run a memory diagnostic, to see which sticks may be failing.

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Crisis averted. From the looks of it, the issue was caused by me running a slightly older version of ESXi 6.5 (U2). VMware's servers have been a bit unreliable for downloads recently, so I had to dig for a copy of ESXi 6.5 U3 on the internet. Now troubleshooting the next issue:

• https://communities.vmware.com/message/2980646#2980646

Also have to look into this at some point (even though the server will be mostly hiding behind a VPN):

• https://kb.vmware.com/s/article/55636

 

[TopHatProductions115]

I'm getting ready to reinstall ESXi 6.5 U3, since HPE's Smart Array assumed I was gonna 'RAID all da things' and lose over half my storage to sub-optimally configured RAID arrays that it made (one of which was in perpetual recovery and included an unsupported SSD). Had to manually configure each drive as a logical volume with RAID 0 fault tolerance, treating each drive as its own virtual array of one disk. I ended up RAIDing all the things anyway, by that logic.

 

[TopHatProductions115]

Now to address the SSD issue in the background, while I look at security patches and initial VM setup.

 

[TopHatProductions115]

Currently trying to make a new datastore via SSH. More info here:

• https://communities.vmware.com/message/2980777#2980777

 

[TopHatProductions115]

So close to the final stage:

• Failed to create VMFS datastore - Cannot change the host configuration.

 

[TopHatProductions115]

While I'm waiting on comments for the previous issue, and a few ISOs to upload to my server, I can start working on investigating this:

• https://kb.vmware.com/s/article/55636
• https://kb.vmware.com/s/article/55806
• https://www.vmware.com/security/advisories/VMSA-2018-0020.html
  • https://kb.vmware.com/s/article/56547
  • https://kb.vmware.com/s/article/56563
  • https://kb.vmware.com/s/article/56896

Gotta search for the patches through this page, by entering the details mentioned in the last 3 kb pages:

• https://my.vmware.com/group/vmware/patch#search

On a side note, also ran into this when setting up my first VM:

• http://www.yellow-bricks.com/2018/0...when-powering-on-vm-on-vsphere/#comment-77449

Reserve Memory beforehand, I guess

 

[TopHatProductions115]

Here to update the Reddit mirror's link - those expire every 6 months

• https://www.reddit.com/user/TopHatProductions115/comments/ijoeg0/project_personal_datacentre/

 

[phill]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

 

[TopHatProductions115]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

 

[phill]

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

 

[TopHatProductions115]

Oh, I will :3

Re: Failed to create VMFS datastore - Cannot change the host configuration.

Hello TopHat, I was asking that because is not available on the HTML5 Client. For configuring VFFS you need to login to vCenter using the Flash client (vSphere Web Client). This is the URL you have to access and you can use same credentials: https://vcenter_server_fqdn/vsphere-client

communities.vmware.com

 

[TopHatProductions115]

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

One step closer