2012R2 first oh shit moment

[Solaris17]

Few months ago I rebuilt a server into a much more powerful machine; during an infrastructure restructuring. I also took some primary programs that were run on the HOST OS and split them into VMs to segregate them. (This was a completely new system so I did abre metal reinstalls of all server OSs both on the host and inside each VM)

Well today was my first oh shit moment. I found that one of the data bases for the software got corrupted somehow. I did not have a readily available backup. However I DO bare metal backups of the entire system via windows Server backup every night at 3am to a seperate drive.

I was blown away at how easy it was to recover. Because Im not seasoned with recoverys (thankfully?) I shut down the offending VM. I then proceeded to restore the entire VM (not the whole bare metal just that VM in the bare metal) after it had finished I closed the prompt. went back to Hyper-v manager and booted the VM back up. It came online instantly and everything was fine.

I was so impressed I just looked at it for a few minutes.

The good news is I wont by replicating the problem. turns out some issues in the software caused bad data to be written. the root problem has been fixed I just thought I'd share my story since I was literally blown away by how simple it was and how robust the backup was.


I did pat myself on the back a little for setting it up in this way though, I can only imagine it would have been far more disastrous if all the software we use was running on the surface of one machine.

Anyone else have stories to tell about recovery situations? We all know what usually happens during data failures. Does windows Server backup genuinely deserve its rep? Did I just get lucky?

 

[BiggieShady]

Not a Windows Server Backup experience story, but it is VM related.
My experience with VMs running Windows Server is on Amazon Cloud. They do a complete daily snapshot of a system disk image, so restoring a VM to an earlier state is really trivial.
Deployment was the same even in tricky situations: we had an auto scaling group of VMs behind a load balancer for horizontal scaling, in production, with substantial load on several machines in the auto scaling group. We would prepare next version for the deployment on a separate VM, saved system disk image of that VM and reconfigured auto scaling group to boot new instances with a new snapshot image. We would then manually start an additional new version instance in the group, then proceed slowly to kill off old version instances in the group while letting the auto scaling group to repopulate itself with new version instances.
I remember being pleasantly surprised how stuff "just worked".

 

[qubit]

Ah, the value of backups, couldn't agree with you more. Well done Solaris!

 

[Solaris17]

Not a Windows Server Backup experience story, but it is VM related.
My experience with VMs running Windows Server is on Amazon Cloud. They do a complete daily snapshot of a system disk image, so restoring a VM to an earlier state is really trivial.
Deployment was the same even in tricky situations: we had an auto scaling group of VMs behind a load balancer for horizontal scaling, in production, with substantial load on several machines in the auto scaling group. We would prepare next version for the deployment on a separate VM, saved system disk image of that VM and reconfigured auto scaling group to boot new instances with a new snapshot image. We would then manually start an additional new version instance in the group, then proceed slowly to kill off old version instances in the group while letting the auto scaling group to repopulate itself with new version instances.
I remember being pleasantly surprised how stuff "just worked".

Thats sounds fun, I have yet to play with load balancing Im emulating other things on my lab ATM but iv always found it pretty fascinating.

Ah, the value of backups, couldn't agree with you more. Well done Solaris!

Thanks! I actually had a few other sysadmins years back tell me it was a bit much to do bare metals every day. Which I can honestly maybe agree with? but The drives I use are commissioned specifically for the purpose of backups so I mean fuck it.

 

[xvi]

Our trouble ticket/account management system is basically just a SQL server. Server has, I can't remember if it's 6 or 8 drives. I put it in RAID 5 because almost every other server we have is RAID 0 (with the occasional 1 in the mix) and RAID 5 is an improvement for us (others were built before me).

Well, drive 3 had an electrical failure and not only that, but it starts screwing with drive 2 above it. Drive 2 then starts writing corrupt data to itself (didn't quite realize this yet) and becomes unrecognizable. Pull drive 3 and drive 2 becomes recognizable again, but the OS fails to boot. Two drive failures in a RAID 5 array means data loss, of course, so I move from trying to recover it and just get down to the post-mortem.

Now, the only computer I know of with enough SATA ports to hook up all of the non-failed server hard drives plus a drive to boot off of is my main computer at home (a little mATX build I'm pretty fond of). Found some raid recovery program that worked really well, despite the half corrupt drive 2, and managed to pull the semi-corrupt SQL database off of it.
We dug up a month or so old backup of the database and managed to overlay the bits of the current database that weren't corrupt over the month old backup. There was some chunks missing, sure, but it could have been worse.

Drive 2 tested good on its own, so I just replaced drive 3 and built a RAID 6 array instead. Shortly after, the brand new drive 3 failed again (suspecting the port on the backplane now) and I think it took out drive 2 again, but the nice RAID 6 array is keeping it all running (with zero redundancy, I do realize). It's still running, so I get assigned other projects and don't have time to revisit.

We backup more often now.

 

[Solaris17]

Our trouble ticket/account management system is basically just a SQL server. Server has, I can't remember if it's 6 or 8 drives. I put it in RAID 5 because almost every other server we have is RAID 0 (with the occasional 1 in the mix) and RAID 5 is an improvement for us (others were built before me).

Well, drive 3 had an electrical failure and not only that, but it starts screwing with drive 2 above it. Drive 2 then starts writing corrupt data to itself (didn't quite realize this yet) and becomes unrecognizable. Pull drive 3 and drive 2 becomes recognizable again, but the OS fails to boot. Two drive failures in a RAID 5 array means data loss, of course, so I move from trying to recover it and just get down to the post-mortem.

Now, the only computer I know of with enough SATA ports to hook up all of the non-failed server hard drives plus a drive to boot off of is my main computer at home (a little mATX build I'm pretty fond of). Found some raid recovery program that worked really well, despite the half corrupt drive 2, and managed to pull the semi-corrupt SQL database off of it.
We dug up a month or so old backup of the database and managed to overlay the bits of the current database that weren't corrupt over the month old backup. There was some chunks missing, sure, but it could have been worse.

Drive 2 tested good on its own, so I just replaced drive 3 and built a RAID 6 array instead. Shortly after, the brand new drive 3 failed again (suspecting the port on the backplane now) and I think it took out drive 2 again, but the nice RAID 6 array is keeping it all running (with zero redundancy, I do realize). It's still running, so I get assigned other projects and don't have time to revisit.

We backup more often now.

Thats pretty crazy. I haven't dealt with a successful raid failure. I think Iv handled 1 5 that was gone. The others were understandably dead. The clients were running RAID 0. I can only imagine that was a satisfying feeling.

 

[remixedcat]

Acronis has saved my ass a lot