Server upgrades, remotely fail and what to do

Well, we had a few remote server update upgrades over the weekend, and 5 of them went well, and 1 did not. All of them are over 25 miles away, so the one that is not responding is 50 miles away (actually 33 miles away per the customer since he read our blog and asked, hey, is that me?) I replied yes, and I have two more business clients within 10 minutes around you that this update worked fine, alas.

 

What happened next was hell for 3-4 hours. I helped him recover everything and get back to business yet it was at best, like not wanting to go to your dentist for a broken crown. I have to admit, backups saved me, him and the company. He had backups, and I had backups, and he was pissed. I did not do anything. All I did was what was necessary. In the end, he and I pulled together, we reconfigured the network, server, and all the other resources as quickly as possible. I was like, shit, this should not have happened, yet it did, and he was like, what else can I do, just fix it, free of charge, per se under warranty and move forward. Well, it ended up being 4 hours of work to reconfigure the network, restore the data, reconfigure a workstation to be a server, and move forward. Alas, I did not want to be paid, because in a way, it was a fault of the update, and a way a fault of the Microsoft emergency software update. To the end, my client paid me though I would have gladly walked away with it fixed and only come back when requested. It was a tough situation for me because I don't create work and I love my clients to be running seamlessly forever.

 

On the other hand, and forethought - this post was added below with consider of why this might have happened, and it was actually due to a number of waterfall issues. So, remember, even with your best efforts, notes, and documentation, there will always be something you might miss.

 

 

The best solutions and salvation was a series of good backups we used. 

 

 

more to come...