Serving up Nightmares

Do you ever have one of those nightmares where at one point you’re like ‘this is completely crazy, I’m totally dreaming’ but you don’t yet wake up?  You have to go through the motions to get to the end of the dream or find that resolution your subconscious mind was trying to figure out for you.  Today I’m awake and still trapped in last night’s nightmare.  No this one wasn’t brought on by acid reflux, this one was brought on by incompetence.

Over the past few months we have had problems with one of our servers, hosted with a company called Netriplex.  We lease our servers from a fairly reputable firm whose job it is to maintain servers.  It makes for a wonderful arrangement for us, normally.  Sometimes though, we wonder what expertise we’re paying for.  About a month ago, we finally narrowed down the issue to a failing drive array (or some such tech speak).  Three weeks ago we were supposed to get the data copied to a new drive array which would fix the situation.  That didn’t happen … they had underestimated the amount of data.  So we had clients we informed of the downtime, and now we’d have to inform them that it didn’t take place.  The next weekend, the same thing happened.  Last week we again went over the procedure, we don’t want anything fancy, we just want our files to be served in good time.  We just needed to clone the system from one drive to another (good) one and we could worry about patches and updates later.  This weekend the update took place and it became the latest nightmare.

What should have taken 2 hours wound up taking about 24, and all files have not yet been updated.  We don’t know if there was any data loss — we have too many files changing to be able to figure it out.  With over 120 websites, some of which with dynamic file systems and databases, anything could have happened.  Instead of running the clone and then synchronizing the final data during a short downtime, they instead tried to install everything on a whole new server with updates and everything while the original server was live.  So no wonder it took so long.  This is not what we asked for, nor wanted, but we got it good.  So now 32 hours into the process, the server is still not up to date and they’re hopefully not overwriting things that had been changed yesterday or today.  So today not only do we have the ever present normal work to do, we also have the cloud of ‘what if?’ hanging over our heads.  I will never claim to be perfect, or that we don’t make mistakes, I just get really frustrated when we’re paying professionals to do something, and they go and do something different and hurt us and our clients in the process.