How Bad is it? Saga Continues

Posted on November 17, 2007 at 2:55 pm by jasonlee

We’ll I had expected that today’s update on our SAN saga would conclude the drama, but that isn’t the case. 

Little frustrates me more than when vendor’s tech support doesn’t do their homework and tells you incorrect information.  When we talked with tech support on Thursday, when ordering the DOM, we were clearly told that the configuration of our array in our SAN would not be affected when the new module was installed.  I wondered about the accuracy, so we asked several times, and the tech reassured us that the SAN would power up and all would be well when the new DOM was installed.  Well… as you can guess we received the DOM Friday morning and all isn’t well.  Jeremie installed the DOM and hmm… the SAN powers up, but doesn’t display any configuration.   While I appreciate the correct diagnosis by the technician of the failed module, I don’t like being told the wrong information.

So, that means we just work with support to rebuild the configuration and get the SAN back online… If only it was that easy.

Jeremie spent most of the day camping out in our server room leaving messages and waiting for our buddy Ray at Intel to call us back.  Even our Channel Partner got in the mix of leaving messages for Ray. Mid-day I asked Jeremie to start the process with a new technician since Ray was MIA.  This new tech sent Jeremie a form to complete and drops off the call.  Jeremie receives the form, but with explanation of where to send the form or with what information to input (the old configuration or our current situation).  So we wait some more trying to contact Ray to find out where this form fits in the mix.

At 5pm Jeremie conferences me in to another call to Intel and guess who we get a hold of… or buddy Ray, who starts to explain that he was calling all afternoon but nobody answered…  (He was calling our main number vs. the DID we told him to call)  I quickly told Ray I didn’t care about the why… but we needed a resolution.

So Ray then instructs us with what information to complete the form, and gives us the email address to where we should send it… and we wait some more.  Ray informs us our issue requires transferring our case to the engineering department, which might be closing at 7pm.  How can support close at 7pm?  It’s at that point I start asking for a supervisor. (Which Ray says will take 60 minutes for a call back)

So as time ticks away to the possibly not so magic 7pm, we wait.  At 6pm I receive a call from Oscar the support supervisor and he says the email hasn’t come thru.  It’s been 60 minutes since Jeremie sent over the form, but they still don’t have it.  I put a little pressure on Oscar and 2 minutes later he is able to search the email account and the magic happens and suddenly they have our email.  But the craziness doesn’t stop, Oscar says that it is 2-5 days to transfer our case to engineering.  I about blow a stack at that point and again I request a supervisor.  So at this point I tell Jeremie to get some dinner and we wait. 

So about 50 minutes later I receive a call on my cell and the Caller ID displays a 916 phone number it must be Intel calling to help us resolve our case. The caller identifies himself as the supervisor of server support for the central region… you know this call isn’t going to be good.  He informs me that all our paperwork is in place and our account is ready to be transferred over to engineering.  So I wait for that hated word ‘BUT’ and here it is, But the engineering department closes at 7pm.

So… Now we wait.  We have weighed re-working the raid array without support, but I am not willing to take that chance.  There is too much data on the SAN to risk and the thought of a total recovery from backup of 14+ servers.. that just doesn’t sound like an attractive option for not being patience.  So the decision is made, we wait until Monday.

Posted in Church IT

Commentary

  1. Ed Buford

    11.17.2007 6:54 pm

    I’m praying for you guys right now!

  2. Jason Powell

    11.17.2007 8:45 pm

    OH DUDE … I’m so bummed for you guys. What terrible coincidence that we both have major downs in the same week.

    Perhaps this free SharePoint training we’re trying to pull of is cursed? :-)

    Good call on waiting until Monday … although I feel your agony in having to wait so long.

  3. Jason Lee

    11.17.2007 9:07 pm

    Nope the Sharepoint Training IS FREE… nothing FREE is cursed.
    Thanks for the prayers, Our leadership is understanding and very patient.

  4. David Szpunar

    11.19.2007 11:42 pm

    Wow, sorry to hear that! I’m a little late but was nursing my own little RAID server crash, which I was fortunate enough to be able to fix with some inexpensive software. But my server appears to have been down for a tidy little subset of the time yours was down! Must have just been a bad weekend for data or something.

    Any resolution now that Monday has come to an end? You’re allowed to sleep before answering :-)

    What if companies had to pay US something every hour we waited on them while the ball was in their court? I know it’s a pipe dream, but I like dreaming of ways to become rich easily (you know, so I could make bigger donations to the Church).

    BTW on your blog comments, the day in the date is being doubled. I’d hack in and fix it for you, but you’re being good and are at the nice current 2.3.1 version that fixes the bugs…

Leave a response