Today’s Hillrunner.com downtime

Welcome! Forums Non-Running Forum Today’s Hillrunner.com downtime

This topic contains 0 replies, has 1 voice, and was last updated by  Ryan 12 years, 2 months ago.

  • Author
    Posts
  • #5229

    Ryan
    Keymaster

    I'm terribly sorry about the situation today with Hillrunner.com being inaccessible for many people for a good part of the day. Considering the fact that I intended to use a day off today to get some website work done, I'm sure I was as disappointed as anyone with the issue.

    I contacted the administrators of the server Hillrunner.com is on this morning and they said they were working on the issue and were hoping for it to be up again soon. Unfortunately, as you are probably aware, it took longer than they expected.

    The explanation of what happened is kind of technical but here's the short story. AT&T made a mistake in their Internet backbone (the equipment that routes essentially all Internet traffic). According to AT&T, this affected about 25% of all Internet traffic. 75% of people presumably were still able to access Hillrunner.com. I hope that's the case.

    The long story for those who care is as follows (copy/pasted with minor edits from the support group that maintains the server Hillrunner.com is on:

    AT&T mistakenly null-routed the ip address of [the server Hillrunner.com is on] on Tuesday.  This change filtered (moved or propogated) from router to router in the AT&T internet backbone until the entire US AT&T routing was affected.  Null-routing sends any requests from the internet, to that particular IP address, to nowhere or null.

    The easy part was to find that there was an error (many folks called but we were already aware there was a problem), the hard part was to convince a giant company, like AT&T, that they had caused the problem.  It took several hours to convince them that the problem was on their side and once they were convinced they did fix the routing table entry fairly quickly.

    However, fixing the routing table entry did not translate into a quick fix as far as web site and email.  Routing table propogation takes anywhere 1-4 hours on the AT&T network in the USA (these are AT&T's time estimates).  As of 3:30PM CST we see that these changes have taken place in the entire US AT&T network.

    There was some good news, though, during this time.  The outage was not complete but was only limited to customers whose ISP's route thru the AT&T network.  AT&T is only one of our 9 network links but does cover about 25% of all ISP backbone services…..this means that about 25% of the internet could not send you email or get to your web site unless your ISP was multi-homed (connected to more than one ISP) and could automatically route around the AT&T outage.

    So your email accounts did collect email (on the server) and about 75% of the internet could view your web site.

    While we cannot do anything about an AT&T mistake to prevent it in the future, we have documented the procedure for finding and fixing this type of problem if it were to occur, again. 

    We, at Real-Marketing,com, sincerely apologize for the problems that this outage has caused.

You must be logged in to reply to this topic.