What? Where?

On Sunday December 29th, Level3 Communications, one of the largest IP transit networks in North America and Europe, suffered an outage affecting sites such as ESPN, Amazon, and CNN. Noticeable missing from Level3? Any official acknowledgment or discussion of the incident; and an outage that disrupts ESPN on a Sunday will definitely be noticed.

When one of the largest internet backbones shows trouble, there are two ways to handle the situation (after fixing the issue of course). The first would be what we here call the “Duck and Cover” method (sometimes referred to as the Vinnie Barbarino approach). In this method, the company either makes no mention of the incident or does a classic Vinnie impersonation… such as:

Concerned User: I think there’s an outage.
Level3: What? Where?
Concerned User: I think there’s a problem with the network.
Level3: I’m So Confused. 
(mugs to the camera and audience goes crazy)

The other approach would be to acknowledge the incident, explain what is known, and explain what will be done in the future to prevent similar issues. This approach is, to paraphrase Billy Joel, is hardly ever heard but mostly what we need. We’re not sure why so few companies use the “acknowledge” approach — maybe it’s an old school, big business ideal that’s ingrained into company culture, but who knows? Open source companies, such as Digium, tend to disclose issues and discuss them, well, openly.

Open discussion tends not to turn folks away. Problems happen. How companies handle issues can either increase customer loyalty or send customers searching for alternatives. In our tech circles, most people seem to embrace the companies they can trust — and those companies tend to avoid the duck and cover approach.

So, where does that leave us with Level3?

Well, sadly, Level3 reminds us that your VoIP powered telephone system depends on Internet. Choosing several providers can certainly increase uptime, and choosing a company you can rely on will certainly help you sleep better at night. As for the 12/29 incident, we here (much like others) can only speculate. Current theories suggest a weather storm in Michigan may have been ground zero.

Other readings:

• Level3 Press Releases (notice no mention) 
• TechCrunch reports “The Tubes are Clogged”
• Broadband Reports “Outages Plague AT&T and Level3”
• SANS “Level3 Outage Confirmed”
• Slashdot “Storm Causes AT&T Outage Across Midwest”

About Level3:

Level 3 Communications (Nasdaq: LVLT) is an international communications company headquartered in Broomfield, Colorado. The company operates one of the largest communications and Internet backbones in the world.