Mysteries of the Unregulated Internet

One Sunday afternoon last February, the YouTube Web site disappeared from the Internet. YouTube didn't take it down.

The problem came from Pakistan, when a telecommunications company suddenly began rerouting traffic to and from the Web site into an Internet black hole.

Incidents like this fascinate University of New Mexico (UNM) graduate student Josh Karlin. With a National Science Foundation (NSF) grant, he built and posted an Internet Alert Registry (IAR) that automatically sends an e-mail to registered Internet service providers when there is trouble with traffic in their section of the World Wide Web. The registry is free and any Internet service provider can sign up for the warnings.

What happened with YouTube 

Karlin says anomalies in the way Internet traffic flows show up nearly 200 times a day. Most of the problems are small and disappear in a few hours. A few, like the hijacked YouTube traffic, are big and require Internet Service Providers (ISPs) to work together to solve the problem.

"What happened is the ISPs that were close to Pakistan Telecom, that were in fact forwarding Pakistan's data, said 'Oh, this is obviously wrong. We're not going to propagate it.' And then they shut it off," said Karlin. "They filtered it out and then suddenly the problem disappeared and YouTube was getting data again."

Karlin points out most users don't think much about how the Internet works. He says we assume someone, somewhere is in charge, taking care of problems, settling disputes, and punishing troublemakers.

But that's not true. The Internet works because thousands of independent ISPs work cooperatively together to keep traffic running smoothly.

How it all works

Every computer in the world that is connected to the Internet has an address. Those addresses come from the Internet Assigned Number Authority (IANA). That entity assigns the numbers, but it doesn't police them.

"The IANA has been giving out these addresses for a very long time, and people have lost track of where they've gone," said Karlin. "So, some companies that were given Internet Protocol (IP) addresses have folded or sold them to other companies or broken them down into small blocks and given them out to other people, so nobody really knows what's where."

For instance, the University of New Mexico has thousands of Internet addresses assigned to it. But there is no agency that monitors whether UNM only uses the addresses it has been assigned. So how does any ISP sort out what is legitimate and what is not?

There are dozens of companies that sell services to help ISPs sort out suspicious activity from normal traffic. The IAR will alert providers as well. But researchers are only now trying to figure out how to handle suspicious traffic when it suddenly appears.

Karlin is one of them. He and his advisor, UNM computer science professor Stephanie Forrest, and Princeton University computer sciences professor Jennifer Rexford are working on an improvement to the Border Gateway Protocol. The modification changes preference to allow ISPs to automatically route traffic around a source that makes an unexpected change in routing.

Getting around problems

Their protocol emphasizes the status quo. If traffic is flowing along like it is normally, it means everything is ok. If traffic suddenly begins to flow in a different way, the yellow flags go up and their protocol automatically selects a more stable and trusted route.

That buys time for the ISPs to figure out whether this is a traffic hijacking, as occurred in the YouTube case, or not. If an alarm is raised, as it was by YouTube, the ISP can avoid using the anomalous route.

The Internet Alert System and the new protocol will eventually work together so that routers can automatically avoid suspicious routes while the pertinent ISPs are informed of the problem. This way, potential problems unfold slowly rather than instantly.

Karlin has noted that the Internet began as a messaging system between researchers who trusted each other, and so far the system still basically works on the idea that routes being advertised around the world are correct. But as more and more networks join the Internet, the likelihood increases that mistakes will be made that cause problems. Karlin's new protocol treats the mistakes as mistakes rather than attacks and allows for a positive, rather than a punitive, solution.

-- Karen Wentworth, University of New Mexico kwent2@unm.edu

This Behind the Scenes article was provided to LiveScience in partnership with the National Science Foundation.