For updates, skip to that section.

Introduction

Robert Lee and Jack Louis of Outpost24 recently went public claiming to have discovered a new and devastating denial of service (DoS) vulnerability in the core TCP/IP protocol stack used for almost all Internet communication. They refuse to release details before their talk at the T2 security conference in Finland on October 17 (see updates). Yet they have given many alarming interviews, and the press is having a field day spreading fear and uncertainty. Articles have appeared on The Register (“DoS attack reveals (yet another) crack in net's core”), Slashdot (“New Denial-of-Service Attack is a Killer”), Search Security (“TCP is fundamentally borked”), and many more publications. In the Register article, Robert Lee says “We haven't found anybody who has a TCP stack that runs TCP based services that isn't vulnerable” and that a target machine “basically self thrashes, and the only recovery after about two to four minutes worth of attack flow, even after the attack stops, is to reboot the machine”. The SearchSecurity article ends with this chilling paragraph:

“The best advice I have right now is don't allow anonymous connections. Make whitelist so only certain IP addresses can come in,” Lee said, acknowledging the impracticality of that for a Web server or mail server or virtually any other TCP-enabled device. “There's no real workaround right now.”

They gave a PodCast interview with an even gloomier prognosis for the Internet:

Q [Brenno de Winter]: “... we can draw the inference at this point that TCP/IP as we know it is broken beyond repair, as far as we know?”
A [Robert Lee]: “Certainly the implementations that we've played with.”
Q: “And that's basically Linux, BSD, Windows, probably all the routers that are out there?”
A: “yeah.”

Right after giving all these doom-and-gloom interviews, Robert complains about “the out of control barrage of fear mongering we all just endured.”

While I know and respect these researchers, I've had enough of the recent spate of people announcing (supposedly) massive security vulnerabilities, then refusing to back up their claims with details until a talk weeks or months away. Obviously Dan Kaminsky ignited this recent trend with his DNS flaw. While many of the researchers are earnest and call this “responsible disclosure”, it often reeks like a PR campaign. When you tell the press that you've discovered a core Internet protocol flaw so severe that you can't even provide any details for fear that the entire network could come crashing down, they just eat that up and it devolves into a media circus. If you think a bug is so serious that you won't share any details until it is patched, why announce and hype it in advance? That is just fear-mongering. It is also an amazing coincidence that when we've seen this sort of thing lately, the only safe (and “responsible”) time to release the details happens to be exactly when the researchers are giving their big presentations (and often being paid to do so).

My opinion on disclosure is simple: put up or shut up! If you go the full-disclosure route and announce the bug and full details immediately, that is great. Or if you want to coordinate disclosure with vendors and keep the bug secret until a patch is released, that is great too. But if you go with the coordinated approach (which vendors refer to as “responsible disclosure”), there is no point announcing and hyping the bug before a patch is released and you're ready to disclose details. I don't tell people how to report vulnerabilities—disclosure has long been one of the most personal and political issues in the security community. So I let them decide for themselves. But when people decide on the partial disclosure fear-mongering approach, I reserve the right to speculate on the issue as I do here. I recognize their vague description of the attack and results because I've written and used a similar DoS tool. I was not the first to do so, either.

Explanation

I am perhaps best known for writing the free Nmap Security Scanner (“Network Mapper”), a low-level tool for network exploration or security auditing. In 2002, I wrote a companion utility named Ndos (“Network Denial of Service&rdquo), which creates DoS situations against listening TCP services. It does this by exhausting resources of the listening applications or the OS kernel they are running on. There are numerous resources you can exhaust this way, but my main focuses were open socket limits in the listening application and TCP stack, TCP send buffers in the OS kernel, and concurrent process limits of the OS kernel (or configured by the application). I built the tool to use against some of my consulting clients. And I don't mean the ones who were slow to pay me :). I mean the ones who wanted me to test the resiliency of their networks against DoS attacks. Like Robert and Jack, I was stunned at how effective these techniques are at quickly taking down services. The basic attack starved web servers from servicing legitimate requests, and slightly more complex variants would sometimes take down the remote OS entirely.

The basic idea is to first firewall your source address(es) using a command such as iptables (on Linux) to prevent your own OS from interfering with your attack. Next you create hundreds or thousands of connections to the TCP port you are targeting (such as port 80 of a web server) as follows:

1. Attacker sends a TCP SYN packet to the target port from his own IP address (or one he controls) to request a connection.
2. The target port is open, so it will respond with a SYN/ACK packet—the 2nd step of the TCP 3-way handshake. Remember that Attacker sent the SYN as a raw packet from userland rather than using his operating system's connect() API to establish the connections. So when Attacker's operating system's TCP stack sees the unexpected SYN/ACK come back, it would normally destroy the nascent connection by sending a reset (RST) packet. This is why the special firewall rule was mentioned—to prevent such interference by Attacker's OS. Instead Attacker's DoS client handles all these packets by sniffing them from userland (generally using libpcap) and building/sending the raw reply packets.
3. Using the initial sequence number and other information from the SYN/ACK, Attacker sends an acknowledgment packet (the final step of the 3-way handshake) to complete the connection.

In many cases, this alone is enough to take out the service. Every open TCP connection uses significant resources on the target machine (various state in the kernel TCP stack, application memory, a socket descriptor, etc.) Many web servers and other daemons fork (or at least occupy) a whole OS process for each concurrent connection. Yet the steps above consume very few resources on the attacker side. So you can simply repeat the steps hundreds or thousands of times until the target exhausts one of its critical resources and malfunctions. As an attacker, you need a unique source port for each connection you make to a single port on a target system. This limits you to 65,535 connections per source IP address you control. Some major web sites may be so well-tuned that they can handle that many concurrent connections and you need to go after them from multiple IPs. On Linux, I generally do this by setting up IP aliases with iptables firewall rules to prevent the Linux TCP stack from interfering with my raw packet communication.

Once you have those thousands of open connections, you can get even nastier by sending malicious data payloads customized for the service you're attacking. For example, you can request a large file from web servers using each of your open connections. The server will then load the first part of that file into the OS TCP stack for sending, using precious kernel memory buffers. Those buffers can't be reclaimed until you (the attacker) acknowledge the data as received. But of course you don't acknowledge any data and the target's memory supply dwindles further with each malicious request you make. I've seen many operating systems (particularly BSD variants such as FreeBSD) reboot when their memory buffers (mbufs) run out from this sort of attack. I advised my clients to increase their kernel maxusers variable to make their systems a bit more resilient. Attackers can make this even more effective by manipulating TCP window sizes in their sent packets. A large window can be problematic for an attacker with little bandwidth, as he might be effectively DOSed by large and frequent retransmissions coming from the systems he is targetting. A low (or even zero) window size may help prevent this. Similary, you can create a TCP connection and then send out-of-order (but still within the server's receive window) data packets, leaving gaps between them. The server has to buffer this data since it cannot be returned to the application out of order.

Other options for nastyness include IP fragmentation and TCP segmentation. For example, you can waste memory by sending many large packets with each having one fragment missing, or you can leave a gap in the TCP streams by sending data at the end of the current window with nothing in between. The target OS may buffer that data until you decide to send the intervening packets.

You can easily tweak this attack to target different resources (such as requesting a dynamic page which requires significant CPU time to compute). These are just modifications of the fundamental attack, which is to use raw TCP packets to make a massive number of connections and (optionally) send malicious application-specific payloads for each connection, while tweaking details such as your packet timing and window sizes to have the most damaging affect.

News/Updates

Here is some news and updates since this page was originally written:

• October 18: After analyzing the attack details under NDA, Cisco has released a report rating the severity as level 3 (“mild damage”) and the urgency as level 2 (“unlikely use”).
• October 17: Robert and Jack give their T2 presentation. As with their SEC-T presentation, they hyped the claimed weakness but refuse to back up the claims with details. They now say they will release details sometime in 2009 (update: The 2009 claim was made in response to a user comment on that page, and Robert has now deleted all the comments).
• October 17: Cisco put out a response to the issue. They say that “the TCP vulnerabilities that Outpost24 announced are an extension of well-known weaknesses in the TCP protocol” and are still evaluating whether it affects them.
• October 16: Robert and Jack are now furiously backpedaling from plans to describe the claimed problems in their T2 talk tomorrow. Robert now says:

  We already gave a talk on this, a trimmed down version of the talk at the Sec-T Conference.... But we didn’t give the details on how those vulnerabilities actually work. And I imagine that we’ll probably give a similar talk at the T2 Conference with some clarifications. So I would say that the talk coming up at T2 is going to be mostly about what our attacks are not. Probably just to address some of the public guesses that have been coming out. But as I said a few times now, we’re under no pressure, no deadlines to release all the details of the information.

• October 5: Obtained Sockstress slides from Jack and Robert's SEC-T presentation, in PDF and Powerpoint formats. Some of the technical details seemed questionable, so I wrote a response. Robert responded that “those slides don't actually describe any of the vulnerabilities that we're alarmed about” and “we see no compelling reason to appease the internet security research community as a whole with full disclosure details”. Oh well.
• October 3: Robert says on his blog that their attack goes beyond what I've covered. I've updated the first three questions of the FAQ to reflect this.

The basic attack I described is certainly not new. I've been using it for many years, and I certainly didn't invent it either. In fact, it is somewhat straightforward for people with a strong background in TCP/IP networking. A thorough description of this attack, with proof-of-concept source code, was posted to Bugtraq by Stanislav Shalunov in April 2000. Then in December 2000, Bindview expanded the research with their much more methodical “Naptha” project.

I'm sure Robert and Jack probably found various ways to extend this attack. Once you start with the basics described above, it is relatively straightforward to optimize it for different scenarios or target operating systems. Robert acknowledges as much in one of his podcast interviews: “I think, total, I have five [attacks] documented. But it certainly isn't hard to come up with new ones.” You have many variables to control, including the number of connections, payload data (a ton of flexibility here), source IP addresses to use, window size, attack timing, IP fragmentation, TCP segmentation, etc.

One thing Robert and Jack have done is bring increased attention to this serious and long-running problem. While it isn't a serious threat to the Internet as a whole, I do consider it an important issue which should be fixed. Unfortunately, the attention only helps when it comes with details about the problem. That is why I wrote this page.

How do you know this is the same bug Robert and Jack found?

I don't, since they have refused to release full details. But this sounds like the same fundamental bug. Robert and Jack are smart fellows, so, again, I'm sure that they've found ways to extend and improve the attack in certain situations. But the simple approach described above is quite effective on its own. You don't even need to use more specific and esoteric attacks when the basics are so effective.

Robert says on his blog that their attack is different.

Well, slides from their SecT presentation on sockstress are now available (PDF, Powerpoint), and slide 17 says:

“Why Full Connection Flooding isn't more popular
  —A full connection requires attacker to consume state tracking resources too”

The attack described previously allows connection flooding without tracking any state, so they must not have known about it. While I think this is the fundamental weakness on which their attacks are built, they have probably extended and improved on it. My main issue is not with the research, per se, but with trying to hype the weakness in press interviews and the like before they are willing to share details about the claimed weakness. I don't believe that sharing the details would cause any problems on the Internet, as there are already many simple and effective denial of service attacks against TCP services (including those listed on this page). Many of the same techniques used to defend against all the other TCP DoS attacks will work against these newer ones.

How serious is this problem? If it is well known, why aren't attackers using it?

In part this is because the sort of attackers who perform DoS attacks are often lazy and unskilled. But the brute force packet flooding preferred by most attackers also has advantages over this admittedly more elegant technique. One is that the brute force packet floods can often be forged, making them harder to track down. For resource exhaustion attacks such as this one which require a full TCP connection, the attackers must use real IP addresses, making the attack sources easier to track down or simply block at the firewall. Note that the sources may be infected zombie hosts rather than the attackers themselves.

Another advantage of brute force packet flooding is that it is harder for an end user to block. With these full-TCP connection attacks, I can just block the attacker IPs. Packet floods, even when not spoofed, can be harder to block since by the time they reach your firewall they may have already caused their damage (wasted bandwidth). So you often have to work with your higher level service providers to block them upstream. This requires more time and coordination.

There are many ways to DoS a web site (or other services). If someone with moderate resources or skills really wants to shut down your site, they will probably succeed (at least temporarily). The good news is that organized attackers don't focus on this as much as other types of security attacks because it is harder to make money through DoS (though we have seen some "protection racket" style extortion cases).

Is it true that there are no workarounds or fixes for this problem? Robert Lee said in an interview that “There's no real workaround right now”.

Once an attack has started, it is relatively easy to identify and block the culprit IP addresses. Robert and Jack have confirmed that their attacks do not allow for IP spoofing. So you just block the attacker's IP addresses, just as is commonly done when fighting off other DoS attacks and resource hogs. Of course the attackers might find new IPs to attack from later. And they may cause significant disruption before you determine what is going on. Inline IDS/IPS software can detect attack patterns and automatically block them. Even the relatively simple Linux iptables firewall command has a connlimit module which can prevent an individual host or network from initiating too many concurrent connections to your servers. If you do this, you need to be careful about your thresholds so you don't block legitimate traffic. Normal web browsers often open many connections to a single site at once to retrieve different components of a page (images, scripts, etc.) Operating systems and applications can also be improved or tuned to be more resilient to this attack by using fewer resources per connection.

Robert and Jack disclosed the problem to CERT-FI, who put out a statement noting that “Based on our evaluation, the vulnerability can be mitigated by source address level filtering.”

Where can I obtain your Ndos tool?

I did not release it, since it was a quick hack (not production quality) and I was worried that it would be abused. In 2004 I co-authored a book, Stealing the Network: How to Own a Continent where I discuss Ndos and show the help/usage screen. My chapter, which is available free online notes that:

“Web sites are taken down by attackers daily, usually using a brute packet flood from many source machines (known as a distributed denial of service attack). Sendai realizes that much more elegant and effective attacks are possible by exploiting weaknesses in TCP protocol implementations rather than raw packet floods. Sendai has taken down much bigger Web sites than Fiasco's from a simple modem connection.”

What does this have to do with SYN cookies and SYN flooding?

A SYN flood attack is a very simple and well known TCP attack. It does not involve creating full TCP connections as this newer attack does. SYN cookies were developed more than a decade ago to combat the SYN flooding problem, but they don't help against this newer attack because it creates a full TCP connection and does not spoof source addresses. In the podcast which seems to have ignited this storm of publicity, Robert uses SYN cookies as an analogy for an aspect of their technique. He calls these “client-side SYN cookies” and they are a relatively unimportant implementation detail. In fact, they're not even required for the attack. But some news reports and commenters misunderstood his analogy and even recommended disabling normal (server side) SYN cookies (example). Doing so does not help prevent this newer attack, and will only make you vulnerable again to the simpler SYN flood attack.

Where can I learn more about this situation?

Here are some resources and articles related to this issue (I'm not endorsing them, just making a list):

• Cisco put out a response to the issue. They say that “the TCP vulnerabilities that Outpost24 announced are an extension of well-known weaknesses in the TCP protocol” and are still evaluating whether it affects them.
• After more evaluation of the attack under NDA, Cisco put out this newer bulletin. Here they classify the attack with a severity level 3 (“mild damage”) and urgency level 2 (“unlikely use”).
• Robert has been posting frequent updates to blog.robertlee.name. Note that user comments are heavily screened and usually not posted.
• The Sockstress slides from Jack and Robert's SEC-T presentation are now available in PDF and Powerpoint formats.
• sockstress.com uses a nuclear explosion logo and asks whether this issue is “the end of the Internet as we know it”. Oh dear.
• A Podcast interview with Brenno de Winter (trimmed to just the English portion by GRC) is available here.
• Article in The Register: “DoS attack reveals (yet another) crack in net's core”
• Slashdot article: “New Denial-of-Service Attack is a Killer”
• Upcoming T2 talk description: Sockstress - The Saga Continues.... Here is the description:

  This talk will divulge new technical details about Outpost24s (Jack C. Louis) research into TCP state table manipulation vulnerabilities that affect availability. Specifically this talk will showcase new attacks that will render a remote system unavailable using a very low bandwidth attack stream. Attacks against Windows, BSD, Linux, and embedded systems TCP/IP stack implementations will be discussed and demonstrated. In-line devices that keep track of state for multiple systems (read firewalls) tend to feel the effects of the attack even more quickly.

• DarkReading article: “New DOS Attack Is a Killer”
• ars technica article: “Researchers disclose deadly cross-platform TCP/IP flaws”
• Steve Gibson has an article and podcast about the issue.
• Dan Kaminsky discusses the issue.
• CERT-FI has a page on the issue, where they state that “Based on our evaluation, the vulnerability can be mitigated by source address level filtering.”
• CNET News article: “TCP flaws puts Web sites at risk”
• SearchSecurity article: “TCP is fundamentally borked”

What if I have more questions for you?

Feel free to email me (Fyodor) at fyodor@insecure.org.