NNSquad - Network Neutrality Squad
[ NNSquad ] Re: An unusual denial of service attack
As was recently highlighted in this forum, networks should be designed
around peak usage. Imagine AT&T whining in a tone similar to the one
Brett has struck about all the "abusers" of the system calling home to
mom on Mother's Day. (Pay attention, because this will be the
metaphore for the remainder of my message.) We all know it's coming,
just as network operator who's ever heard of "Patch Tuesday"
(http://en.wikipedia.org/wiki/Patch_Tuesday) should know that this is
going to occur.
Why are the users at fault for his inadequate network? Because they
love their mothers too much for a cynical codger at the helm of it
all. I truly pity Mr. Glass' customer base. The longer I read his
rants and raves through this forum, the more I come to understand that
everybody in his neck of the woods is frustrated by and with
him--himself included. We should all take heed and notice, for Mr.
Glass represents a functional microcosm of the dire situation faced by
us all: beware the naked man who offers you his shirt.
So Mr. Glass can't make friends with Akamai. So Mr. Glass can't
supply enough bandwidth to cover Patch Tuesday. So Mr. Glass can't
see the irony about the relationship between his own WISP and the
definition of monopoly. (Being the only broadband service provider in
a rural area counts.) What Mr. Glass is guilty of is far worse than
any of these: hypocrisy. Even as he discriminates against the
traffic on his own network, he bemoans the discrimination given to
him by his providers. Worse yet, he blames his users for every
trouble his short-sightedness engenders. (Does he realize that
Automatic Windows Updates is an option that people are PROMPTED to
enable, and do so by choice? Oh, but he blames Microsoft, and
conceals the true contempt we've seen spill out elsewhere on this
forum. Maybe he really needs to blame Dell or HP for enabling this
option by default for a class of computer-illiterates who would
receive little or no preventative maintenance for thier computers
otherwise!)
I wonder how many of his customers have the benefit of seeing the
kinds of things he has to say about them. It's obvious that he has
oversaturated his own network. He has fewer teats than can feed his
litter, and seeks to blame the unoccupied teats of others for his
plight. People, like most mammals, will feed until they are full. If
you can't support them, maybe it's time to kick them out or find
better ways to provide for them before you take them all down with
you.
I can see that the only tie his gripe has to the principles of network
neutrality, is that he'd rather all the children picked one
representative to call mom for all of them, and he'd prefer to be the
man that dictates who and when this may occur. He will lose this
battle the same as any dictator before him. The sooner he realizes
this, and participates in assisting all children in having rich,
meaningful conversations will their mothers on Mother's Day, the
closer he'll be to victory.
What exactly is it you want, Mr. Glass? More bandwidth? Fewer
customers? I'm sure there are business solutions to these that don't
require the kind of legal-intervention you regard with the kind of
on-again-off-again relationship that ordinary people can plainly
identify as destructive. It's like hating taxes, but begging for
welfare. You can't have one without the other, so make up your mind
already. It's obvious your competitive edge is long gone, and
everybody is suffering from it. You're determined to work harder, not
smarter, and we've all become the unwitting spectators to your
egomaniacal, self-tiring death-throes. It'd would be sad, if it
weren't so insulting.
From: Brett Glass <brett@lariat.net>
Date: May 4, 2009 11:07:50 AM EDT
To: dave@farber.net, "Ip ip" <ip@v2.listbox.com>
Subject: An unusual denial of service attack
Dave, and everyone:
This weekend, my ISP suffered an unusual sort of denial of service attack.
Starting on Saturday morning, users were reporting that their Web browsing
had slowed to a crawl, though other services were working properly. I
investigated, and saw that our upstream connection to the Internet backbone
was being saturated -- but not by any one customer. So, I looked at the
statistics on our Web cache (an activity, by the way, which I'm sure that
certain privacy advocates would find tantamount to "snooping," even though
it was for the purpose of managing the network). After awhile, I was able
to figure out what was wrong.
We were facing a distributed denial of service attack from the world's
largest "botnet:" Microsoft's "Windows Update."
Virtually every Windows machine on our network -- and most of our
customers's machines are running Windows XP or Windows Vista -- was
individually downloading many large updates. (See
http://www.computerworld.com/action/article.do?command=printArticleBasic&taxonomyName=Security&articleId=9131573&taxonomyId=17
for a list of some of the many security holes that were being patched.)
Fixing holes in Windows is a good thing, but to command more than 90% of
all of the computers around the globe to "phone home" at the same time is,
obviously, not. It's doubly bad when the updates are explicitly marked as
non-cacheable, making our Web cache of no use to stem the flood.
What's worse -- at least for our small ISP -- is that the updates are
distributed for Microsoft by a company called Akamai. Akamai, as many of
you know, places caches at the hubs of many ISPs' networks -- but, alas,
only those of larger ones. Our smaller ISP, which has never been able to
convince Akamai to place a cache at our location despite many years of
requests, therefore must use backbone bandwidth to service all of these
redundant requests. When I checked -- and it was not at the peak -- the
traffic was consuming about half of our main DS-3 line to the Internet,
leaving only half of its capacity available to carry all other traffic
(including VoIP and bandwidth-intensive streaming video). Our cache's CPU
utilization was above 95%, slowing response times still further.
I solved the problem by telling the cache to throttle traffic to and from
Akamai's upstream caches, which were serving up the updates. Instantly, the
load dropped off and normal service was restored.
As Spider-Man creator Stan Lee once noted, "with great power comes great
responsibility." Microsoft, by virtue of its control over Windows-based
PCs, has the ability to shut down the entire Internet at will -- and must
be careful not to do it, inadvertently, by turning 90% of the world's PCs
into a "zombie army."
Furthermore, content delivery networks such as Akamai, which distributes
Microsoft's updates, must not be allowed to discriminate against smaller
providers by making updates uncacheable (at least by a
standards-conforming Web cache) and then denying smaller ISPs access to a
cache that WILL cache them. (Google, too, is also placing caches at the
hubs of larger ISPs, thus giving them an edge when it comes to delivering
Google services and video.) Small and competitive ISPs already have a tough
row to hoe when competing with the telcos and cable companies. If they are
further disadvantaged by prejudicial business practices of content
providers and content delivery networks, Internet service will --
devastatingly for consumers -- become a duopoly.
--Brett Glass