NNSquad - Network Neutrality Squad
[ NNSquad ] Practical Congestion Solutions (Real Soon Now)
________________________________________
From: Dave Burstein [daveb@dslprime.com]
Sent: Tuesday, July 29, 2008 6:10 AM
To: David Farber
Subject: Practical Congestion Solutions (Real Soon Now)
Folks
David Reed asked about how is Comcast managing traffic today, and what is likely later in the year. I have some of the data. I would enormously appreciate better information on what we can expect from DOCSIS 3.0 upstream when it's widely deployed (?late 2009.) I'm guessing it will be very good, but traffic models or other info that translate from 160/120 shared to realistic expectations per user would really help. My guess is 50/25 will be achieved 95+% of the time and that most of the cable upstream problems will be solved, but I just don't have enough data.
Takeaways:
1) The traffic at AT&T is rapidly shifting from p2p to streaming video such as YouTube and Hulu, per data below on the record and factchecked at a senior technical level. This matters, because any p2p solution will not do very much if p2p is much less of the traffic. It might postpone the need for an upgrade for 6 or 12 months, but p2p throttling can't do much more for most networks.
2) The updated model from Sandvine, Comcast's main supplier, allows much less obtrusive shaping. It can be tuned to neighborhood CMTS units rather than a wide territory, can automatically shut off when congestion clears, and enables more careful policies. Things like "if the user has uploaded more than 3 gigabytes in the last five days, reduce that connection to 256K until the congestion clears." Whether Comcast will choose a system like that, and whether the software will deliver everything promised, is not yet known. It certainly could come closer to "reasonable" than what they are doing now.
3) There really is a problem on (at least some) cable upstreams today, based on what I hear from people I respect who have the data. My hope - which won't be tested until 2009 - is that the DOCSIS 3.0 upstream will resolve most or all of the problems for the next few years. Full DOCSIS 3.0 has a minimum of 120 megabits upstream (shared) among typically 300 homes, something like 400K per subscriber. Current cable modems typically have 8 to 30K per subscriber. This is a huge difference.
Verizon, AT&T, and Free.fr are strongly on the record they do not have significant congestion problems. They do not have a shared local loop, and have allocated much more bandwidth per customer. I've sat at the network console of a large ISP and seen they had essentially no congestion amongst the millions of subscribers they served. They allocate 250K per subscriber, much more than current cable. So my best guess is that when cable gets to levels like that, the problem will drastically diminish. DOCSIS 3.0 120 meg shared downstream is now available to nearly 20 million homes (8M at J:COM Japan, millions more at Numericable France, Virgin UK, Videotron Quebec, Korea, and some others.) Very few systems can send at 50 and 100 megabits, leaving enough capacity for customers to actually receive at 50-100 megabits despite the sharing. Unfortunately, the upstream is not ready. Vendors say 2008, many think late 2009. 50/25 to 20M Comcast customers (promised for 2010) and many o!
thers is totally destabilizing if
the telco doesn't have fiber and the cablecos price like they already are in Japan (100 meg is only about $5 more) and France (100 meg, shared, is effectively $25-30 as part of a bundle.) The price will be the key.
Separately, a note came from Singapore to this list as I was writing, sensibly discussing whether problems are due to the ISP or the carrier. I wish we could think like that over here, but in the U.S. and Canada something over 95% of consumers get everything from the telco or cableco. We just don't have many surviving ISPs.
---------------
I unfortunately don't have the time to integrate these longer comments with the above, so forgive the length. If I had more time, I'd write shorter. The AT&T traffic data at the end is particularly interesting; the rest is not thoroughly factchecked yet but I think mostly on target.
1) The latest Sandvine quarterly call described what they say is now available for testing. Unlike the current models, which need to do something broad like slowing all the p2p across large systems, the new software allows much finer tuning, to the level of the usage of the individual customer. It essentially is a policy server tuned to let policies be set for each customer and enforced in real time. Choice of policy would be by the carrier.
The first improvement is that it can read the network and the individual CMTS, and only invoke policy if that customer is on a congested CMTS. This dramatically cuts how much is shaped.
The second improvement is that it can choose who to shape in a much more sophisticated way. Currently, they get total usage for each customer, so implementing a 250 gig cap for an individual or grossly shaping a whole region is what they are mostly limited to (I believe). Now, they can set a policy that chooses to shape only those who use more than 4 gig upstream between 6 p.m. and 10 p.m. , and only partially shape them (say to 256K up) if that's enough. It can then turn off the shaping automatically when the circuit drops to a more normal pattern (say, less than 60% utilization for 15 minutes.)
Contrast that to today, where apparently Comcast is leaving the stuff on for 18 hours a day, and filtering closer to the core, affecting many locations without problems. (The primary problem is at the neighborhood CMTS, and the current system I believe requires shaping dozens or hundreds at a time because the boxes are deeper in the network.)
How sensible will Comcast be? I can't answer that, but since Sandvine is designing the software working closely with their large customers like Comcast, I suspect the capabilites they are discussing are what the customer intends to use.
None of which means we can let up, but suggests much of the problem can be technically fixed.
2) Neither Comcast nor any of the other cablecos has shared enough data publicly, but privately some have provided information I believe I can trust about one crucial question. There is a real problem with (at least some) cable upstream, which typically designs for 6 to 30 kbps per subscriber. p2p frequently outruns that, even if they do ordinary upgrades doubling and tripling the total shared upstream but staying within that range.
The very interesting question is whether the DOCSIS 3.0 upstream will have similar problems. Verizon and AT&T, with far more effective bandwidth per user, aren't seeing a similar problem. So my first guess is that the the p2p congestion issue is important up to a certain level of bandwidth, but not above that. This is totally unproven, and the best cable engineers I've asked are unsure until they get thousands of live customers. DOCSIS 3.0 is at least a 12x (vs 10 meg upstream cable), for many a 30x (versus 2-3 meg upstream systems.) There's a lot of folks with this "voracious p2p grabs any upstream you can give it" idea, which seems to correspond to some of the experience on cablecos today,
However, Verizon, AT&T, and Free.fr tell me they simply do not have this problem. Bell Canada provided the CRTC with data that shows similar - the only problems of significance were at the back of the DSLAM, which I've determined was they haven't upgraded the link out of the back for many years, and have many OC-3's,etc. I've sat at the network console watching the realtime network map for millions of subscribers. The only congestion at that moment was at an interconnection with Telecom Italia, where Italia refuses to upgrade their side of the link. I believe this is typical of their experience; the network engineer told me he has authority and budget to simply upgrade any link that is a problem. They have fiber throughout, all IP, and are running it as the prototype "stupid network." If they see a problem, they add another Gig-E to the fiber link.
Comcast is going all 3.0 by 2010 - 20 million homes. So one of the most important questions is whether that will be enough to avoid upstream problems. All insight welcome. I have some interesting new data points. AT&T is seeing an overall slight decline in bandwidth demand growth per user, with p2p becoming less and less of a factor. YouTube, Hulu, and other commercial streamers are growing much more rapidly.
Here's a story I have, factchecked by a VP at AT&T Labs.
20% Drop in p2p on AT&T Backbone
Other video, like YouTube and Hulu, twice as high
Easily a third of AT&T's downstream traffic is now "web audio-video," far more than p2p and the gap is widening rapidly. Hulu and YouTube are taking over, while p2p is fading away on DSL networks. One likely result is that managing traffic by shaping p2p is of limited and declining use, perhaps buying a network 6 months or a year before needing an upgrade. The p2p traffic shaping debate should be almost over, because it simply won't work very much longer.
Jason Hillery provided some current AT&T information. p2p is currently still growing but "at a slower pace than other traffic." On the Tier 1 AT&T backbone p2p actually dropped 20% during a period last year. AT&T Labs VP Charles Kalmanek points out that a shift in customer mix, rather than an absolute drop in overall p2p, may explain that surprising statistic. Around the world, the trend is clear: web traffic continues to grow at something like 25-40% per user each year, right in line with the trend since 2001. Video is growing rapidly, but not enough to change the trend so far.
Direct from AT&T
"Overall traffic on the AT&T IP backbone network is growing at a pace of more than 50 percent per year. This growth is a combination of customer growth and growth in traffic per customer. Average growth in traffic per customer is about 25-30 percent per year.
To gauge the application breakdown of broadband traffic, we measure downstream traffic during the weekly busy hour. With this measure, as of June 2008, traffic was about 1/3 Web (non video/audio streams), 1/3 Web video/audio streams, and 1/5 P2P (with other applications making up the remainder). For the first time in June 2008, Web video/audio was the highest traffic-generating application over our IP backbone network.
As for the trends we've seen over the past few months:
-- Web video and audio is growing at a much higher pace than overall traffic (more than 70 percent/year);
-- P2P traffic continues to grow, though at a slower pace than other traffic;
-- and Web traffic is growing at a pace consistent with overall growth."
Many of the policy people believe that p2p is a ravenous monster that is devouring the Internet. The data show that simply isn't true, with the possible but important exception of some cable upstream. Traffic management has become so political few are willing to talk on the record, so reconciling conflicting data is very hard. It appears that large carriers are not having congestion problems on DSL or fiber backhaul, but cable upstream is. With friends, I'm trying to figure out what's going on and in particular whether the problem disappears with DOCSIS 3.0 which will have 12-30+ times more upstream bandwidth. Advice and data very welcome.
Video streamed/downloaded from sites like Daily Motion, Hulu, YouTube and others are coming to dramatically dominate web traffic. YouTube's market share is tending down, Hulu, BBC, Facebook presumably climbing. Heise has a German report that general interest in music over the web is dramatically down, consistent with my belief that most people who want music collections already have more than they can listen to. Hulu has now added movies, free with commercials. Most are old blockbusters The Fifth Element, Ghostbusters, Men in Black, Jerry Maguire and other old favorites. They have also Kurosawa's great Kagemusha, Quills, Enter the Ninja, Some Like It Hot,
AT&T has sensible plans to handle the load without disruption. They are already moving from 10 gig to 40 gig in the core, and planning a transition to 100 gig in a few years. The current projections are they can do these upgrades without raising capex, bringing per bit costs down along a Moore's Law curve and keeping bandwidth costs per user essentially unchanged. Most of the optical vendors believe they can meet those goals, although some worry that the pace of innovation may slow down as the optical components industry is struggling.
Sorry I didn't have the time to write something shorter.
db
Editor, DSL Prime
-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com