By David Oliver

We did have the opportunity to do this blog post as a video recording and put it on YouTube, but we realized that, ironically, as the post is all about how companies use NetFlow to track YouTube, because YouTube can, in many cases, suck down bandwidth, it was probably best just to write this out in text.

As we mentioned a week ago, YouTube is now supporting high definition content, with a high bandwidth to match.  Now, I've done a little bit of research into how YouTube actually works.   So I thought I’d explain to all those companies out who don’t yet have their own solutions some ideas about how to track and manage YouTube and other streaming media data – as well as give users out there an idea of exactly how companies can track your YouTube usage at work. 

Anyway, when you make a request for a video on YouTube, you are directed to YouTube’s servers via one of four IP addresses that are easily found on Google or other search engines.  From there you're going to be relayed to the Limelight network, which will actually feed you the video in the flash-based player.  You can see the flows to and from that initial IP address for the HTTP GET of that video. 

There are many solutions for providing visibility into traffic on the network by looking at the Cisco NetFlow data (which is already on most Cisco routers).  I’m going to refer to NetQoS’s own solution,  ReporterAnalyzer, when I talk about tracking NetFlow data. 

What we can do with ReporterAnalyzer is monitor the Internet-facing link, and create and use custom reports looking for YouTube’s specific IP addresses.  If you see a substantial amount of data being transferred,  that's a good marker of seeing that YouTube video traffic. 

You can rely on those custom reports and run them anytime you want, but companies can also monitor YouTube in real-time.  By mapping HTTP Port 80 traffic that involves one of YouTube's IP addresses to some other ephemeral port, (and naming it something catchy, like "YouTube,") it'll actually show up as it's own protocol in both real time reporting, as well as flow forensics.    You could use that data to create customer reports, to get a comprehensive list of users, and to sort YouTube use by volume. 

The other thing you can do is use analyses to know when YouTube traffic accounts for more than, say, 10% of any of my links' traffic.  Then it will go through on a link-by-link basis and tell you about violations, helping you further localize the source of that traffic. You can also configure it to alert you when and only when YouTube traffic on a particular link passes a threshold that you set. 

(The other option is to try to block it entirely, but that's an engineering nightmare.  Any employee smart enough to provide good value to a company - particularly a high tech company - will likely be smart enough to know how to circumvent blocks through proxies and other means.)

Custom reports to find correct addresses and to localize YouTube traffic may take a couple minutes.  The entire real-time application mapping process takes maybe another 15 minutes.  I can be showing real-time data specific to YouTube traffic just a few minutes after configuration of application mapping.  (If your boss asks in the morning for something to track YouTube usage, the company can get YouTube tracking up and running by that afternoon - if the boss just wants some a quick snapshot of the current YouTube traffic volume, it could take as little as five minutes through custom reports.)

Of course, this isn't limited to YouTube.  You can use similar methods and techniques to find and track streaming audio feeds, other video sites, etc. Any TCP flow is going to create some sort of NetFlow data.  Based on the source or destination address, you can localize that.  So as long as ReporterAnalyzer has visibility of that destination address, they can report on it.  As you know, there are a multitude of media based streaming sites, all of which are going to have their own IP address range, which you can find pretty easily.  You can then further localize and label them so that when you pull up reports, they're already differentiated from other traffic.

While YouTube is great, we’ve found that YouTube traffic congesting corporate networks is a common issue. For any company, WAN links are a finite resource and need to be managed.  It's something that's a concern because you're sizing your network around capacity needs for the business.  YouTube is (usually) non-business traffic, but it's going to share that limited resource.  The more you share a resource, the less is available for the requirements you originally scoped it for.  At NetQoS, we’ve found YouTube traffic congesting corporate networks is a common issue.

Most Americans can barely remember a time when there wasn’t enough electricity running to their house or apartment. Oh, certainly, we can remember times when we’d blow a fuse or trip a circuit breaker; but they were rare and usually happened under excessive strain – when the hairdryer and the air conditioner were on at the same time as the electric stove or George Foreman Lean Mean Fat-Reducing Grilling Machine.

In the 1950s, according to this advertisement uncovered by Modern Mechanix, running out of juice was a real problem because the wiring in house built decades earlier simply didn’t have enough capacity to run all those appliances. Well, of course it was – and this brings me back to my undergraduate days as a history major at the New Jersey Institute of Technology. (Yes, NJIT had a history major. No, it wasn’t a big class…)

Now, the second industrial revolution was years prior – during the 1910s through the 1920s, but consumer adoption of technological advances was stunted for a period of about 40 years. The great depression killed disposable income, when the war came and finally did bring some income, there was significant rationing and many companies specializing in electronics and electromechanical devices were building for the war effort. In a sense, when money started flowing in during the late 40s and 1950s, consumer demand had been “pent up.” Yeah, consumerism went overboard in the 1950s, but can you blame ‘em?

Fast forward to today and replace appliances with applications – there’s a reason that both have the same root word, the Latin “applicare” – and you can see the parallels.

Of course, things are a bit more complicated than the 1950s – there are only a few types of electricity – AC/DC, 110v/220v… etc. The difference is that it was relatively easy to tell whether or not the problem you were having was caused by a lack of electricity. Does the thing light up? Is it working? If the answer is no to both, you’ve got an electricity problem. If the answer to the first is yes, and the answer to the second is no, you’ve got a busted appliance. (If the answer to the first is no and the second yes, you need to replace the lightbulb.)

Today it’s a little bit harder to diagnose the problems you have with application performance – application, server, and network problems can all look very similar – especially to the end-user. Sometimes you spend time and energy working on one area only to find out it’s not the problem.

But sometimes, yes, the network is the problem, and more capacity is needed. The important thing is to be sure about it rather than just guessing.

There’s another lesson here too for consumer applications as well. That is, in the 1950s, the way to solve the problem of new demand on the electrical grid was to upgrade the wiring. Maybe we should be doing that to solve some of our own broadband “shortage” problems, instead of resorting to things like “bandwidth caps” and “aggressive traffic shaping.”

Because it’s not just about not being able to see the latest cat on treadmill video. Videoconferencing via Skype or other method puts people in face-to-face communication. CERN scientists needed to upgrade the Web’s infrastructure to share the massive amounts of data the LHC would create.

But one of the most telling things is what’s going on in Austin right now.

Our local NBC affiliate, KXAN, and our local cable provider, Time Warner Cable, haven’t reached an agreement to show KXAN programming on cable. One of TWC’s gambits is running an advertisement on television showing people how to connect their laptop computers up to their television so that they can watch the streaming video of the NBC shows that they’re missing.

Which, of course, begs the question; if you can get television shows via the Internet directly from the networks themselves, why do you need cable TV or network affiliates in the first place? This move may backfire. One of the reasons I don’t have cable is that I’ve been hooking my TV up to my computer for years now… then again, I have pretty good broadband service.

According to Inside Higher Ed, one of the provisions of the Higher Education Act, which was passed last August mandates for colleges to police their network for illegally copied copyrighted works.  And according with a survey conducted by the Campus Computing Project, this will cost colleges $350,000 to $500,000 a year out-of-pocket to comply.

Colleges are “required to consider the use of technology-based deterrents” to prevent or deter copyright-infringement on peer-to-peer networks.  These can include traffic monitoring and packet shaping.

There are a number of problems with this plan:

First, the record labels want the colleges to pay money because some of their students are violating record label copyright.  The colleges are stuck in a bad place; because the government has mandated it.  It’s highly likely that these measures will be ineffective, but even if they are effective, the copyright infringers will just move off campus and get residential broadband, leaving all the students paying higher tuition and residency bills because the colleges have to pay for the ineffective enforcement somehow. 

So, who benefits from this law?  The labels?  This won’t stop one copyright infringer, and I think they know that.  What it does do is pass on the costs of ineffective piracy countermeasures from the labels to the colleges?  The colleges don’t get anything – nor do the students, pirates or otherwise. 

Nobody benefits, and most people suffer.  This is a crappy law. 

Additionally, there are significant political and academic freedom issues in requiring a network to monitor the traffic for particular activity.  This is especially discouraging when you consider that the rate of false positives is going to be extremely high: there is no traffic monitoring solution that is smart enough to tell whether a particular MP3 file is legal or illegal.   Any activity that blocks illegal content can easily break legal ones: fair use of the work, public domain, creative-commons, or downloaded by the copyright holder.  (Recently, a smaller record label had its site pulled because it had hosted copyrighted MP3 files – its own.)

So, in short, it’s stupid, ineffective, and has potential unpleasant side effects. How do you best comply with a law that’s so dumb that it borders on ridiculousness?

The only way to fight stupidity is by being clever – find solutions that won’t cost hundreds of thousands of dollars and which do not harm the ability of students, faculty, and alumni to advance the academic mission of the school.  For example, enabling and properly using Cisco Netflow information to make intelligent decisions about QoS policies.  (Labs and offices get more leeway than student residences, local proxies of popular legal BitTorrent files – like WoW patches and Ubuntu releases - on the LAN.) 

Considering that the provisions in the law (and I am not a lawyer, this is not legal advice, and even if I was a lawyer, I probably wouldn’t be a good one anyway) basically amount to: “Do something” about P2P copyright infringement, it’s probably best just to use QoS policies to throttle down (without cutting off) that traffic which looks suspicious.  But err on the side of assuming that the use is legal and for the academic mission. Maybe set up a filter that only goes into effect if the protocol is BitTorrent, and the content is a video file, and the seeders and peers aren’t all from IP addresses that correspond to astrophysics laboratories. 

You could also make use of anomaly detection software to track spikes in MP3 traffic – that is, of course, assuming that the downloading of music files by the 18-25 demographic is in any way considered “anomalous” rather than “status quo…ulous…” Something like that.

David Oliver talks about his experiences running the 24 Hours LeMons race in Houston, and how knowing about network performance helped him optimize his junker.

We mentioned previously that Comcast was moving towards capping its residential customers at 250GB a month of Internet data. We also mentioned, repeatedly, that bandwidth caps really don’t solve the problems of network congestion or of poor latency, but if you’re going to go for a data cap, 250GB/mo seems a reasonably fair rate. Silicon Alley Insider has a rundown of exactly what 250GB means, and it actually is quite a lot of data. It misses the point, but at least it is a lot of data.

The big complaint now seems to be that Comcast has provided no way to inform the user of exactly how much of that 250GB limit they are using.

This has two major implications.

First, it encourages people who would otherwise be using the Internet normally to use it more conservatively. I don’t want to abuse the term “chilling effect,” but if your choices were to watch a movie via NetFlix’s online streaming service or ordering it on cable Pay-Per-View, you may have plenty of data to watch it, but if you don’t know how close you are to your cap, or how much a particular application consumes, you’re less likely to use the Internet. It may be a psychological block but it decreases the value of the Internet applications you use, and thus, decreases the value of the Internet connection that you lease. It also decreases the value of the “cognitive surplus,” as we’ve mentioned.

The second is simply that you can’t manage what you can’t measure – it’s as true on the residential level as it is for the largest corporate networks. Silicon Alley Insider’s numbers are, as far as we can tell, accurate, but a tech-savvy family of four could easily go over that limit, and it could be difficult to tell exactly who or what is responsible for data consumption. Dad’s teleconferencing, Mom’s downloading a Linux distro, Junior is watching a documentary on a topic for school via NetFlix, the little miss is live vodcasting, and the dog is downloading a torrent of the entire “Lassie” series. (Point is: without network visibility, everything gets blamed on the dog.)

Not providing a running tally of data “consumed” means that there’s no way to determine what actions and activities drain the most bandwidth – Was it the movie you watched last night or the marathon game session next morning? Has a neighbor been using your (unsecured or inadequately secured) wireless connection, or have you been hosting malware? How much bandwidth does playing World of Warcraft take, and how does that compare to watching YouTube, and how does THAT compare to other services like Blip.TV and Vimeo? How does a person know whether or not they’re coming close to the limit?

The MacObserver has a few tips for monitoring bandwidth consumption on a Macintosh, and there are applications that we’ve used to track bandwidth consumption on a single PC, but right now it seems that the best bet for tracking the consumption of multiple PCs to the Internet is to install a firmware like Tomato onto your home router and monitor it from there.

While we’ve talked quite a bit about what impact the Olympics may have on an enterprise network’s performance, we haven’t talked much about the performance of the NBC site hosting the live streaming of the Olympics. 

According to Jason Perlow at ZDNet, Limelight networks (which hosts the streaming videos) deployed the videos by going to the public internet by hosting the content more locally – at the ISP.  That means you’re viewing the Olympics through your ISP’s internal network, and the broader internet doesn’t even enter into the connection. 

This is smart thinking, it appears to be working, and by all measures this should be applauded.  Perhaps even duplicated – if you know that multiple employees will download the same content, local hosting on the LAN is preferable to duplicate download streams tying up the more expensive, slower WAN lines.

From the enterprise end of the equation, the fact that Limelight is delivering Olympics video more effectively just means that IT managers cannot count on their servers going down from being unable to handle the demand – IT managers still need to monitor their own networks for performance problems when a big event like the Olympics come up. 

However, it would be wrong to assume that Limelight’s strategy is the only reason why Olympic live-streaming hasn’t slowed to a trickle.

First of all, the site blocks 95.44% of visitors from accessing the content – because it limits the content only to those in the United States.  That’s a lot of people.

Secondly, the site requires Microsoft Silverlight. Most people don’t have Silverlight installed.  Some can’t even install it on their systems.  And there are certainly going to be a quite a few people who just didn’t think installing Silverlight was worth the bother to watch five minutes of Olympic footage they may be mildly interested in. 

And finally – none of the really popular sports are being streamed.  Gymnastics, Women’s Beach Volleyball, Swimming (with the exception of synchronized) and most of the track and field events aren’t available live. So you’re left with judo, fencing, and the decathlon.

So while it is a true technological wonder that the lights have stayed on and the site performs admirably – it is important to recognize that Limelight has not found a magic bullet to deal with extremely high internet video demand. 

A number of East Coast based customers of World of Warcraft have been experiencing connection delays and uncomfortable lag – and no one seems to know exactly where the problem is.

The New York Post says that Blizzard is blaming Time Warner Cable is for the problem:

"The only commonality between all the players experiencing these disconnects and extreme latency is Time Warner/Road Runner," the company said in a June 23 support post.

But the Digital Communications Director for Time Warner has said that the lags and disconnections are not on their end and points to the traceroutes as evidence.

Take a look at some of the traceroutes posted to the thread in question ... starting here, at comment #446: http://tinyurl.com/5gqe27

If you follow the commenter's posted trace results, you'll notice that it's only on TWC's Roadrunner (rr) network for the first 6 hops — with maximum response times of 10 ms. The response time jumps drastically at hop # 11 — when the trace is no longer on the Roadrunner network.

Scroll down further on the same page to comment #456, and you'll see something similar — a giant leap in lag times. However, this trace never touches our network. It starts at Verizon, goes to Alter.net at hop #5, and then jumps to ATT.net's network at hop #8. Hop #9 shows a response time of 114 ms — quite a jump from the 49ms at hop #8.

So, what’s going on?

One of the theories is that Time Warner is lying and is throttling World of Warcraft traffic, considering all the bad blood between savvy broadband users and major ISPs over BitTorrent throttling. And while I can’t prove that they’re not doing so, I have to admit that the theory doesn’t seem very likely because of the nature of World of Warcraft.

See, MMORPGs care more about latency than bandwidth. While patch downloads can be huge, the majority of the content of WoW requires low latency to provide instant responses to actions. Latency, in WoW can result in an annoyingly choppy game, and a multi-hundred millisecond delay may be the difference between slain dragon or hobbit pâté.

So from a bandwidth-saving perspective, a ISP wouldn’t have a whole lot of motive in blocking World of Warcraft or other MMORPGs.

Additionally, Comcast, Time Warner, and other cable companies were rumored to use BitTorrent throttling because both legal and copyright infringed video files competed with the standard television cable offerings of those companies. This also doesn’t seem to be the case – as while more generally, time spent playing WoW is time not spent watching TV, it’s not a specific competition. Indeed, MMORPGs are one of the key drivers for broadband speeds in the U.S., and I have trouble believing that TW or any other company would knowingly interfere with such a cash cow.

Indeed, I believe that TW might be reaching out to users to find out more about the problem because TW might be interested in solving the problem instead of losing customers to other ISPs like Verizon FIOS.

Of course, I don’t know anything – and I wish that I had some inside information to figure out what was going on and solve the problem. Not only would I look like a genius but every one of my friends who plays World of Warcraft would hoist me on their shoulders, and treat me like a Lich King for a Day. Sadly, I think that it’s going to take Blizzard and TWC together to try to triangulate why this problem is happening.

Dr. Tim Wu has recently penned an editorial to the New York Times entitled “OPEC 2.0.” which argues that we need an alternative source of bandwidth much like we need alternative sources of energy.

It’s the sort of editorial which causes other tech columnists and blog editors to immediately loathe themselves because it’s obvious in retrospect, but Dr. Wu thought of it first.

Just as the industrial revolution depended on oil and other energy sources, the information revolution is fueled by bandwidth. If we aren’t careful, we’re going to repeat the history of the oil industry by creating a bandwidth cartel.

Like energy, bandwidth is an essential economic input. You can’t run an engine without gas, or a cellphone without bandwidth. Both are also resources controlled by a tight group of producers, whether oil companies and Middle Eastern nations or communications companies like AT&T, Comcast and Vodafone. That’s why, as with energy, we need to develop alternative sources of bandwidth.

Pointing out that the average American spends roughly the same amount of money for bandwidth of some form or another, whether on their cellphones, land-line phones, cable TV, broadband Internet, etc., as they do on heating, cooling, electricity, and gasoline, Wu makes the case that we need the data equivalent of the compost garden, rooftop solar panel, and electric scooter.

The difference, of course, is that while rooftop solar panels have recently started to be affordable on a per-wattage basis, Wu’s suggestion of “running your own fiber to your home” isn’t. Even if you did put down your own last-mile fiber, you still need some place to connect it to… the whole point is that the value of a network is equal to the number of end-points on the network, squared.

The other solutions Wu suggests: re-allocation of the EM spectrum, municipal fiber as a public utility, and increasing the amount of competition are all more practical ideas, but they require significant changes in governmental policy – the same government elected from a two party system where the two parties are both heavily influenced by broadband providers and telecommunications companies.

But Wu is not wrong in identifying a significant problem – especially since e-mail, telecommuting, and teleconferencing are all considered important ways to reduce energy expenditure. It is disheartening to believe that we’re simply switching from one artificially overpriced commodity to another.

Damn. Well, I guess there’s no way around it then. Looks like we’ll have to recreate the Internet from scratch by hooking up everyone to self-installed network connections. So what do you say, you lay the fiber if I use the backhoe?

It’s Download Day.  At 10:00 a.m. PDT, or noon, for us in Austin, Firefox 3.0 was released to the public in what the Mozilla foundation has dubbed “download day.” In fact, they’re attempting to set a Guinness World Record for “most downloads in a 24 hour period.” 

So, it was a bit of a concern to us because with all those people downloading Web browsers, there would be sure to be traffic spikes on our network. But the “Download Day” promotion is such a huge success that Mozilla is having trouble keeping their own server up. 

At 10:16 a.m. PDT, I can see a “The server at www.spreadfirefox.com is taking too long to respond” error.  Mozilla.org is also unable to resolve. 

At 10:30 a.m. PDT, it’s still not connecting, and I decide to stop hitting refresh and go and eat lunch. Mmm.  Roast Beef. 

At 11:30 a.m. PDT, Spreadfirefox.com is still not resolving, but Mozilla.org does.  That doesn’t last, however, as I go to download Firefox, I get a “Http/1.1 Service Unavailable” error.   I bring up a copy of “Waiting for Godot” in another browser window.

It is 12:00 noon on the Pacific.  Spreadfirefox.com is still not resolving. 

12:30 p.m. PDT.  Still not working.  I clean off my work desk, something I’ve been putting off for a wh—ew, is that mayonnaise?  (I hope that’s mayonnaise.)

1:00 p.m. PDT. No Firefox, but My desk is now clean.  (My closet is now dangerous.)  Time to catch up on my RSS feeds to find out if there are any interesting leads that I can investigate. Hmm.  Wine 1.0 is out, but that really doesn’t have a lot to do with network performance.  Reddit seems have problems with Firefox too.  But somebody has to be getting the browser – there’s over 8000 downloads a minute according to the counting tracker.  Wait.  Some users report the counts running backward… what, are people uploading it back?

1:45 p.m. PDT. Aha!  Finally.  The page resolves and I begin my download… and it redirects me to Firefox 2.0.0.14.  Great.

1:55 p.m. PDT. I download Opera 9.5.

2:00 p.m. PDT. Mozilla’s page finally shows a link to Firefox 3.0 – but still shows the logo for Firefox 2.  The 7.1 MB download starts at around 50kBytes/s – which is pretty lame for the usual 700kBytes/s I can get when I download from work. 

2:15 p.m. I install Firefox 3.0 and launch it.  It’s nice.  It’s certainly more responsive and uses less memory.  However, my Tab Mix Plus extension isn’t compatible, and furthermore, there’s no option to undo closed tabs.  All in all, a disappointment – if it were a restaurant, it would be infamous for slow service and bad food.

Leaving aside the whole “Undo Closed Tabs” issue, you would think that an organization actively trying to beat the world record for the most downloads in a 24 hour period might, you know, be prepared enough to make sure the servers don’t go down?

Additionally; Mozilla has been promoting “Download Day” for some time now, so it makes sense for IT departments to be prepared for the onslaught of downloads coming into the network from users upgrading their PCs to the latest version of the browser – and keep track of the impact that traffic has on the user experience for more mission-critical apps.

by Brian Boyko
Editor, Network Performance Daily

Ars Technica reports that the National Institutes of Health released a study which show that wireless-phone only households are increasing – currently 15.8 percent of households. 

But that’s just part of it – other consumers are switching to VoIP services – gobbling up another 13.8 percent of U.S. households. 

There are certainly economic factors which result in this change - me, I cancelled my landline phone service when I realized I was paying less on a month-to-month basis on my cellphone than I was for a service I barely used.  Since then, I’ve moved around a lot (seven times in the past five years) and keeping the same cellphone instead of canceling and re-enabling service makes a hell of a lot of sense. 

But still – three out of ten consumers?  This is a major shift – and one that’s likely to continue as cellphone and VoIP quality gets better. 

Of course, there will be more demand for VoIP networking in the future – and with it, a need to monitor VoIP quality of experience.  But more than that is the idea that there is an entire infrastructure of phone lines – stretching from sea to shining sea and beyond – that connect the last mile to the local phone exchange. 

It seems like such a waste. 

But wait!  DSL service only uses the 25kHz and above part of the spectrum – 4kHz is reserved for voice.   If landlines are repurposed ONLY for data, with VoIP being another application on the all-data network, that could free up the 4kHz spectrum currently being used.  Maybe we can use that 0-4kHz band for broadband to rural homes – which can clearly get 4kHz data if they can get a phone call. 

This is especially important for rural broadband penetration.  The longer the distance to the exchange, the lower the quality of high bandwidth exchanges – this is why your DSL service gets slower the further you are from your local phone exchange.  But the 4kHz currently used for voice can travel much greater distances – it won’t be as fast as the DSL available in the cities but repurposing the 4kHz bandwidth from voice to data might make a huge difference to getting some minimal broadband to the most rural parts of the world.

Now, this won’t make a whole lot of difference to a person living in the city – DSL works by dividing that 25kHz-and-up portion of the spectrum into 4kHz chunks, each one connecting with a speed equivalent of a modem.  It is the multitude of these channels – hundreds in most cases – that makes DSL speed possible.  Repurposing the broadband of 0-25kHz would result in only six additional channels.  Assigning two for upload and four for download, you’d have speeds of around 14.4kBytes/s (or 115.9kbits/s) upload and 28.8kBytes/s (231.3kbits/s) download.  That’s not much of a speed boost. 

Still, if you’ve been plodding along on a “56.6k” modem, at speeds of 7.2kBytes/s, this would be like an oasis in the desert.  And what about those phone calls?  Well, if you make the same phone calls with VoIP that you were with the standard 0-4kHz landline, it would only take about 20.8kbits/s using the G.723.1 codec – that still leaves you with 80% of your broadband capacity when on the phone – and 100% of your broadband when you’re off it.  For someone whose only current Internet connectivity choice is a modem, currently getting 16% of a theoretical data capacity – and 0% when you’re on the phone – that’s a major improvement.   

What do you think?  It seems reasonable, but there might be a flaw in my math – I did only pass Calc I on my third try.  Let me know if you have something to share in our comments section.