Ben Erwin starts off a three-part Whiteboard Series installment on how to manage QoS in your environment. In this first episode, “Leveraging Cisco Tools: Using CBQoS & NetFlow to Manage QoS Policies in Your Environment” Ben goes from the Whiteboard to actual CBQoS monitoring in the NetQoS Performance Center, illustrating some of the problems that can occur with QoS, and what steps to take to resolve them.

Below you’ll find the embedded video, now in widescreen YouTube HD. (Yes, we are aware of the irony of telling you how to watch out for things like, say, excessive YouTube traffic, with an excessively large YouTube video.) A low definition version can be found here.

President-Elect Barack Obama, recently put a new video on Change.gov, the official Web site of the office of the President-Elect. In the video, Obama is seated in the office of the President-Elect, sitting in the chair of the President-Elect in front of the desk of the President-Elect. And if I had to guess, he’s probably reading prepared notes from the teleprompter of the President-Elect into the YouTube camera of the President-Elect.

These YouTube videos aspire to function much like an online, 21^st century version of the FDR’s “Fireside Chats.” Coincidentally – or perhaps not - Roosevelt’s first Fireside chat, broadcast in 1933, was entitled, On the Bank Crisis. This was also the subject of Obama’s broadcast. And like FDR, Obama is proposing a federal employment program much like Roosevelt’s New Deal.

During the New Deal, the Civilian Conversation Corps, or CCC, was a work-relief program designed to keep people employed with a semi-steady paycheck. One of the main ways that the CCC employed young Americans was by putting them to work solving the environmental problems of that era – which mainly involved flood prevention, soil erosion prevention, wildland fire suppression, reforestation, et al. You can go to Wikipedia for the full details.

Similarly, Obama’s plan, (or the sketches of it outlined in the short YouTube video), calls for increased energy efficiency in national infrastructure. Obama didn’t mention it specifically, but it’s not a far guess to think that part of this process will be “greening” Federal IT – and that usually means server consolidation (perhaps aided with WAN Optimization) and virtualization – to do more with less of a power draw – as well as putting the network to new uses, such as teleconferencing instead of spending money on airfare. Both of these will require network monitoring and adept management.

What Obama did specifically mention:

“It is unacceptable that the United States ranks 15^th in the world in broadband adoption.”

Hmm. Sounds familiar. Sounds like something we’d say. (Mr. President-Elect, are you subscribed to our RSS feed?)

The CCC, and similar program WPA, also focused on building and improving the infrastructure of the country – broadband improvements, of course, are improving digital infrastructure. (Of note – American broadband improvements must be done on American soil, and can therefore be almost “outsource-proof.”)

Also of interest was Obama’s pledge that:

“In addition to connecting our libraries and schools to the internet, we must also ensure that our hospitals are connected to each other through the internet. That is why the economic recovery plan I’m proposing will help modernize our health care system – and that won’t just save jobs, it will save lives. We will make sure that every doctor’s office and hospital in this country is using cutting edge technology and electronic medical records so that we can cut red tape, prevent medical mistakes, and help save billions of dollars each year.”

We at NetQoS have a little bit of familiarity with how difficult medical networking needs can be. In Volume 3 of Performance Edge Journal, we published a case study of OSF Healthcare, which is a network of multiple acute care, long-term care, and college of nursing facilities and also a primary care physician network.

One of the applications that OSF wanted to implement included Picture Archival and Communications System (PACS) – in order to send and manage very large cardiac images. The application had very slow response times, and you know, when something’s wrong with your ticker, time’s of the essence.

Anyway, NetQoS considers this one of our big success stories. Through SuperAgent, they found that the delay was caused by excessive server response times, and using ReporterAnalyzer, they were able to figure out that some cardiac images were being sent to different sites across the WAN and not stored locally, slowing down retrieval times considerably, taking up large amounts of bandwidth unnecessarily.

Whether or not Obama’s plan will actually work, at least he seems to be aware of the real need for network performance in the healthcare industry.

At the risk of waxing political; Obama talking about technology policy means that the next four years (at least) will be interesting times for geeks. Under the Clinton and Bush administrations, discussions of what should be done about networking issues were mostly confined to – well, to blogs like this. (And even then, it was mostly confined to Slashdot). But, whether or not you agree with him, Obama seems to be using the power of his office – or of his future office – to call attention to the problems that up to now, only us techies were paying attention to.

By David Oliver

We did have the opportunity to do this blog post as a video recording and put it on YouTube, but we realized that, ironically, as the post is all about how companies use NetFlow to track YouTube, because YouTube can, in many cases, suck down bandwidth, it was probably best just to write this out in text.

As we mentioned a week ago, YouTube is now supporting high definition content, with a high bandwidth to match.  Now, I've done a little bit of research into how YouTube actually works.   So I thought I’d explain to all those companies out who don’t yet have their own solutions some ideas about how to track and manage YouTube and other streaming media data – as well as give users out there an idea of exactly how companies can track your YouTube usage at work. 

Anyway, when you make a request for a video on YouTube, you are directed to YouTube’s servers via one of four IP addresses that are easily found on Google or other search engines.  From there you're going to be relayed to the Limelight network, which will actually feed you the video in the flash-based player.  You can see the flows to and from that initial IP address for the HTTP GET of that video. 

There are many solutions for providing visibility into traffic on the network by looking at the Cisco NetFlow data (which is already on most Cisco routers).  I’m going to refer to NetQoS’s own solution,  ReporterAnalyzer, when I talk about tracking NetFlow data. 

What we can do with ReporterAnalyzer is monitor the Internet-facing link, and create and use custom reports looking for YouTube’s specific IP addresses.  If you see a substantial amount of data being transferred,  that's a good marker of seeing that YouTube video traffic. 

You can rely on those custom reports and run them anytime you want, but companies can also monitor YouTube in real-time.  By mapping HTTP Port 80 traffic that involves one of YouTube's IP addresses to some other ephemeral port, (and naming it something catchy, like "YouTube,") it'll actually show up as it's own protocol in both real time reporting, as well as flow forensics.    You could use that data to create customer reports, to get a comprehensive list of users, and to sort YouTube use by volume. 

The other thing you can do is use analyses to know when YouTube traffic accounts for more than, say, 10% of any of my links' traffic.  Then it will go through on a link-by-link basis and tell you about violations, helping you further localize the source of that traffic. You can also configure it to alert you when and only when YouTube traffic on a particular link passes a threshold that you set. 

(The other option is to try to block it entirely, but that's an engineering nightmare.  Any employee smart enough to provide good value to a company - particularly a high tech company - will likely be smart enough to know how to circumvent blocks through proxies and other means.)

Custom reports to find correct addresses and to localize YouTube traffic may take a couple minutes.  The entire real-time application mapping process takes maybe another 15 minutes.  I can be showing real-time data specific to YouTube traffic just a few minutes after configuration of application mapping.  (If your boss asks in the morning for something to track YouTube usage, the company can get YouTube tracking up and running by that afternoon - if the boss just wants some a quick snapshot of the current YouTube traffic volume, it could take as little as five minutes through custom reports.)

Of course, this isn't limited to YouTube.  You can use similar methods and techniques to find and track streaming audio feeds, other video sites, etc. Any TCP flow is going to create some sort of NetFlow data.  Based on the source or destination address, you can localize that.  So as long as ReporterAnalyzer has visibility of that destination address, they can report on it.  As you know, there are a multitude of media based streaming sites, all of which are going to have their own IP address range, which you can find pretty easily.  You can then further localize and label them so that when you pull up reports, they're already differentiated from other traffic.

While YouTube is great, we’ve found that YouTube traffic congesting corporate networks is a common issue. For any company, WAN links are a finite resource and need to be managed.  It's something that's a concern because you're sizing your network around capacity needs for the business.  YouTube is (usually) non-business traffic, but it's going to share that limited resource.  The more you share a resource, the less is available for the requirements you originally scoped it for.  At NetQoS, we’ve found YouTube traffic congesting corporate networks is a common issue.

By Brian Boyko
Age 29

I’ve just come back from vacation. I’ve been in New Zealand for the past two weeks. (Did you miss me? I missed you.) Anyway, after the stress of the U.S. elections, (I am a political animal*) I needed it, badly. 

I headed to New Zealand, mostly because I’m obsessed with the country’s multi-party electoral system of proportional representation.  This is especially important in New Zealand, where voting for the “lesser of two evils” invariably means casting a ballot for Saruman.  Also, I really, really wanted to try Zorbing.

Zorbing is basically crawling into a giant, air-cushioned hamster ball, and being rolled down a hill.  It’s just like being in an XKCD comic!

Here’s some pictures and video:

Inside the Zorb from Brian Boyko on Vimeo.

But, of course, as the adrenaline wore off and I reflected on the experience, I realized that, despite all the fun, I would have to start back at work today, so I did what I always do in these situations: Take something totally unrelated to network performance, and relate it to network performance.

For example, the Zorb in Rotorua has two tracks, both of which start from the same point, and both of which come to a stop at the bottom of the same hill; but I only had the time to go down one before the last bus of the day left.  Wanting to extend my fun as long as possible, instead of choosing the straight, downhill, fast track, I chose the bumpy, zig-zag one.  This track was more circuitous and involved multiple hops.  More hops = longer time.  In a network context, this delay is called “serialization delay.” In a zorb, this delay is called “WHEEEEEEEEEEE!”

Okay, it’s a lame comparison, but give me a second to get up to speed – I just got home after traveling for 34 straight hours, including 16 hours of layover and 18 hours of flight. 

The other thing about New Zealand is that everyone there complained about the speed of Internet access – there’s really nothing to do about it, as New Zealand is extremely remote to the rest of the world.  Even Australia is 1,000 miles away, and the alternatives to undersea cabling is satellite communication.  There are bound to be latency issues due to propagation (speed-of-light) delay.

Anyway, I’m tanned, rested, and ready.  We’ll finish up Joel’s series and post some other stuff in the following days.  Thanks for sticking with us. 

* some sort of flightless bird, most probably.

This is interesting; I actually had a conversation recently with my boss, in which I argued that the elections may have an impact on network performance so much as who is elected will help determine U.S. telecommunications policy, but otherwise… well, there wasn’t much to write about on the blog.

Apparently, I’m wrong. Because Slashdot just posted “Watching Tonight’s Presidential Debate Online,” and Slashdot is the IT blog. Maybe I made a poor judgment call – especially after hearing from poster ObsessiveMathsFreak:

The US presidential elections are actually very important. I see Slashdotters posting comments to the effect that both parties are equally bad and it doesn't matter which way you vote and excuses, excuses, excuses. I can tell you from the point of view of someone who is very much affected by the results of your national elections, this is a pretty depressing thing to hear. It's clear to anyone who has half a clue that there are very wide and deep differences between the two main candidates, and it's quite irritating to find out just how flippantly many Americans go about voting, or not voting, for their president.

Your election affects me. It affects people around me. My nation's economy, policies, laws, and culture, yes culture, are significantly affected by your selection of a president, through his administration's policies. This is "Stuff That Matters" to me.

The other important thing is that the original submitter prefaced his article “For those of us that no longer have a television…”

This is important because for whatever reason, geeks are early adopters. It’s geeks who first bought GPS devices; now they’re mainstream. It’s geeks who were first on the Internet. Now it’s everyone. It was geeks who first started blogging, geeks who first started eco-driving, geeks who first started “podcasting and vodcasting” before iTunes and YouTube brought it to the masses. And it was geeks who were rolling dice and pretending to slay dragons way before World of Warcraft.

If geeks have learned that they can eschew their televisions completely – not just the cable but the actual box – then where are they getting their audiovisual media from?

And that’s where we get into network performance – because there’s the possibility of hitting that tipping point where most people will be getting their audiovisual content via the Internet.

What’s strange is that this goes back to Clay Shirky’s thesis that we’re moving through a shift where the cognitive surplus created by the move to the forty hour workweek is no longer masked by television – that is, that we now have other things to do with our time. Television is a technology that essentially is designed for the person to sit down and “see what’s on.” Internet video is designed for two types of people: those who know what they want to watch and when they want to watch it, and those who treat video media as any other media, following hyperlinks where they may lead – if it’s text, graphics, audio, or video, so be it, so long as they get the information.

It is not enough to dismiss all video content as “recreational network traffic” because eventually we will start to communicate more and more with multiple media as costs go down and we all become more familiar with producing and sharing. This means that networks have to be able to handle the traffic that video will require.

In the meantime, enjoy the debate – online, or on the air.

I wanted to name this post “Things to view in Denver: Talking heads” but no one around the office got the reference.

Well, if you were worried that Obama’s VP announcement would overwhelm your network, you were spared. The announcement came in the middle of the night – 3 a.m., in fact – on a weekend. But convention season is starting, and that’s a whole additional set of worries.

At 5 p.m. EST today, the Democratic National Convention will begin, and with it, streaming video from multiple news networks and the DNC itself, which, like NBC’s Olympics coverage, uses Silverlight to project a “high-definition” (480p?) image.

Unlike the Olympics however, there’s bound to be some event – a protest that goes wrong, a verbal gaffe, a moving speech – that becomes a viral video. The reason I’m pretty sure about this is that it’s in the interest of both political parties that something goes viral – something good for the Democrats, something embarrassing for the Republicans. Politicians will find a compelling video of an unplanned, sincere, candid, spontaneous moment, even if they have to manufacture one.

There’s bound to be online coverage from the major TV networks as well.

And then after that, the Republican National Convention – with streaming through Ustream.tv - is next week. And McCain is yet to announce his vice presidential nominee.

Those interested in Obama and McCain’s technology policies will find this article at Ars Technica interesting, while C|Net has technology policy information for Biden.

The one thing I hate more than anything else is seeing people get the blame for something that they didn’t do because the people in charge are ignorant about technology. No, wait. Paper cuts. I hate paper cuts more than anything else. But that whole “travesty of justice” thing – very close #2.

This (possibly true, possibly not, definitely plausible) story from The Daily WTF had me ticked off. “Cam” – apparently a pseudonym - had set up a Web cam so he could prove to his bosses that he was working from home instead of just saying he was working from home. But on that day….

During a quick lunch break, Cam got a panicked call from his boss's boss, Ron. "Cam, do you still have your webcam on?"

"Yeah, wh-"

"Turn it off. NOW," he said in all caps over the phone.

"Uh, ok." Cam flicked the switch on the webcam off. "So, why exactly is it so urgen-"

"Can'ttalknowbigproblems-" *click*…

See, it seems that there was a brief but major hiccup in a router somewhere between the bank's data center and their T3 provider, causing a dramatic slowdown in outbound network performance, which rippled out into hundreds of branches and affecting thousands of online banking customers. In the troubleshooting process, the lead network engineer caught wind that Cam had been "streaming live video" over the network, and was going to tell! He complained loudly to Ron that Cam had caused the issues and lost some revenues for the bank in the process. Adding to this theory was the fact that the issue had apparently resolved itself close to the time that Cam turned off his webcam.

One week later, Cam is sitting with his boss Joel to discuss the issue. "Cam, I'm going to need you to sign this disciplinary action report before we file it with HR," Joel said weakly.

Appealing to reason, Cam began, "Joel, you know exactly what happened. You know that all that was coming across the network was a static web page with a new image every so often. I never had more than five HTTP sessions at a time. It would take thousands, if not hundreds of thousands of simultaneous users accessing my web site at the same time to consume the bandwidth that it says I consumed on this report."

"I know," he said as his expression sank. Clearly, he'd fought for Cam and been overruled.

"Besides that," Cam continued, "I'm hosting my site at my house. My upstream connection is capped at 360 kbps. There's literally no physical way that anything I did from my house could even make a dent in our massive T3 lines, even if my upstream connection was 100% saturated!"

"I know," Joel said as his face slipped into his hands. At this point, it dawned on Cam that he was lucky that all that was happening to him was a writeup. It sounded as though upper management would prefer to see him hanged. Still, it was absolutely unfair that he'd be made to take the fall.

"Furthermore," Cam pressed, "what about our QoS policies? Surely internal users browsing external web sites have lower priority than-"

"I know," Joel said again. "Look, I've fought them on this. You know I trust you, and that I know you wouldn't ever — that you couldn't ever — do something like this. I'm saying this as a friend; you're better off just signing this. It's not just you; management is pissed at me now, too. It's not fair, but it's how it is."

In the end, there wasn't much Cam could have done.

Of course, the network engineer who latched onto the “streaming video” theory should have gotten the blame for misdiagnosing the problem using the same kind of “If she weighs the same as a duck, she’s made out of wood and therefore a witch!” logic that can destroy the best laid plans of IT.

Maybe, if the moment the router started having huge performance problems, there was some sort of alert delivered to the network engineer – one detailing the problem, how bad it was, and where it was originating from, that might have helped. Someone might want to look into making something like that. Or – or, bear with me - if there was a way to look at the traffic patterns and Netflow data to see exactly how much bandwidth the Webcam was taking, providing exculpatory evidence for Cam, that might be nice. Someone should get on that.

In the meantime, I’ve just set up my own Network Performance Daily webcam to give you an idea of what my job is like. Hope you enjoy it!

John Mao, product manager at NetQoS, quickly explains the impact of WAN Optimization on Cisco IOS NetFlow/IPFIX information gathering in a short video, as part of our "Whiteboard Series."

If you have questions about the video, please leave a comment below and we'll do our best to answer them.
--------------
More information:

On WAN Optimization:
- NetQoS and Cisco Webinar: Evaluating WAN Optimization Options and Quantifying the Results

On NetFlow Monitoring :

Also in our Whiteboard Series:
- The impact of WAN Optimization on TCP Applications

With John Mao:
- Six Tips for Improving Network Visibility and Performance Using Cisco IOS NetFlow Data

Ben Erwin, product manager at NetQoS, quickly explains the impact of WAN Optimization on TCP-based Applications in a short video, kicking off our "Whiteboard Series."

If you have questions about the video, please leave a comment below and we'll do our best to answer them.
--------------
More information:

On WAN Optimization:
- NetQoS and Cisco Webinar: Evaluating WAN Optimization Options and Quantifying the Results

Also in our Whiteboard Series:
- The impact of WAN Optimization on NetFlow/IPFIX measurements

With Ben Erwin:
- Third Party Integration and UI in the Enterprise