Happy New Year, everyone.  Only six more years till hoverboards! 

At the beginning of the new year, there’s almost always a slew of tech news articles and editorial talking about how 2009 will be the year of X. 

To wit, it has been the year of “the Linux Desktop” every year since 1999.  (Personally, I think that 2008 was the year of the Linux Desktop, but that’s an article for a different day.)

But while predictions may provide some source of humor when they go awry, the paradox is that they should be taken seriously.

Part of the problem with making predictions is that if the result is positive, they can become self-fulfilling, negative, and they become self-avoiding.  Look at Y2K.  The world thinks nothing happened – however, if you were in IT during the late 1990s, you know that Y2K required a major overhaul, increased upgrades, etc.  Y2K wasn’t a disaster overhyped, it was a disaster avoided through massive hard work.  It also may have had no small part in the zenith of the tech boom of the 1990s, as companies bought new hardware before the usual refresh cycles. 

Network World is making one big prediction that I think does deserve some attention. 

The first is Steve Taylor and Larry Hettick’s prediction that 2009 will be the year of IP video. 

Our observations: We recall the days when public and enterprise networks were engineered first for voice and data second, but as data traffic demands grew, the engineering focus by necessity had to change to data first, voice second. We see the same evolution in network engineering focus as video demand grows to surpass data. We also note that while some enterprise IPV will be sent and received entirely across private networks (especially for telepresence), inter-company and business-to-consumer traffic will principally cross the Internet - so service providers will need to accommodate both consumer and enterprise video traffic in a way that does not compromise voice and data network integrity. And with the consumer market for Internet delivered commercial video also beginning to burgeon, the task of managing all the video traffic across the network cores will not be trivial.

I suppose that now would be a good time to mention that we’re probably going to be stepping up our Whiteboard Video series articles in the new year. 

In addition to teleconferencing, video’s simply a simple way to convey information to those who learn by both sight and sound.  It’s much easier to explain a concept when you can show them, rather than just telling them about it.  Maybe one new years resolution could be to check with your marketing and sales departments to see if they have any plans to put video online for customers, or with HR to see if there’s any important training videos going out on the intranet. 

The problem is that video, voice, and data all travel on the same pipe.  If one of those three monopolizes the pipe at the expense of the other two, it doesn’t matter how large or how small the pipe is.  Network monitoring and proactive management is sorely needed in a “triple pipe” environment.  Proper network management allows you to mitigate the worst problems with an oversubscribed line until you can get more bandwidth; but improper network management will cause problems no matter how much bandwidth you add. 

Ben Erwin concludes his three-part Whiteboard Series installment on how to manage QoS in your environment. In this episode, Ben shows you how to use NBAR in the NetQoS Performance Center to manage QoS policies in your environment. 

Below you’ll find the embedded video, now in widescreen YouTube HD. A low definition version can be found here.

Ben Erwin starts off a three-part Whiteboard Series installment on how to manage QoS in your environment. In this first episode, “Leveraging Cisco Tools: Using CBQoS & NetFlow to Manage QoS Policies in Your Environment” Ben goes from the Whiteboard to actual CBQoS monitoring in the NetQoS Performance Center, illustrating some of the problems that can occur with QoS, and what steps to take to resolve them.

Below you’ll find the embedded video, now in widescreen YouTube HD. (Yes, we are aware of the irony of telling you how to watch out for things like, say, excessive YouTube traffic, with an excessively large YouTube video.) A low definition version can be found here.

The Obama-Biden transition team promised last Monday, Dec. 8^th, that they would provide most policy documents from meetings with outside groups – i.e., lobbyists – would be posted on the Change.gov Web site.

By Wednesday, Dec. 10^th, this policy already saw some interesting results. David Kravets over at Wired’s Threat Level blog pointed out that the site has already published a paper detailing the requests of the MPAA’s lobbying organization, which include requesting filtering information from technology companies.

We’re not against the MPAA using the means available to protect their intellectual property concerns, but there are two problems with filtering: false positives, and performance degradation.

False positives are already a major problem with the content industry – back in 2003, the RIAA sent a cease and desist letter to Penn State University – they had confused work from Prof. Peter Usher at the Department of Astronomy and Astrophysics with that of Usher, the R&B pop singer.

This is also a recent problem; in October of 2007, Google launched a copyright filter for the YouTube Web site. It, too, has many false positives. For example, a fan production of the reality TV show “The Mole” was removed, presumably, because it was confused with the real thing by the filter. Judging from the production values of the fan-film, it’s very unlikely that a human censor would confuse the two.

(Fun fact I learned while researching this article: Andy Warhol made a “Batman” fan film back in 1964.)

Videos removed for copyright complaint – legitimately or not - have been catalogued (but not archived) at YouTomb, a project from MIT Free Culture.

But YouTube is one, privately operated Web site. Filtering the content as it is uploaded merely affects the time to publish, not the time to distribute. Additionally, videos can also be hosted on competing sites.

If one were to try to use filtering on the Internet as a whole, as the MPAA seems to be lobbying, it is likely that the results would be similar to the results of the tests run by the Australian government – where even the best of filters degraded network performance, and the better the filter was at avoiding false positives and false negatives, the more performance degraded. Even the best filter wasn’t very effective.

The lesson to learn from all of this is that too often, measures taken in the name of “computer security” – even if it’s to instill a false sense of security – can have serious impacts on network performance. For this reason, those in the enterprise responsible for making sure that networks remain secure and those responsible for making sure that applications remain responsive absolutely need to coordinate efforts.

President-Elect Barack Obama, recently put a new video on Change.gov, the official Web site of the office of the President-Elect. In the video, Obama is seated in the office of the President-Elect, sitting in the chair of the President-Elect in front of the desk of the President-Elect. And if I had to guess, he’s probably reading prepared notes from the teleprompter of the President-Elect into the YouTube camera of the President-Elect.

These YouTube videos aspire to function much like an online, 21^st century version of the FDR’s “Fireside Chats.” Coincidentally – or perhaps not - Roosevelt’s first Fireside chat, broadcast in 1933, was entitled, On the Bank Crisis. This was also the subject of Obama’s broadcast. And like FDR, Obama is proposing a federal employment program much like Roosevelt’s New Deal.

During the New Deal, the Civilian Conversation Corps, or CCC, was a work-relief program designed to keep people employed with a semi-steady paycheck. One of the main ways that the CCC employed young Americans was by putting them to work solving the environmental problems of that era – which mainly involved flood prevention, soil erosion prevention, wildland fire suppression, reforestation, et al. You can go to Wikipedia for the full details.

Similarly, Obama’s plan, (or the sketches of it outlined in the short YouTube video), calls for increased energy efficiency in national infrastructure. Obama didn’t mention it specifically, but it’s not a far guess to think that part of this process will be “greening” Federal IT – and that usually means server consolidation (perhaps aided with WAN Optimization) and virtualization – to do more with less of a power draw – as well as putting the network to new uses, such as teleconferencing instead of spending money on airfare. Both of these will require network monitoring and adept management.

What Obama did specifically mention:

“It is unacceptable that the United States ranks 15^th in the world in broadband adoption.”

Hmm. Sounds familiar. Sounds like something we’d say. (Mr. President-Elect, are you subscribed to our RSS feed?)

The CCC, and similar program WPA, also focused on building and improving the infrastructure of the country – broadband improvements, of course, are improving digital infrastructure. (Of note – American broadband improvements must be done on American soil, and can therefore be almost “outsource-proof.”)

Also of interest was Obama’s pledge that:

“In addition to connecting our libraries and schools to the internet, we must also ensure that our hospitals are connected to each other through the internet. That is why the economic recovery plan I’m proposing will help modernize our health care system – and that won’t just save jobs, it will save lives. We will make sure that every doctor’s office and hospital in this country is using cutting edge technology and electronic medical records so that we can cut red tape, prevent medical mistakes, and help save billions of dollars each year.”

We at NetQoS have a little bit of familiarity with how difficult medical networking needs can be. In Volume 3 of Performance Edge Journal, we published a case study of OSF Healthcare, which is a network of multiple acute care, long-term care, and college of nursing facilities and also a primary care physician network.

One of the applications that OSF wanted to implement included Picture Archival and Communications System (PACS) – in order to send and manage very large cardiac images. The application had very slow response times, and you know, when something’s wrong with your ticker, time’s of the essence.

Anyway, NetQoS considers this one of our big success stories. Through SuperAgent, they found that the delay was caused by excessive server response times, and using ReporterAnalyzer, they were able to figure out that some cardiac images were being sent to different sites across the WAN and not stored locally, slowing down retrieval times considerably, taking up large amounts of bandwidth unnecessarily.

Whether or not Obama’s plan will actually work, at least he seems to be aware of the real need for network performance in the healthcare industry.

At the risk of waxing political; Obama talking about technology policy means that the next four years (at least) will be interesting times for geeks. Under the Clinton and Bush administrations, discussions of what should be done about networking issues were mostly confined to – well, to blogs like this. (And even then, it was mostly confined to Slashdot). But, whether or not you agree with him, Obama seems to be using the power of his office – or of his future office – to call attention to the problems that up to now, only us techies were paying attention to.

By David Oliver

We did have the opportunity to do this blog post as a video recording and put it on YouTube, but we realized that, ironically, as the post is all about how companies use NetFlow to track YouTube, because YouTube can, in many cases, suck down bandwidth, it was probably best just to write this out in text.

As we mentioned a week ago, YouTube is now supporting high definition content, with a high bandwidth to match.  Now, I've done a little bit of research into how YouTube actually works.   So I thought I’d explain to all those companies out who don’t yet have their own solutions some ideas about how to track and manage YouTube and other streaming media data – as well as give users out there an idea of exactly how companies can track your YouTube usage at work. 

Anyway, when you make a request for a video on YouTube, you are directed to YouTube’s servers via one of four IP addresses that are easily found on Google or other search engines.  From there you're going to be relayed to the Limelight network, which will actually feed you the video in the flash-based player.  You can see the flows to and from that initial IP address for the HTTP GET of that video. 

There are many solutions for providing visibility into traffic on the network by looking at the Cisco NetFlow data (which is already on most Cisco routers).  I’m going to refer to NetQoS’s own solution,  ReporterAnalyzer, when I talk about tracking NetFlow data. 

What we can do with ReporterAnalyzer is monitor the Internet-facing link, and create and use custom reports looking for YouTube’s specific IP addresses.  If you see a substantial amount of data being transferred,  that's a good marker of seeing that YouTube video traffic. 

You can rely on those custom reports and run them anytime you want, but companies can also monitor YouTube in real-time.  By mapping HTTP Port 80 traffic that involves one of YouTube's IP addresses to some other ephemeral port, (and naming it something catchy, like "YouTube,") it'll actually show up as it's own protocol in both real time reporting, as well as flow forensics.    You could use that data to create customer reports, to get a comprehensive list of users, and to sort YouTube use by volume. 

The other thing you can do is use analyses to know when YouTube traffic accounts for more than, say, 10% of any of my links' traffic.  Then it will go through on a link-by-link basis and tell you about violations, helping you further localize the source of that traffic. You can also configure it to alert you when and only when YouTube traffic on a particular link passes a threshold that you set. 

(The other option is to try to block it entirely, but that's an engineering nightmare.  Any employee smart enough to provide good value to a company - particularly a high tech company - will likely be smart enough to know how to circumvent blocks through proxies and other means.)

Custom reports to find correct addresses and to localize YouTube traffic may take a couple minutes.  The entire real-time application mapping process takes maybe another 15 minutes.  I can be showing real-time data specific to YouTube traffic just a few minutes after configuration of application mapping.  (If your boss asks in the morning for something to track YouTube usage, the company can get YouTube tracking up and running by that afternoon - if the boss just wants some a quick snapshot of the current YouTube traffic volume, it could take as little as five minutes through custom reports.)

Of course, this isn't limited to YouTube.  You can use similar methods and techniques to find and track streaming audio feeds, other video sites, etc. Any TCP flow is going to create some sort of NetFlow data.  Based on the source or destination address, you can localize that.  So as long as ReporterAnalyzer has visibility of that destination address, they can report on it.  As you know, there are a multitude of media based streaming sites, all of which are going to have their own IP address range, which you can find pretty easily.  You can then further localize and label them so that when you pull up reports, they're already differentiated from other traffic.

While YouTube is great, we’ve found that YouTube traffic congesting corporate networks is a common issue. For any company, WAN links are a finite resource and need to be managed.  It's something that's a concern because you're sizing your network around capacity needs for the business.  YouTube is (usually) non-business traffic, but it's going to share that limited resource.  The more you share a resource, the less is available for the requirements you originally scoped it for.  At NetQoS, we’ve found YouTube traffic congesting corporate networks is a common issue.

Boy, what a difference a year makes.

“Black Friday” usually refers to the day after Thanksgiving, when retailers, both online and offline, started getting rushes of orders in order to fulfill Christmas demand. But unless you’re a Wall Street firm, for whom Christmas has come early, you’re probably cutting back on expenditures this holiday season.

Now, it’s probably referring to any Friday that you re-read your quarterly 401(k) or 529 statement.

Still, whether or not people –spend- more online this holiday season, they’ll probably be making a similar number of transactions – that is, Hershey bars instead of Godiva chocolate, Playstation 2s instead of Wiis, Go-Bots instead of Transformers…

And with every dollar counting, the one thing that retailers and suppliers can’t afford on Black Friday and Cyber Monday this year are performance slowdowns like the ones that hit Costco, Victoria’s Secret, Lowe's, and Macy's last year.

Additionally, even if you aren’t a retailer, Cyber Monday typically sends some Web traffic spikes over company networks as employees use the high speed connections work provides in order to make their purchases.

In either case, you can analyze network traffic flows to identify what traffic is mission critical, what is mission irrelevant, and what is mission impossible. After quantifying the impact of certain types of traffic on network performance, you can then implement quality of service policies to ensure that business critical apps have priority access to network resources.

We know that on Friday and next Monday, there will be a higher than normal volume of Internet traffic.  The trick is finding out how much of an impact it will have and preventing it from impacting application performance.

Peter Sevcik and Rebecca Wetzel of consulting firm NetForecast recently came out with an article in Network World regarding some results from a survey of 300 companies which asked about application performance management. If you’re a regular reader of this blog, you’ll probably have already guessed that those survey results underscore a point we’ve been saying all along.

The article, entitled, “Four steps to application nirvana” makes a number of very good points points out that implementing a best-practices strategy leads to better application performance.

The survey results show extremely positive correlations between best practices benchmark scores and actual application performance delivered to the business.

On the whole, enterprises with excellent best practices deliver 100% better results to their users than those with poor practices.

Here's where the rubber meets the road. Our survey results show that best practices exert their most dramatic effect on improving the time it takes enterprises to solve problems, with a 338% score improvement in problem-resolution time among those with best practices compared with their poorer-performing counterparts. …Those with the top best practices scores were more than twice as likely (144%) as those with poor scores to discover problems through systems vs. learning about them from users — and they were twice as likely (93%) to favorably assess the overall response times for their important applications.

In the article, Sevcik and Wetzel describe the four best practices as “Understand, measure, report, and link.” Of course, that doesn’t make a whole lot of sense out of context, so they provide 12 ways to apply the four best practices. Which really makes it more like 12 best practices to me, but I’m not complaining, since it seems to work.

Another thing we really like about this survey: they ask what’s going wrong, as well as going right.

Finally, we asked enterprises to identify impediments to improving application performance. Insufficient cross-group collaboration, insufficient manpower, and lack of proper tools tie for the top of this year’s list of impediments with nearly 50% of respondents mentioning them.

Ahem.

More seriously, Sevcik and Wetzel – and isn’t that an awesome name for a 1920s vaudeville act? [Edit: I’ve been informed that I’ve been repeating my zingers…]– have put out a blog called “App Performance View” on Network World’s site and asked for user feedback on vendor products – including NetQoS Performance Center.

According to Inside Higher Ed, one of the provisions of the Higher Education Act, which was passed last August mandates for colleges to police their network for illegally copied copyrighted works.  And according with a survey conducted by the Campus Computing Project, this will cost colleges $350,000 to $500,000 a year out-of-pocket to comply.

Colleges are “required to consider the use of technology-based deterrents” to prevent or deter copyright-infringement on peer-to-peer networks.  These can include traffic monitoring and packet shaping.

There are a number of problems with this plan:

First, the record labels want the colleges to pay money because some of their students are violating record label copyright.  The colleges are stuck in a bad place; because the government has mandated it.  It’s highly likely that these measures will be ineffective, but even if they are effective, the copyright infringers will just move off campus and get residential broadband, leaving all the students paying higher tuition and residency bills because the colleges have to pay for the ineffective enforcement somehow. 

So, who benefits from this law?  The labels?  This won’t stop one copyright infringer, and I think they know that.  What it does do is pass on the costs of ineffective piracy countermeasures from the labels to the colleges?  The colleges don’t get anything – nor do the students, pirates or otherwise. 

Nobody benefits, and most people suffer.  This is a crappy law. 

Additionally, there are significant political and academic freedom issues in requiring a network to monitor the traffic for particular activity.  This is especially discouraging when you consider that the rate of false positives is going to be extremely high: there is no traffic monitoring solution that is smart enough to tell whether a particular MP3 file is legal or illegal.   Any activity that blocks illegal content can easily break legal ones: fair use of the work, public domain, creative-commons, or downloaded by the copyright holder.  (Recently, a smaller record label had its site pulled because it had hosted copyrighted MP3 files – its own.)

So, in short, it’s stupid, ineffective, and has potential unpleasant side effects. How do you best comply with a law that’s so dumb that it borders on ridiculousness?

The only way to fight stupidity is by being clever – find solutions that won’t cost hundreds of thousands of dollars and which do not harm the ability of students, faculty, and alumni to advance the academic mission of the school.  For example, enabling and properly using Cisco Netflow information to make intelligent decisions about QoS policies.  (Labs and offices get more leeway than student residences, local proxies of popular legal BitTorrent files – like WoW patches and Ubuntu releases - on the LAN.) 

Considering that the provisions in the law (and I am not a lawyer, this is not legal advice, and even if I was a lawyer, I probably wouldn’t be a good one anyway) basically amount to: “Do something” about P2P copyright infringement, it’s probably best just to use QoS policies to throttle down (without cutting off) that traffic which looks suspicious.  But err on the side of assuming that the use is legal and for the academic mission. Maybe set up a filter that only goes into effect if the protocol is BitTorrent, and the content is a video file, and the seeders and peers aren’t all from IP addresses that correspond to astrophysics laboratories. 

You could also make use of anomaly detection software to track spikes in MP3 traffic – that is, of course, assuming that the downloading of music files by the 18-25 demographic is in any way considered “anomalous” rather than “status quo…ulous…” Something like that.

EMA analyst Dennis Drogseth had a column in Network World yesterday talking about end-to-end application management. In it, he had this to say:

You might believe, and with some real justification, that the term “end to end” is only used by vendors who custom-fit the definition to the scope of their particular product.

Does “end-to-end” application management, for instance, include the mainframe? You bet it does if you’re a vendor that manages the mainframe environment! Does it include capturing the end user experience at the end station, desktop, or mobile device? Once again, the answer is a definitive “yes” if you’re a vendor that has strong QoE (Quality of Experience) roots. Or how about insights into the code and design of the application itself? If you’re one of the few vendors that does this, you’re proud of it and wouldn’t have it any other way!

And this concerned me because, if you do a google search for: [site:networkperformancedaily.com “end-to-end”], you get 122 results. The phrase, “end-to-end” appears in a little more than 1 in 5 posts we’ve made to this blog.

So, what do we mean by “end-to-end?”  We’re usually using the phrase in connection with network response times and the end-user experience at the end station; NetQoS is a “vendor that has strong QoE roots.”

Now, we do have some insight into the code and design of the application.  But that isn’t the focus of our tools; the focus is to tell you whether the problem is in the network, server, or application, and if it’s in the application, give you a good idea of where to start your investigation.  (For example, an application that is slow due to unnecessary round-trip transactions behaves differently from an application that is slow due to a memory leak on the server where it is being run.) 

Drogseth is right when he says that no one vendor is optimized to do it all.  In the future, there could be, but then you run into the quality vs. quantity problem.  Is it better to do it all adequately or to do a few things extremely well?

EMA defined five major technology spheres, and last June, they polled more than 400 respondents to find out which of them they believed “most critical to end-to-end application management in 2008.”  The answer was “Network Application Management,” focusing on application flows and end-to-end (as we define it) transaction capabilities. 

For more information on this, I recommend you read the original article up at Network World.  Additionally, Drogseth promises to follow-up in his next two columns.