Just a quick note: Ubuntu Hardy Heron (8.04) was released today. Look for heavy showers of BitTorrent and FTP traffic, as well as additional LAN traffic later in the day. “Heron” comes with increased virtualization support and is a “Long Term Support” release, which makes the platform significant for enterprise applications.

You may want to download one copy and put it on the LAN to keep traffic off the WAN connections.

In a few minutes, Jim Metzler of Ashton, Metzler, and Associates, will be delivering his keynote on the Next Generation NOC at NetQoS Symposium 2008 at Barton Creek Resort in Austin. Last week, we pre-recorded a podcast with Dr. Metzler regarding the speech he is about to give and what he means by a "next generation NOC."

He talks about the changing role of the NOC and moves in enterprises towards integrating what were once seperate stovepipe functions to focus on application delivery.

The podcast is below.

Today, in this podcast, we speak to Dr. Jim Metzler at Ashton, Metzler, and Associates regarding his handbook, "The Handbook of Application Delivery 2008" and his upcoming keynote speech a NetQoS Symposium 2008.

I’d make a better pun, but just about the only things that come to mind when I think of Denmark are Hamlet and the Jyllands-Posten Muhammad cartoons, and I can’t draw.  So, here goes:

Tis a fault to heaven,
    A fault against the dead, a fault to nature,
    To reason most absurd

-- King Claudius, Hamlet, ACT I Scene 2                  

In this case, a network performance problem lead to a network fault – a misconfigured “piece of network equipment” in caused “IBM’s network” to be “overwhelmed,” and that disrupted business at many Danish companies.  (The quotes are not for emphasis, irony, or skepticism – just to show that the WSJ story was very non-specific about what exactly went wrong.)

Find out the cause of this effect,
   Or rather say, the cause of this defect,
   For this effect defective comes by cause.

-- Lord Polonius, Hamlet. ACT II Scene 2.

Among the consequences: 188 people went without dairy delivery for a day, and bank patrons couldn’t use their ATM cards and had to visit a real-life teller.

For that last one, the International Edition of the Halsingin Sanomat, which I totally reserve as the name of my heavy metal band if I ever start one, was able to get more answers from Sampo Bank, one of the banks affected.

"It was a more extensive malfunction, which meant that the security codes of our cards did not reach their destinations. That is why the cards that needed verification did not work right", says Sampo's head of communications Hannu Vuola.

The end result is that Denmark’s Ministry of Foreign Affairs has said that many of the affected companies are looking for some sort of compensation from IBM for their trouble.  And for a country that is the “world’s most networked country,” a massive network outage is a major problem.

It is a custom
  More honour'd in the breach than the observance.
-- Hamlet, Hamlet, ACT I Scene 4

Kevin Davis, a senior consultant at NetQoS, will be presenting a few training sessions at Symposium about SuperAgent, the end-to-end response time module of the NetQoS Performance Center. This will include a training session about how to use time-based network metrics in troubleshooting.  He talks about his upcoming training session below.

In the session, I’m going to be covering the importance of using a time-based metric in troubleshooting, because end-users complain foremost about time.  For example, they’ll say “the application is running slow,” or they believe “the network is slow.”  To users, everything is based on time, that’s what they’re complaining about.  And they’re correct.

It’s very new to many people to think of performance in “time” although that may seem counterintuitive - because most people are used to reading utilization graphs.  With utilization graphs, however, we don’t know if 70 or 80 or 90 percent utilization is necessarily impacting the user experience.  I mean, we buy networking equipment, routers, switches, firewalls, servers, and we want them to be highly – or efficiently - utilized.  Seeing high utilization could indicate a problem – or it could just indicate that you haven’t over-purchased.  So you can have a link at 90% utilization or a router at ninety percent CPU utilization but you won’t know if that’s impacting the end-user without a time based metric.

It’s time-based data that tells you how the users are being impacted.  Sure, the utilization data – the interface utilization, memory utilization, I/O utilization, can often tell what is doing the impact.  But the time base shows you the degree of the impact – the real-world effect on end-users.  With a time-based instrument, such as NetQoS SuperAgent, you can find out where the delay increase is occurring, and whether it’s based in the network, server, or application. 

In fact, you can take a look at time-based data and make a determination very quickly as to which entity is creating the performance issue – the beautiful thing about SuperAgent, in particular, is that it trends by time 24/7, so not only can you determine how your important business applications are being impacted today, but you can go back and look at recurring patterns in performance issues.  You can see if today is worse than yesterday or last week or last month.

In the session, I’ll also be going over how to architect the data center for performance.  Placement of servers that participate in inter-architectures is critical for the health and performance of the application and indeed the data center.  We also talk about how different protocols, for example, Microsoft’s TCP/IP stack, can impact application performance by enhancing or degrading it. 

It’s important for servers that are serving the same application.  For example, a front-end Web server and a back-end Oracle database really should be on the same switch on the same VLAN.  That way they receive optimum service from the network.  If they do leave the switch, they’ll have to contend with bandwidth going up and down the switch links, and they’ll be switched and routed multiple times. 

Based on measurements from customer environments and from our own laboratories, when two servers are on different switches they can have up to 18 milliseconds delay between them.  If we think of that in the terms of network engineers of one millisecond per 100 miles, what in effect we’re doing when we put two different servers on different switches, or two different VLANs on the same switch, we’re making it look like those servers are 1800 miles apart – like one server is in Los Angeles and the other is in Memphis. 

Gerald Combs is the original author and lead developer of the open-source, multi-platform, Wireshark network protocol analyzer. Combs works for CACE Technologies – a company which makes products that compliment Wireshark.  Today he mostly takes care of the administrative parts of the project but still does development as well, and he controls the version numbers and release schedule.

After ten years of development, Wireshark finally reached the milestone of a 1.0 release.  We speak to Mr. Combs in an interview below: 

NPD: So what is Wireshark?

Combs: Wireshark is a network protocol analyzer.  It’s kind of a traditional analyzer in that it’s a GUI that has three panes, the top pane shows a list of the packets captured off the wire, the middle pane a detail of whatever packet you have selected, and the bottom page shows the actual hex output – the bytes in the actual output.

NPD: Why did you decided to build Wireshark?

Combs: Years ago, I worked at a small ISP in the Midwest, and unfortunately, they couldn’t get me a Sniffer, which was the standard analyzer at the time, and of the products out there that were available, none of them ran on the platforms we used at the ISP – namely Solaris and Linux.  So I decided to sit down one day and start writing my own analyzer. 

I did the first release in July of 1998, and soon after started getting a steady stream of contributions from a bunch of really smart people.  After that, we built up a pretty good following of users.  And right now, Wireshark is the world’s most popular network protocol analyzer. 

NPD: Why did you decide to open-source the project?

Combs: I’d used open source software for a long time at that point.  Before then, I worked at a university and we made a lot of use of open source software.  It just made sense to me.  I wanted to give back to the community and it just seemed like a good way to go.  As it turned out, it was a great way to go, because Wireshark is appealing for a lot of people who write code for it.

NPD: Why has it taken ten years to reach Version 1.0? 

Combs: I just wasn’t comfortable until recently putting out the 1.0 release.  I’ve known for years - shortly after we made the initial release, people started using it in production environments.  And some people had trepidation because it wasn’t 1.0 yet.  But for the most part, people just didn’t care about the version number and they used it wherever they wanted to and wherever they needed to. 

But for me there were a number of features that were crucial and missing until recently that prevented me from putting a 1.0 stamp on it.  Probably the last one, one of the main ones, was privileged operation on Linux – getting it so that you could capture as root but run the GUI as non-root user. 

NPD: Have people come up to you and told you about how Wireshark helped them out?

Combs: I get e-mails from time to time from people, saying that I’ve helped them out.  I have some former coworkers that have mentioned that.  It’s actually pretty encouraging. 

We get a huge amount of code each month.  Between each release, we have 200,000 and a million lines of changes.  That’s just changes.  The actual source code is about 1.5 million lines now.  That’s a bigger job than I can do individually.  And luckily there are a bunch of smart and talented people that can help me with that.

NPD: What was the greatest challenge in developing Wireshark?

Combs: The greatest challenge is just the day-to-day maintenance, keeping the project going.  But several years ago, we had an initial push of fixing security bugs a while back and it was a huge undertaking. I just remember spending several months doing nothing but fixing these security bugs, and it was a big chore.   We have a huge codebase now, and unfortunately we just don’t have the resources to audit that.  But we have a lot of automated processes in place like fuzzing and static analysis that help us find those bugs. 

I can’t say this enough: Thank you to all the Wireshark developers out there and the user team – this has just been a great journey and it’s one that I hope to continue. 

These are some of the notes sent to Tier two support from the help desk by a man who is referred to as “George,” on a Web site called: “The Chronicles of George”

[Name] is havening problems with getting on to network,shesays she gets nocked of the network.

[Name] is havening problems connectioning to the network

[Name] needs to dell servers  need to be installed

[Name] needed access microsoft network

she doesnt  have any off her driver install ,she said they aregone so when she went reinstall them back she recieved an error message that said”rp server is not availble”

he tried to install [program].but it said admin right ,he thinks as something with server.

he needs access  to the raz server

[Name] is not able to access her email throught the web base

[Name] needs access  to manger discusion datebase

[Name] needs permission to [Shared Network Drive]

[Name] is havening problems replicating his emails, he says gets and error that says he cannot find mail file server.

[Name] called and he is havening problems getting onto the network.

[Name] is recieving anerror that states  the server is not responding.

I had to type that out in Notepad because Word would automatically correct all the errors. 

So, why am I sharing this with you?  Because the help desk staff are, in far too many companies, the first people, besides the users, to know when there is degraded application performance.  And while 99.999% of help desk personnel are well versed in their field and their written language, sometimes you get George.  (And yes, the author points out that George is a native English speaker who is most likely not dyslexic.)

One of the reasons that it is so important to be proactive with network trouble prevention – having the guys in the NOC be the first to notice application performance problems – is because when you rely on the help desk, you’re hearing about the problem third-hand.  George illustrates the problem with having the end-user, who may not have the technical knowledge to adequately describe the problem to begin with, passing the information along to another person before it gets to the people who might be able to solve the problem.

by Patrick Ancipink
Director of Product Marketing, NetQoS

There’s been a lot of growth (and attendant hype) in technology areas like WAN optimization and application acceleration over the past few years, and for good reason. Anything that helps companies speed up and reduce the risk of strategic IT initiatives like consolidating data centers, turning up new branches or serving an increasingly mobile and scattered user community will be popular.

To help with cope with the increasing reliance on the WAN and keep latency in check, there are a dizzying array of vendors and products out there – but if you’re trying to determine precisely which techniques and technologies to implement for your specific needs, the array of vendors quickly goes from “dizzying” to “disorienting” and finally “nauseating.” 

Cisco’s been in this Tilt-a-Whirl™ of a market for a while (and NetQoS has been right there with them) and they’ve taken some big steps recently to provide a more holistic approach that centers on building an “application aware” network, rather than trying to highlight one type of implementation against another for a narrow set of capabilities.

NetQoS started working exclusively with Cisco closely to help customers evaluate, measure, and prove the effectiveness of WAN optimization and application acceleration deployments. As customers are moving from pilot phases into full production, the before/after measurements and comprehensive monitoring are critical to ensure customers are getting the benefits they intended and doing what they need to deliver application performance. 

To help get the word out, Cisco just launched a new section of their web site today that contains a wealth of information about, as they call it, “WAN and Application Optimization.” The downloadable presentation, Cisco WAN and Application Optimization Technical Overview Presentation, puts Cisco technologies (and complimentary ones, NetQoS included) into a useful context with a methodical approach and framework built around four steps: Profile and Baseline, Optimize, Evolve, and Operate. A whole Campbell’s Factory of Cisco alphabet soup technologies are included—WAAS, ACE, NBAR, Netflow, CBQoS, IP SLA, PfR—to show how they work in concert and what role they play in the bigger picture.

There’s also the Cisco WAN and Application Optimization Solution Guide , a very in-depth publication—like 227 pages deep—that is targeted for “technical personnel involved in the specification, design, and implementation of specific WAN and application optimization solutions.” We, here at NetQoS, are proud to have contributed several sections to book regarding the methodology and implementation of network performance monitoring for WAN optimization and application acceleration. 

(If you are looking for some lighter fare, the video on the site tells a nice story in about 6 minutes including an airshow, snowmobiles, windsurfers, and skydiving—interesting choices for demonstrating the criticality of serving video over the WAN.  Then again, some company somewhere has to make the recreational products, I suppose.)

Joel Trammell, CEO of NetQoS will be producing the welcoming address at NetQoS Symposium 2008. We asked him a few quick questions about what he’ll be talking about when Symposium starts April 20, 2008.

NPD: This year’s welcome address is called: "Why networks fail and why the role of the network engineer is secure." Could you tell me a little bit about why you chose that topic?

Trammell: Well, I think with fault management issues, people really understand the network up and down. It's very clear to them the value that keeping the network up provides. With performance, it's somewhat less clear sometimes, to people. What problems are people actually solving? What causes these performance issues? They tend to be more nebulous in nature than fault issues.

NPD: So when you say "Why networks fail," you mean "Why do networks fail to meet goals?"

Trammell: Yes, "why networks fail to perform," is really the title here.

NPD: How do networks fail to perform?

Trammell: Well often it's built around changes in the environment. There are a lot of things going on in the environment these days that add a great deal of complexity, whether it be the introduction of new applications such as VoIP, whether it be different uses of the network with folks expecting anytime anywhere access to the network, so therefore instead of just having people sitting in a building on a wired connection, you may have a wireless connection or at their homes, or at hotels, or wherever they're coming into contact with network services.

There are new technologies being deployed in the network, WAN optimization being one that has been particularly hot in the last few quarters. So all these changes introduce many opportunities to cause the network to behave in a way which is different from what the users have come to expect.

NPD: The other half of the topic is "Why the role of the network engineer is secure."

Trammell: So, yes, there's one school of thought out there in the industry that IT and particularly networking as an area will become a pure utility in the near term and that companies will no longer invest in networking, just like they no longer invest significantly in their own power distribution systems or their own power generation systems. I don't believe that's the case. I don't believe, when you think about the network - is the wild west. There are all kinds of applications being introduced, both intentionally and unintentionally on that network. Lots of technology changes going on - it's a very wild environment. It's not an environment that's conducive to a utility type approach. And therefore people with expertise in networking, and particularly performance of applications across that infrastructure will continue to find the field very lucrative.

NPD: One thing that gets me is that when people talk about the idea of the network becoming like a utility - they often bring up, "Well, people don't have their own power plants. But, they do have their own power plants - if you're a hospital, you have a power plant. If you have a data center that needs 100% uptime, you probably have at least a backup power plant.

Trammell: And people often choose to locate their facilities near where power supply is good and cheap, so even though it's a utility, geography doesn't have equal access to power at the same costs. So even power is not as pure a utility as a lot of homeowners may think of it, as when you get to use it on an industrial and commercial scale. And I think networking adds at least another layer of complexity to that.

NPD: There are significant changes of course; there is truth that Software as a Service does change a lot of things. Obviously you can't ignore that. But you still need a network to access SaaS apps.

Trammell: SaaS assumes, as sort of a priori knowledge and capability that you have this highly performing, ubiquitous network available to deliver SaaS. If anything, it makes the network even more important, because now, the only way to access the software is through the Wide Area Network.

Traffic shaping is not a tool of the devil, nor do we believe the solution to bandwidth problems is simply to provision more dark fiber and build more underground fiber optic lines. But as time has gone on, the issues around network neutrality have become more pronounced.

For example, Bell Canada has been throttling P2P service, much like Comcast in the United States. However, what makes this different is that Bell Canada is in a position much like AT&T – in that throttling the network on the backbone affects all the people – including people who are not Bell’s customers – along the line.

Worse still, Bell has been reselling the capacity to provide ADSL service to smaller ISPs without letting the services know that the bandwidth is throttled for certain applications. One of those smaller ISPs, Teksavvy, said: “We are not throttling anything and as far as I am aware will never throttle anyone. We don't believe in it.”– so the idea that Bell will leave them with no choice in the matter is a little worrisome. There isn’t much choice in the matter – the only other big broadband provider in Canada is Rogers Cable, which also throttles traffic.

There are arguments that “net neutrality” will be solved by the forces of the free market – that is, if one ISP throttles, they can go to their competitors. The problem is that, in this case, this is exactly what savvy customers were doing by moving from larger companies, like Bell and Rogers, to smaller companies like Teksavvy. From the consumer, it’s reducing their choice. For the small ISP, it’s could be considered downright anticompetitive, and the Canadian Association of Internet Providers applied for relief before the Canadian Radio-television and Telecommunications Commission that would require Bell Canada to cease and desist.

We contacted Rocky Gaudrault, CEO of Teksavvy Solutions, but because this was now a legal matter, he explained that he was unable to comment. It was clear that he is passionate about the issue, but Teksavvy’s staff keeps him from speaking out by supplying him with timbits and beer to keep his mouth and hands busy.

Particularly interesting is this comment by a Slashdotter – both Bell and Teksavvy charge on a “tiered and metered” basis – which pretty much cuts through the false choice between deep packet inspection and metered bandwidth; Bell has both. (One profanity-laden post implied that the only reason that Bell Canada did this was to coldly eliminate the most compelling competitive advantage that smaller ISPs had – Bell had throttled traffic, small ISPs didn’t.)

The upshot is that network neutrality concerns have been brought to Canada’s Parliament during Question Time. (link via Prof. Michael Geist at the University of Ottowa, who we hope to have an interview with on Monday.) It’s unsurprising because these matters do not just affect consumers but large enterprises as well - an unannounced and sudden change in the QoS policies of the backbone provider is exactly the type of thing that can foul up capacity planning, VoIP switchover, teleconferencing, etc. Especially worrisome are those technology companies who rely on some form or another of P2P traffic to help cut their bandwidth costs.

Deep packet inspection is a powerful tool, and used in the right hands, in the right way, it can help make QoS planning easier, can help streamline business critical applications, can provide overall better end-user response times, and may indeed be a great technological boon.

But we can’t see any benefit in this case for throttling the traffic of resold bandwidth, and for not disclosing the changes in advance. If businesses that control backbone traffic want to avoid governmental regulation, they need to show that they can be responsible with the power they have and use it in a manner which is neither anti-competitive nor deceptive to wholesale resellers and end-user customers.