NetworksFirst.com has recently created an online “Impact of Network Downtime” calculator, which you can use to estimate how much money it would cost if your network went down.  It makes a compelling case for fault management and worrying about outages. 

However, the cost of poor application performance is harder to quantify – or at least, requires more sophisticated tools and data - than the cost of fault.  That may be the reason that many companies still consider fault management, and not performance management, to be the core responsibility of the IT team. Our most recent research conducted with Ashton, Metzler & Associates bears this out:

Fifty percent of respondents indicated that they measure and report on the mean time to repair (MTTR) for a network or application outage. However, only thirty percent confirmed they actually measure and report on the MTTR for degraded application performance, revealing a continuing legacy of fault and availability management over performance management.

As technology has improved, fault performance problems have, for the most part, been solved.  It’s no longer a distinguishing feature for a network service provider to promise 99.999% uptime.  The next big challenge is maintaining good performance throughout the network. 

But in many ways, it’s a hard sell, because unlike a fault cost calculator, it’s difficult to show you exactly why you need performance management tools until you have the more nuanced calculation of what poor performance costs your business. What’s the difference in employee productivity when an application is 10% slower, 50% slower?

These types of metrics have typically been calculated for customer facing applications like Web retailers, but getting the data for internal IT users has been far less popular since it’s considered a soft cost in some arenas. But it really starts to add up if you pay attention.

One NetQoS customer said their typical critical business application “brownout” (before deploying NetQoS products) cost them $6000 per hour and they had about 20 of these per year, each taking about six hours to isolate and resolve. That’s $720k gone per year due to poor application performance ($6k * 6 hours * 20 events = $720k/year). True, the brownout costs less per hour than most estimates you see for out-and-out downtime, but they occur a lot more frequently.

It took some investigation and understanding on the customer end to establish the value of different applications, who was using them, and then run the numbers, but now they have some idea of the cost of all of those shades of gray between up and down and this helps them justify their investments in technology and process improvements to reduce the brownouts as well as the blackouts.

This is why vendors, such as ourselves, are willing to come out and have a conversation and demo with your company.

But even so, consider this idea as an inaccurate but useful shorthand in the form of a Zen koan: If the network is so slow that nothing gets done, is it any different than if the network were down all together?  And what is the difference between a network down for half a day than a network that takes twice as long to get anything done for a full day? 

And if a computer goes down in the woods, but no one receives an error message, did it really have an error at all? And what is the sound of one router crashing?

Recently, in the Herald Tribune, there was an interview with John Chambers, CEO of Cisco. One of the things about the interview that I found particularly interesting was that it seems that Chambers is really into collaboration technologies.

You would expect a networking hardware and software company to be into collaboration – the entire point of networks is to interconnect computers so that two or more computers can work together. But Chambers focuses more on Web 2.0-style collaboration, talking about video and blogs.

Today’s world requires a different leadership style — moving more into a collaboration and teamwork, including learning how to use Web 2.0 technologies. If you had told me I’d be video blogging and blogging, I would have said, no way. And yet our 20-somethings in the company really pushed me to use that more... By the second [video blog], I realized this was going to transform communications — not just for the C.E.O., but it would change how we do business.

From MediaNet to FlipCams to TelePresence, it seems – and I’m only guessing here – that Chambers is trying to take Cisco from a “networking company” to a “collaboration company” – much like Xerox tried to move from being the “copier company” to “the document company” in the late 1990s.

But it also is worth pointing out that collaboration tools are more bandwidth heavy than they have been in recent years; Chambers choosing to video-blog, for example, rather than text-blog. Telecommunications used to be about voice, now it’s about high definition video. When you have a population of 20 and 30-somethings not afraid to use the technology, they’re going to push that technology to the limits. That’s a good thing, but it bears pointing out that you only want to push the technology to the limits… not over it’s limits – which is why network monitoring tools are so important to knowing exactly what those limits are – and give you an idea of how you can start to overcome them.

I’d like to be skinny. And have a million dollars. G’night folks!

------------

I have just been informed that, even though it is Friday, I still need to put at least some effort into writing an intelligent blog post.  So, here goes.

Here’s my first thought about health and the economy: Obviously, there have been massive layoffs across the board, and IT has not been spared.  Over the past two years, not only have there been layoffs due to the general contractions (or as I refer to them, death-spasms) of the economy, but since 2006, there has been an increase in the number of internationally outsourced jobs by IT service vendors, according to Network World. 

Data prepared by Everest Group Inc., a research and outsourcing consulting firm, shows in broad brush fashion the shift of jobs overseas by some major IT services vendors. In 2006, U.S. and European firms typically had less than 20% of their workforces offshore; Now, for most companies that figure may well be generally over 30%.

At the same time, many laid off workers are starting their own businesses.  Certainly not all of them, but when you need a job, and no one is hiring – entrepreneurship and despair seem the only logical choices. 

A quarterly survey of 3,000 job seekers conducted by Chicago-based outplacement firm of Challenger, Gray & Christmas, Inc., released Thursday, shows a near doubling in the year-to-year growth of job seekers turning to self-employment.

The problem with this is that because more people are becoming unemployed or self-employed, it creates additional political pressure on the health care debate.  The unemployed and self-employed have to rely on the private insurance market for healthcare, which is where the majority of healthcare horror stories exist, according to NYT opinion columnist and Nobel Laureate Paul Krugman. Krugman argues that government regulations require that employer contributions to health care “can’t discriminate based on pre-existing medical conditions or restrict benefits to highly paid employees,” and thus people don’t see the worst parts of private insurance until they’re either unemployed or self-employed.

Which brings me to my second thought on health and the economy: The “American Recovery and Reinvestment Act” (a.k.a. “the Stimulus”) which give $19B for healthcare related information technology, but also requires that each American has an electronic health record in 5 years.  TechTarget argues that meeting that deadline simply isn’t that easy. 

"The concern is that when you have these programs that are time limited … that the quality of those implementations could go down," said Chad Eckes, the chief information officer for Schaumburg, Ill.-based Cancer Treatment Centers of America (CTCA). "There can be spectacular failures of electronic health records, because folks didn't anticipate what might happen if it was unstable, and that can have disastrous consequences for patient care."

We’ve seen some of the difficulties with medical data networks before – with the importance of maintaining network performance when lives are literally on the line. 

There’s no question that electronic data records are faster than paper records, and (usually) more accurate… or at least, more accessible in a crisis, which is why there is a mandate.  The problem arises when there is poor network performance, in which case records are less accessible.  Moving the patient records from paper to data places the onus of providing that information from the administrative staff of a hospital or medical center to the IT department.

Deep Packet Inspection, infamously used by Comcast to forge reset packets to disrupt the BitTorrent protocol, and by the NSA to spy, and by the government of Iran to identify protestors (pursuant to imprisoning and murdering some of them) is making a comeback in enterprises, according to Christopher Rhoads at the Wall Street Journal.  

Out of 145 top-tier communication-services providers, 77% of respondents said they were either strongly or somewhat interested in DPI, according to results of a survey last year by Yankee Group and RCR Wireless News. Most said they wanted it to improve network security, according to the survey.

The concerns, as always, are with traffic prioritization and security.  For traffic prioritization, the obvious uses are placing streaming teleconferencing videos ahead of streaming YouTube videos of cats or wedding dances, and in the case of security, it mostly deals with being able to identify malware on the network, for example, by content, rather than by anomalous behavior. 

The point is that most network traffic monitoring solutions identify traffic by context: Flows, flags, and facts about your devices. DPI identifies traffic by content.  True, DPI gives you a lot of information, but it gives you far more information than you need, with uncomfortable privacy concerns. 

From a more pragmatic standpoint, by focusing efforts on content rather than context, network engineers and network management might end up spending too much of their time micromanaging the network.  That is, it should not be the priority of the network team to prevent non-critical traffic – it should be the priority of the network to preserve critical traffic.  For most organizations, having a controlled network is not as important as having a network that meets the application performance needs of the business.

And somewhere in the middle of controlling every aspect of the network by content and not knowing or caring what goes on in the network is the middle ground of knowing how your network is being used. 

According to Yahoo Finance, which, you would imagine might have an accurate take on such things, Microsoft and Yahoo have finally agreed to a partnership.  You will remember that Microsoft tried to purchase Yahoo outright last year, but the deal fell through.  Instead, Yahoo will now use Microsoft’s Bing search engine to power search, while Yahoo will handle the online advertising. 

Why Yahoo decided to switch to Bing is unclear at this time, as Yahoo’s engine already serves nearly 20% of the market, compared to Microsoft’s 8.4% (and Google’s 65%).  I’m not prepared to speculate further than saying that Yahoo’s value isn’t really in the search engine, but the SAAS solutions that are so ubiquitous, one barely thinks of them.  Yahoo Mail, Yahoo Groups, Flickr, Del.icio.us, Yahoo Voice, and Upcoming.org.  Yahoo still has more overall users on the Web and more overall pageviews than Google.

Details are still sketchy, but the deal doesn’t seem to affect Yahoo’s SAAS offerings.  Perhaps that’s because Microsoft has gotten more aggressive on the online services front since they last tried to acquire Yahoo in February of last year, offering an ad-supported online version of Office.  Actually, that may explain the deal – Microsoft no longer needs to own Yahoo’s cloud software, but it still would benefit from Yahoo’s ad revenue model. 

We’ve talked in general about the effects of cloud computing on application performance. (Long story short: Just because it’s on the cloud doesn’t mean you can forget about making sure apps perform well.) However, one has to consider that if Office goes ad-supported, and widely adopted, how much traffic will be used up serving up those ads – especially if they’re large files, like those annoying flash-based video ads that pop up.  I suppose we’ll find out more as time goes on – whether they’re inconsequential, or eroding network performance in a matter not unlike being nibbled to death by ducks. 

We recently put together a report with Ashton, Metzler & Associates, trying to gauge the state of network management today. After our best efforts, we have learned a few things.

For example, the state of network management is not Ohio. That’s the Buckeye State.

After checking the 50 states of the U.S., the six states of Australia, and the 31 Estados of Mexico – even broadening our definition to include Canadian Provinces – we still couldn’t find the state of network management.

Then we thought about surveying more than 300 network engineering, operations, and management professionals about how IT organizations manage application performance.

Here’s what we found out:

• 93 percent of respondents indicated their organization had either formally or informally identified a set of applications that are considered critical to the business. However, only 41 percent of those surveyed indicated that the company’s business managers were involved in identifying the critical applications.
  
• 75 percent of respondents said identifying the company’s critical applications has led to at least a moderate change in the way they design, manage and troubleshoot the network infrastructure. The most common change cited was implementation or enhancement of quality of service (QoS) policies.
  
• 80 percent of respondents reported that their IT organization has mapped the supporting network infrastructure components upon which key applications depend. These organizations are far more likely to focus their monitoring efforts either exclusively or primarily on these critical components than the non-critical ones.
  
• Half of respondents indicated that they measure and report on the mean time to repair (MTTR) for a network or application outage. However, only 30 percent confirmed they actually measure and report on the MTTR for degraded application performance, revealing a continuing legacy of fault and availability management over performance management.

What this means is that we still have a long way to go – that many companies still look at networking problems from a perspective of fault, and not of performance, and that end-users are still likely to notice slow-performing applications before the IT organization.

On the other hand, the good news is that the report shows that IT professionals are focusing more on applications as part of the network, not as a separate discipline.

In “The Mandate for a New Age MOM” Dr. Metzler recommended specific goals IT organizations must meet to effectively manage the network for application performance:

• Discover all applications that are on the network and identify the handful of them that are the most critical to the running of the business.
  
• Baseline the performance and usage of the company’s primary IT resources - the most important business applications and the components of the IT infrastructure that support those applications.
  
• Implement tools and processes that allow the IT organization to monitor the key performance metrics (e.g., response time, utilization) of the company’s primary IT resources, and allow the IT organization to quickly respond to a situation once it has impacted the end user.

Yesterday, TechCrunch and Slashdot, among others, reported that AT&T users were unable to access img.4chan.org; one of the subdomains hosting the infamous “b” board. 

If you’re unfamiliar with 4chan, do not google it. I have not provided a link to the site in the blog, and that is for very good reason.  It is rather disgusting. 

Still, while crude, 4chan has had a profound influence on Web culture, and is one of the largest participatory Web sites out there – so large that Time.com did a profile on its founder, Moot, who was named Time Magazine’s Most Influential Person of the Year… after 4chan rallied enough followers to completely dominate the online poll rankings so that the first letter of each of the top 21 people on the list spelled out a secret message. 

Here’s the problem: AT&T blocked part of 4chan in order to cut off a DDoS attack in its tracks last night.

AT&T made a statement to TechCrunch this morning, explaining exactly what happened.

Beginning Friday, an AT&T customer was impacted by a denial-of-service attack stemming from IP addresses connected to img.4chan.org. To prevent this attack from disrupting service for the impacted AT&T customer, and to prevent the attack from spreading to impact our other customers, AT&T temporarily blocked access to the IP addresses in question for our customers. This action was in no way related to the content at img.4chan.org; our focus was on protecting our customers from malicious traffic.

Overnight Sunday, after we determined the denial-of-service threat no longer existed, AT&T removed the block on the IP addresses in question. We will continue to monitor for denial-of-service activity and any malicious traffic to protect our customers.

However, none of the users of the site - nor its owner - understood why the site was blocked for AT&T users. (AT&T claims that they tried to contact Moot, Moot says he was never contacted.)  In the absence of solid information, a conspiracy theory popped up that AT&T decided to “censor” 4chan.  Within hours, 4chan denizens, known collectively as “Anonymous” made plans to take on AT&T, much like they took on Scientology, though it looks like with this morning’s disclosure, those plans are on hold – though individual 4chan users may still make decisions – like cancelling service - based on bad information.  It’s a misguided effort, of course, considering that already, the site is back up, AT&T has explained their position, and there was no harm meant by the temporary blockage. 

But the damage has been done.  That’s the problem with making networking changes without informing people – if you block a particular site, or make a major network change affecting tons of people, you owe it to your users in order to explain why you’ve made that decision. 

A post by “anonimouse” on the Project AT&T web site sums it up:

Why is img.4chan.org blocked?
That is the question you should be asking. Without a why we don't have a reason to do anything. Now, we know this is not a mistake from the customer service convos but we don't know exactly why it is banned.
If it's about Net Neutrality, they have a war coming on.
If it's about the DDoS like the rumor says, we are getting out panties in a bunch for nothing.

If you’re messing with the Web experience anyway, wouldn’t it make sense to return, as a small HTML page something explaining what the problem is and why the decision was made? In fact, the statement AT&T made to TechCrunch would have explained everything – if AT&T had disclosed the information to 4chan’s userbase instead of trying to communicate through the tech media after-the-fact. 

When you don’t explain why you’ve made changes to the network, people will assume the worst about what you’re doing – in this case, that AT&T censored out of sheer spite.  We’ve seen this with Bilderberg.  We’ve seen this with the undersea internet cables accidentally cut by ships’ anchors.  Now we see this with AT&T. 

Because there wasn’t a little disclosure, a millions-large community of Internet users are now suddenly more aware of the net neutrality issue and likely to support regulation of companies like AT&T – or, in extreme cases, just interested in making life difficult for AT&T in general.  Either way – this is not good from AT&T’s perspective. 

Network World reports that Wal-Mart is going to be selling an AMD-Sempron 2.1GHz powered laptop with 3GB of RAM for less than $300. It’s a bit more powerful than what we think of as a “netbook” – which can go for as little as $238.

We’ve talked about how netbook ownership has gone hand-in-hand with cloud computing, but it struck me that we seem to have passed a point long ago where hardware was not the limiting factor for desktop applications.

That is, there was a time, not too long ago, when digital video editing was impossible for many desktop and notebook computers. (I’ll be referring to video editing and rendering a lot, as it’s the most processor intensive item I can think of.) Professionals could spend thousands of dollars – or hundreds of man-hours – to create videos, but home movie making didn’t really take off until the hardware could push enough pixels in a short enough amount of time.

Encoding MP3s used to be a chore. DVD playback required onerous hardware requirements. There were just some things that you just couldn’t do without a fast computer. The “top of the line” computers could do things that “bargain” computers couldn’t.

I’m not sure exactly when, but I think that we hit the point where having a faster computer didn’t open new doors to you, it just made what you already do, faster. Differences in degree, not in kind.

Certainly, video editing and rendering is faster on a quad-core i7 chip than on a single-core Sempron, but the point is that you can do video editing on a Sempron if you are willing to wait a while for the finished product. If you know you’re going to do a lot of processor intensive stuff, like gaming, or video editing, or audio mastering, or protein folding, you may decide that having the more powerful computer is a worthwhile investment, but it’s no longer talking about “need” but “convenience.”

I may be wrong on this, and I may even sound naively like Charles H. Duell in 1899, but I think that 20 years from now, we’ll still be using computers to do the same things that we do today, just faster. We’ll all be editing 4k or 8k cinema instead of high def, but it’ll still be video editing. We’ll still be playing games and browsing the web, compiling spreadsheets, etc.

Which is another factor in the rise of the “Cheaptop”; the fact that a lower-powered, cheaper computer can do the same things as its expensive cousins.

We have not, of course, reached that stage of network development; there are things you can do with an expensive, robust network that you cannot do with a simple, cheap one. And cloud computing has a way to go; not just because we’ve yet to find workable replacements for all our desktop apps on the Web, but also because the real limitations in network performance make some tasks, especially those that require low latency (like gaming) or high throughput (like video editing) difficult.

But it’s also why people are trying to find solutions to putting gaming and video editing on the cloud – because the challenge is still there. The things we cannot yet do will not be desktop applications – the things we cannot yet do are things that we will be doing on the cloud. It’s why the hype is so powerful and pervasive with cloud computing – because we techies are always looking for the next big challenge, always looking at ways to do more things. Doing them faster is great – that’s engineering. But doing new things – that’s invention. And that’s a hell of a lot “sexier.”

If you were to ask me my five favorite comedy troupes of all time, I’d probably name the Muppets, the Kids in the Hall, Backpack Picnic, Monty Python, and the Legislative Branch of the U.S. Congress.

This is just a quick post today, but I wanted to follow up on something I wrote yesterday about user interfaces.

The Washington Post has a copy of the flowchart used by Rep. John Boehner (R-OH) explaining his opposition to the Democratic Health Care Plan.  Look, I’ve only taken one class on graphic design, but that’s pretty much textbook “confusing” and “scary.”  It’s actually kind of ingenious – by using different fonts, different shapes, confusing-to-follow arrows, and a color scheme best described as “Eegah!”, the Republicans have made their case that the health care plan will result in scary, confusing… charts, apparently.

Then you take the flowchart put out by the New Republic which is similarly complicated, but less scary, showing the current state of health care in America, and while it’s not designed to be scary, it certainly is complicated.

Which brings us to graphic designer Robert Palmer, of California.  Palmer took the healthcare plan, and tried to create a flowchart that presented the information about the Health Care Plan in a way that’s intended to educate, rather than confuse. 

Now, whether or not the health care plan is a good idea is beyond the scope of this blog.  But it illustrates a point about how important it is to present information in a way that those who need to understand it, can easily understand it. 

There’s an interesting article in the automotive section of the New York Times.  It talks about how the Ford Motor Company creates fictional personalities which detail a “typical” end-user for an automobile they hope to design – and then they design the vehicle around that end-user.  Ford came up with the process because they found that car designers were designing cars that they themselves would like. 

“Invented characters get everyone on the same page,” Mr. Callum said. “Personalizing gives context to the information we have. Sometimes the target demographics are difficult to relate to by, say, a 35-year-old male designer.

“We found in the past that if they didn’t understand the buyer, designers would just go off and design something for themselves,” he added.

One of the interesting things about that broad range of categories we call “IT” – network architecture, software development, even the first couple of days you work tech support - it’s a lot easier to design technology for yourself than for someone else – the intended user – who can’t always articulate what they want or need. 

This is perhaps most evident in the user interface for open-source programs.  Not to say that open-source programs are bad or anything, but an open-source developer coding an app primarily for his own needs might very well code a console app, or an app with a confusing UI – it doesn’t matter to the developer however, as he knows that he’s designing it only for himself.  Even when taken to a broader audience, developers often code for other developers. 

We’ve mentioned before that we try to design our products with multiple audiences in mind – with executive level reporting but an ability to drill down into the details for the network engineer on the front lines, for example.  But the NYT article got me thinking about something else – and that is, do network architects design networks with the end-user in mind?  Or do they design networks for network architects? 

After all, the network for a development house has different needs for a network for an accounting firm, which also has different needs than a network for a video production facility.  Should we be thinking more along the lines of designing the network around the business need, rather than adapting a generic network to the business need?

Food for thought.