Peter Sevcik and Rebecca Wetzel of consulting firm NetForecast recently came out with an article in Network World regarding some results from a survey of 300 companies which asked about application performance management. If you’re a regular reader of this blog, you’ll probably have already guessed that those survey results underscore a point we’ve been saying all along.

The article, entitled, “Four steps to application nirvana” makes a number of very good points points out that implementing a best-practices strategy leads to better application performance.

The survey results show extremely positive correlations between best practices benchmark scores and actual application performance delivered to the business.

On the whole, enterprises with excellent best practices deliver 100% better results to their users than those with poor practices.

Here's where the rubber meets the road. Our survey results show that best practices exert their most dramatic effect on improving the time it takes enterprises to solve problems, with a 338% score improvement in problem-resolution time among those with best practices compared with their poorer-performing counterparts. …Those with the top best practices scores were more than twice as likely (144%) as those with poor scores to discover problems through systems vs. learning about them from users — and they were twice as likely (93%) to favorably assess the overall response times for their important applications.

In the article, Sevcik and Wetzel describe the four best practices as “Understand, measure, report, and link.” Of course, that doesn’t make a whole lot of sense out of context, so they provide 12 ways to apply the four best practices. Which really makes it more like 12 best practices to me, but I’m not complaining, since it seems to work.

Another thing we really like about this survey: they ask what’s going wrong, as well as going right.

Finally, we asked enterprises to identify impediments to improving application performance. Insufficient cross-group collaboration, insufficient manpower, and lack of proper tools tie for the top of this year’s list of impediments with nearly 50% of respondents mentioning them.

Ahem.

More seriously, Sevcik and Wetzel – and isn’t that an awesome name for a 1920s vaudeville act? [Edit: I’ve been informed that I’ve been repeating my zingers…]– have put out a blog called “App Performance View” on Network World’s site and asked for user feedback on vendor products – including NetQoS Performance Center.

According to Inside Higher Ed, one of the provisions of the Higher Education Act, which was passed last August mandates for colleges to police their network for illegally copied copyrighted works.  And according with a survey conducted by the Campus Computing Project, this will cost colleges $350,000 to $500,000 a year out-of-pocket to comply.

Colleges are “required to consider the use of technology-based deterrents” to prevent or deter copyright-infringement on peer-to-peer networks.  These can include traffic monitoring and packet shaping.

There are a number of problems with this plan:

First, the record labels want the colleges to pay money because some of their students are violating record label copyright.  The colleges are stuck in a bad place; because the government has mandated it.  It’s highly likely that these measures will be ineffective, but even if they are effective, the copyright infringers will just move off campus and get residential broadband, leaving all the students paying higher tuition and residency bills because the colleges have to pay for the ineffective enforcement somehow. 

So, who benefits from this law?  The labels?  This won’t stop one copyright infringer, and I think they know that.  What it does do is pass on the costs of ineffective piracy countermeasures from the labels to the colleges?  The colleges don’t get anything – nor do the students, pirates or otherwise. 

Nobody benefits, and most people suffer.  This is a crappy law. 

Additionally, there are significant political and academic freedom issues in requiring a network to monitor the traffic for particular activity.  This is especially discouraging when you consider that the rate of false positives is going to be extremely high: there is no traffic monitoring solution that is smart enough to tell whether a particular MP3 file is legal or illegal.   Any activity that blocks illegal content can easily break legal ones: fair use of the work, public domain, creative-commons, or downloaded by the copyright holder.  (Recently, a smaller record label had its site pulled because it had hosted copyrighted MP3 files – its own.)

So, in short, it’s stupid, ineffective, and has potential unpleasant side effects. How do you best comply with a law that’s so dumb that it borders on ridiculousness?

The only way to fight stupidity is by being clever – find solutions that won’t cost hundreds of thousands of dollars and which do not harm the ability of students, faculty, and alumni to advance the academic mission of the school.  For example, enabling and properly using Cisco Netflow information to make intelligent decisions about QoS policies.  (Labs and offices get more leeway than student residences, local proxies of popular legal BitTorrent files – like WoW patches and Ubuntu releases - on the LAN.) 

Considering that the provisions in the law (and I am not a lawyer, this is not legal advice, and even if I was a lawyer, I probably wouldn’t be a good one anyway) basically amount to: “Do something” about P2P copyright infringement, it’s probably best just to use QoS policies to throttle down (without cutting off) that traffic which looks suspicious.  But err on the side of assuming that the use is legal and for the academic mission. Maybe set up a filter that only goes into effect if the protocol is BitTorrent, and the content is a video file, and the seeders and peers aren’t all from IP addresses that correspond to astrophysics laboratories. 

You could also make use of anomaly detection software to track spikes in MP3 traffic – that is, of course, assuming that the downloading of music files by the 18-25 demographic is in any way considered “anomalous” rather than “status quo…ulous…” Something like that.

David Oliver talks about his experiences running the 24 Hours LeMons race in Houston, and how knowing about network performance helped him optimize his junker.

Network World is reporting that Gartner has recommended 20 steps which can be taken to cut IT costs in a recession. 

These are the same 20 steps which can be taken to cut IT costs during prosperity as well, so even if a miracle happens and the stock market repairs itself tomorrow – might want to keep this around the office in a drawer somewhere. 

Below is the list – with some annotations that we’ve added to give some perspective. 

1. The most obvious place to start is people costs. Gartner estimates that 37% of the average IT budget is dedicated to personnel, so this represents a major opportunity to save money. Gartner recommends a blend of hiring freezes, reducing or eliminating special bonuses, cutting back on outside contractors. Also, global companies that have opened offices in remote areas should consider bringing those workers back home.

This is a good plan, of course, but one of the key things to remember about bringing IT workers back from branch offices to the headquarters involves moving the servers that they manage with them.  If you don’t do this, no matter how good your administrators are at logging in remotely, there’s going to be a problem, sooner or later, that will require expensive airfare. 

WAN Optimization, virtualization, and its brother-in-arms, server consolidation will make this possible.  But remember that when you move the server closer to the HQ, you’re essentially moving the workers farther away from the servers. This means that traffic that once went on local area networks now travels on the WAN.  It would be a good idea to make sure that the WAN can handle the new traffic, and if applications aren’t properly developed for a higher-latency, lower-bandwidth network, it’s probably time to find that out.

2. Flatten the organization. Instead of having one person manage six or seven employees, trim some of that middle management and have your IT execs manage more like 20 people. A flat organization not only saves money but also can lead to more efficiency.

3. Move to shared services. In other words, consolidate things like help desk into one group that services the entire company.

4. Even if you have to borrow somebody from another part of the company, bring a finance person into your leadership team so that person can analyze your budget and find ways to help you trim costs.

5. Don't ignore "unmanaged" costs like printers or data center power.

6. Go back and check your invoices to make sure your vendors are charging you what your contract specifies. An example would be if your wireless vendor agreed to give you free shipping when it sends new cell phones to remote workers. A few months later, shipping charges might start appearing on your cell phone bill, and if you don't check, you'll never know.

7. Eliminate unused software and modules.

Another way to look at this is to round up the unintegrated, non standardized and overlapping “bag o’ tools” that engineering groups generally have lying around. Beyond the license and maintenance implications, tools standardization can lower training costs and provide a more common language and methodology for a team.

8. Get tougher with vendors when it comes to negotiating contracts. Don't be afraid to switch vendors, or at least go the first step of determining what it would cost to switch.

This is especially true when you’re talking about managing service level agreements for leased lines – it’s important to independently verify the claims that they will improve performance or provide a certain level of service.  Do you have something to measure that? If you’ve got Cisco gear, you can use IP SLA tests to get edge-to-edge measurements. More than ever, you need visibility into these service levels so you know when there is degradation and you can hold your vendors accountable.

9. Buy a telecom expense-management service. It pays for itself and more.

10. Deploy a corporatewide plan for buying cell phones. Then, buy a cell phone plan that optimizes expenses. This will be cheaper than letting employees buy phones and plans and then expense them.

11. If there are places where you don't need five nines of availability, settle for three nines. It will save you money when you negotiate with your vendor.

Additionally, I’d say that application performance would be the key metric – not availability. There’s really little difference between 99.999% uptime and 99.9% uptime from a practical point anyway. (5 minutes of downtime per year compared to 8 hours, 45 minutes of downtime a year.)

However, there is a huge difference between an application that performs well and one that performs poorly, if only because you’d rather run a fast performing application for 364 days, 16 hours a year than a slower one for 364 days, 23 hours, 55 minutes a year.

Do you know when performance issues are the fault of your service provider? Can you measure application performance day to day, month to month, to know what your ends users are experiencing? There are management tools that help you do this .

12. Consider buying a videoconferencing unit rather than constantly renting.

Additionally, make sure that your network can handle the traffic from the videoconferencing stream – video takes up huge amounts of bandwidth and can impact the performance of other applications if not managed properly. Know how to do this?

13. Where possible, use the Internet as a replacement for expensive WAN transport services.

The key phrase in the above sentence is where possible.  There are some applications which belong on the WAN and would perform inadequately over the Internet.  However, that doesn’t mean that some applications can’t be moved to the Internet – just make sure that you’re aware of the trade-offs in reliability and performance in exchange for cost. 

14. Defer moving to Vista. If your PC hardware is holding up, consider sticking with it another year.

Really, though, is anyone planning to move to Vista?  I think we’re all hanging with XP until we get Windows 7. 

15. Use commodity products wherever possible, and skip best of breed in cases where "best of need" will suffice.

16. Consolidate and virtualize servers.

Just make sure that you do not lose visibility into the network when you do virtualize. Virtualization adds complexity and instead of having to monitor inter-server interactions, you now have to manage intra-server interactions as well.

17. Reduce storage costs via data deduplication and other methods.

18. Use better processes and policy to make better use of existing tools.

19. Deploy IP telephony and VoIP as a way of cutting costs for moves, adds and changes.

Again, VoIP can play havoc with your network if you haven’t configured it correctly and set appropriate policies for the VoIP traffic to coexist with the TCP traffic that holds application data.  Additionally, make sure that VoIP provides a quality of experience comparable to that of a POTS phone line. 

20. Harvest unused software licenses and reuse them when a new employee makes a request.

Brian,

I'm heading out of town from the European Symposium, and so I won't be able to get a blog post out before we land. 

We could talk about the attendee survey – we got 49 responses and all of them said that the overall rating for Symposium Europe was either a 4 or a 5 on a scale from one to five – and this was from an audience that was 98% first-time attendees. And all – that’s 100% - of the survey respondents said that they would recommend NetQoS European Symposium to others. Kevin Davis’ “NetQoS Myths and Realities” session was the highest rated, but the ratings across all sessions were outstanding. “Access to NetQoS staff” was off the charts as well.

Problem is, that’s kinda dull for the outside world. But I had a blog idea that might be fun to play around with. Run with it or ditch it, your choice. 

Basic idea was to talk about conservative/liberal political philosophy in the US.  Say something like even a liberal American is way more conservative than the most conservative European.  Like for example, the whole idea behind red light districts being a part of life here. And that while even US may lag Europeans on acceptance of red-lights in general, we all agree that “red-lights” in performance management should be avoided.

Needs a lot of work, but you could get some miles out of this thought for sure.

Subject: Re: European Symposium

From: Brian Boyko | Editor, Network Performance Daily 

Do you really think it’s a good idea to start a blog post with politics and end with prostitution?

- Brian

This is interesting; I actually had a conversation recently with my boss, in which I argued that the elections may have an impact on network performance so much as who is elected will help determine U.S. telecommunications policy, but otherwise… well, there wasn’t much to write about on the blog.

Apparently, I’m wrong. Because Slashdot just posted “Watching Tonight’s Presidential Debate Online,” and Slashdot is the IT blog. Maybe I made a poor judgment call – especially after hearing from poster ObsessiveMathsFreak:

The US presidential elections are actually very important. I see Slashdotters posting comments to the effect that both parties are equally bad and it doesn't matter which way you vote and excuses, excuses, excuses. I can tell you from the point of view of someone who is very much affected by the results of your national elections, this is a pretty depressing thing to hear. It's clear to anyone who has half a clue that there are very wide and deep differences between the two main candidates, and it's quite irritating to find out just how flippantly many Americans go about voting, or not voting, for their president.

Your election affects me. It affects people around me. My nation's economy, policies, laws, and culture, yes culture, are significantly affected by your selection of a president, through his administration's policies. This is "Stuff That Matters" to me.

The other important thing is that the original submitter prefaced his article “For those of us that no longer have a television…”

This is important because for whatever reason, geeks are early adopters. It’s geeks who first bought GPS devices; now they’re mainstream. It’s geeks who were first on the Internet. Now it’s everyone. It was geeks who first started blogging, geeks who first started eco-driving, geeks who first started “podcasting and vodcasting” before iTunes and YouTube brought it to the masses. And it was geeks who were rolling dice and pretending to slay dragons way before World of Warcraft.

If geeks have learned that they can eschew their televisions completely – not just the cable but the actual box – then where are they getting their audiovisual media from?

And that’s where we get into network performance – because there’s the possibility of hitting that tipping point where most people will be getting their audiovisual content via the Internet.

What’s strange is that this goes back to Clay Shirky’s thesis that we’re moving through a shift where the cognitive surplus created by the move to the forty hour workweek is no longer masked by television – that is, that we now have other things to do with our time. Television is a technology that essentially is designed for the person to sit down and “see what’s on.” Internet video is designed for two types of people: those who know what they want to watch and when they want to watch it, and those who treat video media as any other media, following hyperlinks where they may lead – if it’s text, graphics, audio, or video, so be it, so long as they get the information.

It is not enough to dismiss all video content as “recreational network traffic” because eventually we will start to communicate more and more with multiple media as costs go down and we all become more familiar with producing and sharing. This means that networks have to be able to handle the traffic that video will require.

In the meantime, enjoy the debate – online, or on the air.

Just a quick post here this morning: This Friday, at noon (PST) we’ll be hosting a joint Webinar with EMC on “The Benefits of Compehensive Application Availability and Performance Management.”

Ultimately, when applications crash or perform poorly, it costs the company money and annoys the people that use the application. We teamed up with EMC so that Smarts customers can monitor and manage application performance, (as well as application availability), from a single console.

The Webinar will cover how the EMC Smarts and NetQoS console can help you determine the root cause and impact of failures across the IT infrastructure quickly, can help you track performance and identify network bottlenecks before they impact service levels, and troubleshoot problems with an end-to-end view of application traffic. You can register for the Webinar here.

Network World has a Halloween themed slideshow up for the “2008 IT Industry Graveyard.” Interesting enough, but we’ve got to disagree with some of their choices. Some of them were declared dead prematurely… in fact, you could say that they were…

Buried Alive!

For example, the first one is Windows XP – with the idea that XP’s dominance will end simply because Microsoft has refused to sell XP directly to customers, or to OEMs that sell to customers. But though Microsoft tried to kill it, the reality of the situation is that XP is preferred by most customers over Vista, marketing campaigns aside, and if a customer has tried – and hates – Vista, he’ll resort to computer piracy (discounting those who resort to Linux). This is simple incentive: With piracy, you break the law once, and are annoyed only occasionally by anti-piracy measures. With an OS you don’t like, you’re just continually frustrated.

The ninth is “Comcast P2P traffic throttling.” I suppose, on a technically, Comcast isn’t planning to go against the ruling handed down by the FCC. So it’s dead as far as Comcast is concerned. But other companies still find P2P traffic throttling tempting, and even the entire nation of Australia is planning to implement blocks of “illegal” traffic. There is no opt-out list, so it’s a massive censorship plan… with a “false positive” rate of about 10%. I’m guessing that once this is fully implemented, people will scream the minute they find out that one of the “false positive” Web sites is business critical.

(If you ask me, whichever Aussie came up with this policy is up the boohai shooting pukakas.) And this is just one example of trying to control the content rather than controlling the quality of experience.

But it’s the second one that’s a big deal killer. “The IT Department?” Really? Yes. We’ve read Nick Carr’s book. We just don’t agree with the conclusions. Relying solely on SAAS means that you’re at the mercy of another company's quality control, have no control over the performance of the network and no guarantees of five nines of uptime.

Now, don’t get me wrong – SAAS itself is great. But relying completely on anything is asking for trouble.

There are applications which save money when they’re leased as a service rather than managed by an in house IT department. But there are some applications which can’t be put on the cloud, and there are other applications which can be put on the cloud – but shouldn’t.

According to The Register, Cisco installation CDs for VPN networks contained music.

Specifically, music that sounded exactly like this.

Now, Mexican folk music of the “narcocorridos” variety has a rich tradition and requires extreme skill to produce, and is greatly enjoyed by many music aficionados. But still, if you’re going to come up with a piece of music designed to surprise the hell out of everyone, you could probably choose no better music in the world.

Knowing Cisco, there’s no way that this was deliberate; but this brings to mind two things: First, is there someone out in Baja California with a copy of VPN software in his or her hand, wondering to themselves: “¿Dónde está mi música?”

Second, will this start a trend of “narcorrido-rolling” network engineers?

Cisco is doing everything they can to recover from this error, and in a statement, said:

Cisco is aware that some customers have received defective VPN Client CDs as part of recent orders.

Manufacturing is aware of this problem and is actively reshipping new media to impacted customers.

Defective VPN Client CDs can be identified by the following marking on the back of the media which ends in "MX21511/4"

Of course the moral of the story is that you need to test before you deploy. In this case, it was a little embarrassment, and we all pretty much just have a chuckle about it. But deploying technology on the network without knowing the full effects is just asking for trouble.

I mean, what would have happened if the music actually installed? Is your enterprise prepared to handle accordion configuration?

Other than Unix Beards and “funny” T-shirts with hex code on them – which more accurately qualify as fashion disasters – the biggest project disasters in IT, according to today’s top story in Computer World, tend to repeat themselves:

When you look at the reasons for project failure, "it's like a top 10 list that just repeats itself over and over again," says Holland, who is also a senior business architect and consultant with HP Services.

You’ve got your usual run of top-ten disasters in the article, including IBM’s Stretch project (Overpromised and underdelivered), Knight-Ridder’s Viewtron (misread the market), California and Washington States’ DMV overhaul and FoxMeyer’s ERP program, (didn’t make sure the new system worked better than the old one), Apple’s Copland (succumbed to feature-creep), Sainsbury’s warehouse automation (just plain didn’t work), and Canada’s Gun Registration System (cost much more than anticipated due to poor planning), and three U.S. government projects (multiple failures with perhaps more in the future).

But one of the things that I noticed was that it’s relatively rare (not unheard of, but relatively rare) to see networking take a prime role in the huge IT disaster stories that get passed around the campfire during IT tribe meetings. And I think that there are a few reasons why that is – the first is that most of these blunders would fall under the category of “strategic errors” as opposed to “tactical errors.” That is, network problems are usually subtle errors caused by mis-configurations and highly technical mistakes. The networking screw-up can be one of the most subtle, stealthy types, compared to the grandiosity of all-out strategic incompetence.

Or in other words, networking performance problems can cause the best laid plans to often go astray; the worst laid plans need no additional help.

Take, for example, a common error from back when they were first rolling out VoIP deployments – companies would roll out VoIP on the network as if it were just another data application, but then found that their other applications slowed to a crawl or even stopped working.

The problem was that VoIP packets are based on protocols designed to use as much of the pipeline as possible, while most applications are based on the TCP protocol, which is designed to throttle back it’s use of the pipeline if packets don’t go through. So what happened was that the VoIP packets would take more of the pipe, TCP applications would be crowded out and drop packets, which would cause the TCP protocol to throttle back, and the VoIP packets would now see the free space and take up more of the pipe, crowd out TCP packets and TCP would throttle back… creating a vicious cycle.

Was this a problem with strategy? Was it some form of bureaucratic incompetence? No – it’s just that it was a very subtle effect and if you didn’t know enough about the TCP and VoIP protocols (or even if you did, but didn’t put two and two together until it was deployed) you ended up with a problem.

Networking problems may have major effects but they’re rarely caused by major boneheaded screw-ups. I think that’s one of the reasons why the two major areas where IT departments spend a great deal of money – networking and security – is because those two problems are extremely subtle to detect and tricky to solve; security problems by malicious design, networking by nature.

Networking problems are subtle, can strike quickly, can often leave little trace of their presence. They’re the ninjas of IT problems.

Of course, ninjas can be defeated.