Part 6 in a series adapted from Joel Trammell’s Keynote Speech at NetQoS Symposium 2008

How many of you out there are doing a server virtualization project?

More specifically, how many of you are doing a server virtualization project that you know of? Because often in companies, the server group will start consolidating servers in virtual servers – and neglect to tell the networking group. Worse still, when a problem develops, the network, not virtualization, gets the blame first.

There’s also desktop virtualization. If you haven't heard of that, that's the idea that, well, a user doesn't really need a machine. What they really need is an image and that image can be pulled down across the network to any piece of hardware that they might be operating on at that given period of time.

Because we all know there's this ubiquitous and highly performing network everywhere in the world for this user to access this image, right? And even though that may be a Windows image that has hundreds of megabytes of data, we're just going to zip that right down onto their local hardware of whatever type it is. Doesn't even matter what type because we virtualized it and we're going to run our desktop locally.

While most of the readers of this blog clearly caught the sarcasm in the previous paragraph, to a CIO, that’s a very attractive pitch. I can promise you there's a lot of venture money being put right now behind companies doing desktop virtualization because nobody wants to miss the next VMware from a returns prospective. So, it's coming. Whether it's a good idea or not, in what environments it works well, that's a whole different question.

Some applications are going to perform fine under virtualization. However, there's a whole other class of applications that don't work well in virtualized environments. Virtualization is not like having another server ‑ it's like having part of another server. And that part, depending on the application, may be good or bad.

So, with virtualization, it's going to be very important that you have the ability to compare performance in the virtual environment with the non‑virtual environment. You want to get in there and get a baseline before they do any virtualization so when they come back to you and say “oh, things have gone to Hell in a hand basket”. It must be the network' you can say, 'no, guys, look, nothing changed with the network ‑ here's the response time on the server you were getting before you virtualized, here's the response time you're getting now on the server. That's your problem. Virtualization caused your problem.

You'll be able to understand better what applications work well under virtualization, because tracking server response time before and after virtualization is the ultimate test. If I virtualized an application and server response time with the load stayed the same, that's a good application for virtualization. If I virtualized an application and server response time goes up by a factor of ten with the same load, I didn't get anything for my money, right? That's not a good application to virtualize. I promise you a lot of server folks are not thinking of this and you're going to be the brunt of dealing with the problem when it occurs.

Then, of course, there's servers running out of resources because now it no longer makes sense to look at the server as a single hardware running a single OS.

So, in that case response time's going to drive the ability. In this case server response time is going to be really critical in you helping prove that this is not your problem. And then from there, device statistics are going to be important as you virtualize servers.

By Brian Boyko
Age 29

I’ve just come back from vacation. I’ve been in New Zealand for the past two weeks. (Did you miss me? I missed you.) Anyway, after the stress of the U.S. elections, (I am a political animal*) I needed it, badly. 

I headed to New Zealand, mostly because I’m obsessed with the country’s multi-party electoral system of proportional representation.  This is especially important in New Zealand, where voting for the “lesser of two evils” invariably means casting a ballot for Saruman.  Also, I really, really wanted to try Zorbing.

Zorbing is basically crawling into a giant, air-cushioned hamster ball, and being rolled down a hill.  It’s just like being in an XKCD comic!

Here’s some pictures and video:

Inside the Zorb from Brian Boyko on Vimeo.

But, of course, as the adrenaline wore off and I reflected on the experience, I realized that, despite all the fun, I would have to start back at work today, so I did what I always do in these situations: Take something totally unrelated to network performance, and relate it to network performance.

For example, the Zorb in Rotorua has two tracks, both of which start from the same point, and both of which come to a stop at the bottom of the same hill; but I only had the time to go down one before the last bus of the day left.  Wanting to extend my fun as long as possible, instead of choosing the straight, downhill, fast track, I chose the bumpy, zig-zag one.  This track was more circuitous and involved multiple hops.  More hops = longer time.  In a network context, this delay is called “serialization delay.” In a zorb, this delay is called “WHEEEEEEEEEEE!”

Okay, it’s a lame comparison, but give me a second to get up to speed – I just got home after traveling for 34 straight hours, including 16 hours of layover and 18 hours of flight. 

The other thing about New Zealand is that everyone there complained about the speed of Internet access – there’s really nothing to do about it, as New Zealand is extremely remote to the rest of the world.  Even Australia is 1,000 miles away, and the alternatives to undersea cabling is satellite communication.  There are bound to be latency issues due to propagation (speed-of-light) delay.

Anyway, I’m tanned, rested, and ready.  We’ll finish up Joel’s series and post some other stuff in the following days.  Thanks for sticking with us. 

* some sort of flightless bird, most probably.

For the fifth consecutive year, NetQoS has been named to Deloitte’s Technology Fast 50 Program for Texas, a ranking of the 50 fastest growing technology, media, telecommunications, and life sciences companies in North America’s most awesome state and/or province. Seriously. We kick your state and/or province’s butt.

(We’d also say it kicks the butt of the Australian states, but we have to admit that while we have Chuck Norris, we only have one of them, and New South Wales has thousands of drop bears.)

We ranked #21 on a scale based on the percentage of revenue growth over five years from 2003-2007 – where our revenues increased by 671%. It’s increasingly difficult to retain a high ranking on these types of lists, mostly because it’s almost impossible for companies to grow at 3 digit percentage rates year over year as revenues climb.

Our growth in the network performance management market is very strong because we are acquiring new customers fast. Half of our business comes from new customers and new customer acquisition accounts for our much greater than average growth.

To celebrate this achievement, for one night only, I’m planning on giving away candy to anyone who asks for it.

We were also recently named to Software Magazine’s Software 500 ranking of the world’s largest software and service providers for the fourth consecutive year.

If you want to score some easy points with the geek crowd, tell them that DRM (Digital Rights Management) stinks. But with the stress of the elections, I could use a few easy points, so humor me.

When you’re evaluating a change to the network, you have to think – always – what will the real effect on network performance be? And this is important – if there’s one overarching theme this blog has had over the past two years, it is that the network does not begin and end with the router. It doesn’t even begin and end with applications. No – when you think about what the real effect on network performance will be, you have to think beyond the technical into the realms of the personal and psychological. That’s true end-to-end performance.

There are two stories that have been making their rounds through the Web recently; the first, which we’ve covered extensively, is the Australian Internet filtering software. The second, is the release of the highly anticipated Fallout 3 PC video game being bundled with SecuROM DRM software. This is notable for two reasons: First, Fallout 3’s publishers, Bethesda Softworks, earlier took a stand against DRM for the release of their other major product, Oblivion. Second, SecuROM, made by Sony, is particularly invasive, and particularly when used with EA’s hit, “Spore,” caused a particularly nasty backlash – which included a campaign to pirate the game via BitTorrent just to spite EA.

Bethesda Softworks insists that the SecuROM software is only used for a CD check and is not nearly as restrictive as the software that came with Spore.

A common complaint with DRM schemes is that they actually cause more problems for the people who legally purchase the game than for those who break the DRM in order to pirate the game. The worst DRM schemes can make using the product a hassle, cause system instability, and just generally be a pain in the butt, while doing nothing to stop piracy. The best DRM schemes don’t get in the end-user’s way, allows for reasonable use and portability, and is never under threat of expiration. Of course, these don’t do anything to stop piracy either, but let’s only concern ourselves with the latter for right now.

Point is, when you’re hassling only the people who adhere to the rules, you’re creating a situation where people will get around the rules.

In the earlier post we did on Australian net filtering; the point was made that for five dollars a month, you can set up a VPN in the United States get around all the restrictions of any of the proposed filters. However, in doing so, you’re essentially routing all traffic through the United States and back again to Australia – putting added stresses on the very expensive international pipes even when accessing local content that would have been better served via local pipes.

This is obviously a large-scale version of the problem, but enterprises that deny access (rather than de-prioritize bandwidth) to particular protocols, sites, or applications will find employees will get around the obstacles in order to do their job.

But even this is a very specific example of a larger point – one that goes beyond tactics, beyond strategy – to network philosophy. Even if you don’t think of it as a change to the network, when you change how users behave – even if it’s a new policy from the HR department that gets printed on a piece of paper, you change how the network is being used. Changes can occur to the network from outside the network as well – when culture changes, so does the way that people use the network.

All of this is leading to the point: Even though you aren’t planning on changing the network yourself, you should consider always keeping a close eye on your network with network performance monitoring tools. Change management is not just for when you change the network; change management is for when the network changes you.

Alex Payne, API Lead at Twitter, recently wrote about a blog post beginning “The Internet is built wrong.”

We happen to concur. There aren’t enough pneumatic tubes.

But aside from that, Payne’s criticism is that the Internet, designed years ago for pushing text, research data, and code was designed poorly for the problems we have now – the examples he states are IPv4 – which works well enough but is inadequate for the world’s need for IP addresses, and SMTP, the simple but unsecured nature of which has lead to problems with Spam and DOS attacks on e-mail boxes.

Moreover, he talks about performance:

“You needn’t do more than attempt to watch a streaming video on a busy office LAN or oversubscribed DSL circuit to understand that even the best-served markets for Internet connectivity are struggling to keep up with demand for networked content. Add to this that providing adequate security models for such content is a virtual impossibility on today’s Internet, and the need for a better approach is even clearer.”

The problem is that while Payne makes the case that the Internet doesn’t work as well as “something else” would, and we’re only using the technologies of the Internet because it works just well enough that “something else” can’t replace it even if it’s better – he leaves that “something else” to Van Jacobson and Jacobsen’s idea of “Network Channels.”

Jacobson gave a talk at Google in 2006 on this idea.

The cynics over at Slashdot immediately considered Payne’s idea of a “content centric approach to networking” as some sort of buzzword that means “owned by the media cartels.” However, I don’t think this is what Payne means at all. (If it is, I apologize for not being cynical enough.)

As Jacobson points out, TCP/IP is a very successful technology, but it has a few problems. You can’t connect to things that move – and we’re not talking Wi-Fi. Take trains, for example – as you switch seamlessly from one cell node to another, it’s easy to make a cellphone call on a train. But it’s not so easy to have a continuous Internet session. (In fact, if you do, it’s probably through the cellphone network…)

Additionally, the protocols that were designed for conversations between specific endpoints don’t work as well as they could with broadcasting because the network protocols in use were designed for conversations between two applications on two machines. But Jacobson believes that even the idea of the conversational model isn’t adequate to solve today’s new Internet problems.

From Jacobson’s video:

“We got a chance to look at the data on the routers downstream of NBC’s servers for the Olympics. At one time, their main router got severely congested when Body hit the pole on the slalom. In that router there were 6,000 copies of the same data. Everybody was pulling down the URL. The poor router can’t do anything about – you can’t optimize it, because its dynamic content. It’s all going out in separate conversations. All the router knows is I’ve got 6,000 separate TCP conversations. It’s the same data. If you could broadcast it, you could turn both the router and the downstream links from the server, reduce the bandwidth by three orders of magnitude, but our protocol architecture doesn’t support that. It works at the conversation level…

…Any of the measurements that I’ve seen recently saying that the high 90% level of traffic is people trying to get some named chunk of data. They hand in a URL, and they want to get something back. That’s not a conversation. That’s not a conversational model… that kind of interaction is a dissemination… It’s a point to multipoint or multipoint to multipoint.

Now, this is not to say that the traditional TCP/IP model doesn’t work – Jacobson is keen to point out that the problems we have today with networking only exist because the problems we used to have with networking that TCP/IP solved were solved extremely well. We just have new problems.

Weirdly enough, anecdotal evidence bears Jacobson’s model out. Look at BitTorrent, which is essentially multipoint-to-multipoint dissemination over the TCP stream. And it consists around 50% of the traffic on the Internet. Add eDonkey, another multipoint P2P app, and you get 70%. If that’s not a clear indication of changing demands, I don’t know what is.

Australia’s federal government has planned to require Australian ISPs to use filtering software to remove “illegal” content from Australia’s Internet. They’re spending around $77M (USD) to implement the program which the government had lead people to believe would be optional. Instead, it will be mandatory.

Mark Newton, a network engineer with Internode in Australia (but not working on behalf or speaking for Internode), did an analysis of the data gathered from Australian government trials of filtering software. He concluded that, among other things, more accurate filters degrade Internet speeds over 70%, and less accurate filters can have up to a 15% false positive rate.

In retaliation, Belinda Dennett, a policy advisor to Australia’s communication minister, Senator Stephen Conroy (Labor), wrote an e-mail to Newton’s employer, asking them to reign in the network engineer’s dissent.

We called Sen. Conroy’s office but we were not able to get a response before press time.

We have an audio interview in podcast form with Mark Newton below, with a transcript below the cut.

[Ed. Note: Due to problems with rendering in Internet Explorer 7, we've temporarily disabled the flash player version of the podcast. You can download the podcast as an MP3 file here.]

Continue reading "Interview with ‘Bullied’ Network Engineer on Australian Gov’t Net Filters" »

Most Americans can barely remember a time when there wasn’t enough electricity running to their house or apartment. Oh, certainly, we can remember times when we’d blow a fuse or trip a circuit breaker; but they were rare and usually happened under excessive strain – when the hairdryer and the air conditioner were on at the same time as the electric stove or George Foreman Lean Mean Fat-Reducing Grilling Machine.

In the 1950s, according to this advertisement uncovered by Modern Mechanix, running out of juice was a real problem because the wiring in house built decades earlier simply didn’t have enough capacity to run all those appliances. Well, of course it was – and this brings me back to my undergraduate days as a history major at the New Jersey Institute of Technology. (Yes, NJIT had a history major. No, it wasn’t a big class…)

Now, the second industrial revolution was years prior – during the 1910s through the 1920s, but consumer adoption of technological advances was stunted for a period of about 40 years. The great depression killed disposable income, when the war came and finally did bring some income, there was significant rationing and many companies specializing in electronics and electromechanical devices were building for the war effort. In a sense, when money started flowing in during the late 40s and 1950s, consumer demand had been “pent up.” Yeah, consumerism went overboard in the 1950s, but can you blame ‘em?

Fast forward to today and replace appliances with applications – there’s a reason that both have the same root word, the Latin “applicare” – and you can see the parallels.

Of course, things are a bit more complicated than the 1950s – there are only a few types of electricity – AC/DC, 110v/220v… etc. The difference is that it was relatively easy to tell whether or not the problem you were having was caused by a lack of electricity. Does the thing light up? Is it working? If the answer is no to both, you’ve got an electricity problem. If the answer to the first is yes, and the answer to the second is no, you’ve got a busted appliance. (If the answer to the first is no and the second yes, you need to replace the lightbulb.)

Today it’s a little bit harder to diagnose the problems you have with application performance – application, server, and network problems can all look very similar – especially to the end-user. Sometimes you spend time and energy working on one area only to find out it’s not the problem.

But sometimes, yes, the network is the problem, and more capacity is needed. The important thing is to be sure about it rather than just guessing.

There’s another lesson here too for consumer applications as well. That is, in the 1950s, the way to solve the problem of new demand on the electrical grid was to upgrade the wiring. Maybe we should be doing that to solve some of our own broadband “shortage” problems, instead of resorting to things like “bandwidth caps” and “aggressive traffic shaping.”

Because it’s not just about not being able to see the latest cat on treadmill video. Videoconferencing via Skype or other method puts people in face-to-face communication. CERN scientists needed to upgrade the Web’s infrastructure to share the massive amounts of data the LHC would create.

But one of the most telling things is what’s going on in Austin right now.

Our local NBC affiliate, KXAN, and our local cable provider, Time Warner Cable, haven’t reached an agreement to show KXAN programming on cable. One of TWC’s gambits is running an advertisement on television showing people how to connect their laptop computers up to their television so that they can watch the streaming video of the NBC shows that they’re missing.

Which, of course, begs the question; if you can get television shows via the Internet directly from the networks themselves, why do you need cable TV or network affiliates in the first place? This move may backfire. One of the reasons I don’t have cable is that I’ve been hooking my TV up to my computer for years now… then again, I have pretty good broadband service.

Peter Sevcik and Rebecca Wetzel of consulting firm NetForecast recently came out with an article in Network World regarding some results from a survey of 300 companies which asked about application performance management. If you’re a regular reader of this blog, you’ll probably have already guessed that those survey results underscore a point we’ve been saying all along.

The article, entitled, “Four steps to application nirvana” makes a number of very good points points out that implementing a best-practices strategy leads to better application performance.

The survey results show extremely positive correlations between best practices benchmark scores and actual application performance delivered to the business.

On the whole, enterprises with excellent best practices deliver 100% better results to their users than those with poor practices.

Here's where the rubber meets the road. Our survey results show that best practices exert their most dramatic effect on improving the time it takes enterprises to solve problems, with a 338% score improvement in problem-resolution time among those with best practices compared with their poorer-performing counterparts. …Those with the top best practices scores were more than twice as likely (144%) as those with poor scores to discover problems through systems vs. learning about them from users — and they were twice as likely (93%) to favorably assess the overall response times for their important applications.

In the article, Sevcik and Wetzel describe the four best practices as “Understand, measure, report, and link.” Of course, that doesn’t make a whole lot of sense out of context, so they provide 12 ways to apply the four best practices. Which really makes it more like 12 best practices to me, but I’m not complaining, since it seems to work.

Another thing we really like about this survey: they ask what’s going wrong, as well as going right.

Finally, we asked enterprises to identify impediments to improving application performance. Insufficient cross-group collaboration, insufficient manpower, and lack of proper tools tie for the top of this year’s list of impediments with nearly 50% of respondents mentioning them.

Ahem.

More seriously, Sevcik and Wetzel – and isn’t that an awesome name for a 1920s vaudeville act? [Edit: I’ve been informed that I’ve been repeating my zingers…]– have put out a blog called “App Performance View” on Network World’s site and asked for user feedback on vendor products – including NetQoS Performance Center.

According to Inside Higher Ed, one of the provisions of the Higher Education Act, which was passed last August mandates for colleges to police their network for illegally copied copyrighted works.  And according with a survey conducted by the Campus Computing Project, this will cost colleges $350,000 to $500,000 a year out-of-pocket to comply.

Colleges are “required to consider the use of technology-based deterrents” to prevent or deter copyright-infringement on peer-to-peer networks.  These can include traffic monitoring and packet shaping.

There are a number of problems with this plan:

First, the record labels want the colleges to pay money because some of their students are violating record label copyright.  The colleges are stuck in a bad place; because the government has mandated it.  It’s highly likely that these measures will be ineffective, but even if they are effective, the copyright infringers will just move off campus and get residential broadband, leaving all the students paying higher tuition and residency bills because the colleges have to pay for the ineffective enforcement somehow. 

So, who benefits from this law?  The labels?  This won’t stop one copyright infringer, and I think they know that.  What it does do is pass on the costs of ineffective piracy countermeasures from the labels to the colleges?  The colleges don’t get anything – nor do the students, pirates or otherwise. 

Nobody benefits, and most people suffer.  This is a crappy law. 

Additionally, there are significant political and academic freedom issues in requiring a network to monitor the traffic for particular activity.  This is especially discouraging when you consider that the rate of false positives is going to be extremely high: there is no traffic monitoring solution that is smart enough to tell whether a particular MP3 file is legal or illegal.   Any activity that blocks illegal content can easily break legal ones: fair use of the work, public domain, creative-commons, or downloaded by the copyright holder.  (Recently, a smaller record label had its site pulled because it had hosted copyrighted MP3 files – its own.)

So, in short, it’s stupid, ineffective, and has potential unpleasant side effects. How do you best comply with a law that’s so dumb that it borders on ridiculousness?

The only way to fight stupidity is by being clever – find solutions that won’t cost hundreds of thousands of dollars and which do not harm the ability of students, faculty, and alumni to advance the academic mission of the school.  For example, enabling and properly using Cisco Netflow information to make intelligent decisions about QoS policies.  (Labs and offices get more leeway than student residences, local proxies of popular legal BitTorrent files – like WoW patches and Ubuntu releases - on the LAN.) 

Considering that the provisions in the law (and I am not a lawyer, this is not legal advice, and even if I was a lawyer, I probably wouldn’t be a good one anyway) basically amount to: “Do something” about P2P copyright infringement, it’s probably best just to use QoS policies to throttle down (without cutting off) that traffic which looks suspicious.  But err on the side of assuming that the use is legal and for the academic mission. Maybe set up a filter that only goes into effect if the protocol is BitTorrent, and the content is a video file, and the seeders and peers aren’t all from IP addresses that correspond to astrophysics laboratories. 

You could also make use of anomaly detection software to track spikes in MP3 traffic – that is, of course, assuming that the downloading of music files by the 18-25 demographic is in any way considered “anomalous” rather than “status quo…ulous…” Something like that.

David Oliver talks about his experiences running the 24 Hours LeMons race in Houston, and how knowing about network performance helped him optimize his junker.