Last time Cisco came out with a game to market their routers, I pretended to knock off Yahtzee from Zero Punctuation. This time I’m not going to do that because my fake British accent from Australia stinks.

I will however talk about Cisco’s new Cisco Edge Quest 2 game – designed to market the ASR 9000, as the original Cisco Edge Quest promoted the ASR 1000. CEQ2 is a much better game than its predecessor. In the new game, you go around collecting packets (green squares) on 5 different tracks. You avoid network disruptions (red) and collect speed-ups (blue). If you pick up damaged packets (purple), and have 10 seconds to repair them (orange) otherwise you suffer a network disruption. Network disruptions lower your uptime, and if your uptime is less than 99.999 (it starts at 100.00000), you lose the game.

The mechanics are functionally similar to Audiosurf, and just as Audiosurf is a fun game to play, CEQ2 is as well. As a criticism, however, I’d have to say that CEQ2 may be more fun to play, but it’s less educational.

With the previous CEQ, it at least illustrated different technologies, like “parallel processing” within the metaphor of the game. There’s no metaphor here – it’s just a game. You can call the green squares “packets” instead of “green squares,” but they’re fundamentally without any networking context.

There are interstitial advertisements that try to talk up the ASR 9000, but they’re pretty much skippable; and people will probably skip them even on the first time through. This is mostly because the format that they take, “faux news reports,” typically doesn’t work. Even if it was working, these ads have nothing to do with the actual game itself. It seems like a really, really bad commercial, interrupting the flow of the good time you have with the game.

That said, CEQ2 is probably good for Cisco’s branding. Maybe we should look into making a game of our own someday. Hmm…

The Cisco EnergyWise solution, which we wrote about earlier when it was unveiled at CiscoLive! in Barcelona, is starting to gather some attention.  For example, the Forrester Infrastructure Blog recently wrote about how EnergyWise can change basic assumptions within the ruling theory of IT.  (And yes, I wrote that all out so that I wouldn’t have to write a phrase as clichéd as “paradigm shift.”)

Still, when the shoe fits – the idea of “Green IT” has always been about reducing the amount of power consumption of IT itself; lowering utility costs.  Using IT to reduce the amount of power consumption for the company as a whole is an abrupt step change, advancing and augmenting the capabilities and responsibilities of IT in a revolutionary manner.  (Or… I guess you could call it a “quantum leap” if you had to…)

(Darn.  I’ve been working in this industry long enough that I’m beginning to think in buzzwords.  But I digress.)

Point is that we too often have been thinking of energy efficiency and network performance as – if not inversely proportional, than at least two goals that can interfere with each other – in that sometimes companies may choose to sacrifice one of them in order to improve the other, though most of the time, it’s mostly about finding a happy middle ground. 

But what’s happening with Cisco EnergyWise is that energy efficiency is now being shifted from a concern to a networked application.  The network can become the core of energy efficiency, rather than just a cause of energy consumption. 

This places IT teams in the precarious position of being responsible not just for the computers but for the facilities – heating, cooling, and lighting.  In other words, IT teams have just been given a hell of a lot of – and forgive the bad pun – power. 

Bad pun, yes, but resistance to IT taking over responsibilities for facilities will likely be strong because in an organization responsibility often is power.   When bad times come, and layoffs with them, companies will be more hesitant if someone is the only person in the company, “who knows how to turn on the darn lights…”

In an unrelated note, Network Instruments just released a new filter for use with GigaStor and Observer devices specifically designed to track the Conficker/Downadup worm.  More information on GigaStor can be found here. 

Ben Erwin, Technical Marketing Manager at NetQoS, comes back for another Whiteboard Series video to explain how NetQoS’s integration into the Cisco NAM means that you need fewer data collectors and can lower your costs by leveraging your existing infrastructure.

He also, for some completely non-sequitur reason, plays ping-pong.

A high definition version can be found at the NetQoSVideo YouTube Channel, here.

We talked a bit about the SuperAgent/NAM integration earlier.

As far as enterprise technology companies go, Savvis is probably one of the more interesting ones. They provide managed services for enterprise IT functions, building “IT infrastructure as a service.” (This type of outsourced IT model is more typically found in Europe, though as Savvis shows, there are advantages to using this method in the United States.)

One of the caveats that we often pointed out is the idea that relying on an outsourced IT department means that it’s difficult to make sure that your network is being managed correctly; the corporate equivalent of trusting that your auto mechanic knows what he’s doing. And to that end, we’ve recommended that companies that rely on outsourced IT have some way to independently monitor network performance.

Savvis decided to skip that step and offer network performance monitoring on their “Global Network Solutions” themselves; using NetQoS’s tools. This way, their customers get insight into traffic composition, network utilization, and application performance assurance on Savvis’ MPLS-enabled Application Transport Services.

Or in other words, independent monitoring of network performance on managed IT is now standard on Savvis. It’s kinda like when Saab made seatbelts standard in 1958.

Here’s the obligatory “the economy stinks” paragraph: You can imagine that outsourced IT functions would be appealing for many companies as they refocus on priorities. A good IT department can help you better sell widgets, but if the widget company goes under, the IT department is going to go down with it. IT enables business. IT is extremely important to the business. But IT is not the core focus of the business (unless, of course, you’re Savvis or another outsourced IT provider). Outsourced IT means that companies don’t have to worry (as much) as they would if they tried to operate an internal IT department while figuring out how to make and sell the core product.

Companies do still have to worry that they’re getting the service they paid for, though, and that’s why network monitoring will remain important no matter what solution – insourced or outsourced – companies choose to handle their IT infrastructure.

Is it that time again?  Another media outlet suggests that the Internet is so clogged up with worms, spam, and people who can be broadly classified as “jerks,” that the only solution is to ditch it and come up with something else?

I suppose it is – although this time, it’s John Markoff of the New York Times presenting the argument, which makes things a little hairier. 

Bad enough that there is a growing belief among engineers and security experts that Internet security and privacy have become so maddeningly elusive that the only way to fix the problem is to start over.

What a new Internet might look like is still widely debated, but one alternative would, in effect, create a “gated community” where users would give up their anonymity and certain freedoms in return for safety. Today that is already the case for many corporate and government Internet users. As a new and more secure network becomes widely adopted, the current Internet might end up as the bad neighborhood of cyberspace. You would enter at your own risk and keep an eye over your shoulder while you were there.

Most of this anxiety is regarding the Conficker/Downadup Worm; a worm so deadly that just one drop of it on a dog’s tongue can kill the strongest man.   Okay, no – but seriously, the worm disables Windows Automatic Update, Windows Security Center, Windows Defender, and Windows Error Reporting before downloading and installing additional malware, while creating an HTTP server to distribute itself more effectively to other computer.  Finally, the worm starts a dictionary attack against administrative passwords.  It’s not a superworm, but it is nasty. 

The Conficker worm travels via infected laptops, infected USB flash devices, and any computer that it can guess the Admin password to.

The bad news/good news situation is that any client infected with Conficker will likely experience congestion on the network from malware downloads and uploads; the good news is that this congestion makes the worm easy to detect, with a very specific pattern.  Brett Roberts, of Microsoft New Zealand, pointed out that Conficker:

“… will try every three hours to connect to specific domains over HTTP (‘phoning home’) however, unlike many other worms which use a static list of domains, Conficker’s domain list is dynamically generated by an algorithm which has now been reversed engineered.

“Because of this, it may be possible to identify infected hosts on your network if you’re able to log outbound traffic and then analyse those logs. If you see an entry in your logs for one of your systems connecting to one of these domains, that system may be infected by Conficker.

“You can also use this information to block access to those domains at your network perimeter by adding these domains to any “block lists” you might have.”

With a situation like this, network monitoring and network security overlap – and  the same network management strategies, tactics, and tools are needed to provide both security and performance.

But going back to John Markoff, while you can debate the merits of having an information superhighway which requires drivers licences, does the idea of a “gated community” internet even begin to address these problems?   Conficker is mostly an Intranet, rather than Internet worm; it spreads through USB and corporate networks, which are already much more of a “gated community” than any reform to the Internet will ever be. 

Gated communities on the Internet have tried – and while some have succeeded as business models (Facebook, Myspace,) they cannot be said to be completely secure.  Perhaps the last time we saw a “gated community” model that actually was secure was back during the days of AOL, Compuserve, and Prodigy, which essentially provided all the content internally and did not allow users to generate their own.  Any move towards this model would be counter to the trends of user-created content which have made sites such as Flickr, YouTube, and Digg successes. 

There’s a maxim that any computer or program that prevents it’s users from doing stupid things also prevents it’s users from doing clever things.  I can’t imagine that an Internet that prevents stupidity would encourage intelligence. 

I’d rather risk it out here with the bad guys and the bugs, because among them are the truly brilliant.

In yesterday’s post, we talked about the problem with input/output delay on virtualized desktops, though to be fair, we only really talked about one type of virtualized desktop solution. That is, we talked about the challenges associated with server-side desktop virtualization, where the OS runs on the server and the end-user essentially sits at the end of a dumb terminal.

There is another method, called client-side desktop virtualization where the entire OS is downloaded over the network onto the hard drive (or RAM disc) of the client computer, where input and output are localized to the client computer, which means that you don’t get any added latency to the input and output.

However, this type of setup has its own problems. First, downloading an entire OS at once is likely to create bursts of traffic and spikes of high Internet activity. Specifically, what do you think is going to happen when everybody turns on the computer and logs in at 8:30? Or logs out at 5?

This method might be more applicable for work-from-home telecommuters, who can access the “work computer” from home during work hours, and log-off to use the “family computer” at night. There you run into a last-mile problem – will broadband be sufficient to download the OS each morning, and – more importantly, considering most personal broadband is asynchronous – be able to upload the OS each evening?

Now, none of these criticisms of virtualization – either form – should in any way constitute a criticism of the technology. We hope that someday, we’ll be able to realize the full benefits of virtualized desktops without network performance degradation. But do consider the impact of virtualized desktops on your network when considering whether they’re feasible.

Network World’s Jim Duffy’s latest article, “WAN critical to virtualization’s payoff” has me in knots. 

It’s well known that by virtualizing servers, you can consolidate them more easily in a single data center; but in order to maintain performance, you need high performing, low-latency WAN connections.  We’ve written about this before, and it’s only more relevant with VMWare announcing on Feb. 3^rd that they are providing an Open Source Virtual Desktop client.

But Duffy’s article seems to be focusing on the idea of virtualizing the desktop on backend servers, rather than virtualizing backend servers.  The rest of the article denotes all the challenges of doing so.

Now, there are benefits to virtualizing desktops – key among them the idea that software problems can simply be solved by replacing a user’s virtual desktop with a separate virtual desktop – instead of dispatching someone from IT down to the user’s physical location.  This is a laudable goal.

However, virtual desktops are prone to poor performance over the WAN, because, like video or VoIP, it’s an interactive, latency-sensitive service.  Specifically, let’s look at mouse input.

When you operate the mouse on your computer – feel free to do so right now, (unless you’re reading this on an iPhone, in which case, carry on being smug) -  it may seem like an instantaneous reaction.  You move and the mouse moves with you.  This is not true.  It is an illusion. The output, however, comes so quickly after the input that the human body can’t tell the difference.  This is because the input has to go from the mouse to the computer’s IO port (usually USB), get processed by the CPU, and painted on the screen by the GPU out to the monitor. 

However, in a virtualized desktop, the input goes through the user’s IO port, gets processed by the CPU, and then pumped out to the Ethernet connection to the WAN, is processed by the CPU on the server, is output back over the WAN, which is then processed by the CPU of the user and painted onto the screen via the GPU. 

Point is, you’re adding an entire round trip across the WAN to the process.  So even if there’s no congestion, if the round-trip latency due to distance between the user and the data center is 100ms, you’re adding a tenth-of-a-second delay.  That will probably be noticeable.  And there are other sources of delay on the WAN.

This does not even take into consideration the data involved in pushing video data out to the remote desktop.  Long story short, virtualized desktops are probably a good idea on the low-latency connections that you’d find on the LAN, but trying to operate them over the WAN probably won’t work that well.  Is the benefit of virtualizing the desktop (or of consolidating virtual desktops) so great that you’re willing to suffer through poor end-user experience?

Which means that the premise of the entire article frustrates me.  Yes, it’s obvious that you need to have low-latency performance with your WAN to provide for virtualization – and good performance is a good thing; but at some point you are going to hit the law of diminishing returns.  If you’ve been monitoring your network performance and have the capacity to try to serve remote users virtual desktops in under 150ms, feel free to give it a try, but ultimately, will virtualizing the desktop over the WAN provide the best benefits as a whole? 

What may bear more fruit is to have applications virtualized and sent over the WAN onto existing, local desktops, much like XWindows apps back-in-the-day. 

Websites tend to have a love-hate relationship with Slashdot.org.  A link on Slashdot can send a world of traffic to your doorstep.  It can also send so much traffic to your doorstep that your servers melt into piles of tapioca pudding. 

It is with a sense of irony that Slashdot slashdotted itself. 

Uriah Welcome, Sourceforge’s chief network engineer, wrote about the experiences in a Slashdot post.  His explanation is written in the dialect of GeekSpeak known as Engineeresque, so what follows is a paraphrase – the original text can be found on Slashdot.

At around 9pm, Welcome found that there were problems connecting to the site.  He tried to log in remotely, but had difficulty due to the network problems.  When he finally succeeded in logging in, he found that there was a massive amount of traffic, saturating 40Gbit/sec lines. 

The incoming ports showed very little traffic so he ruled out an external cause.  He then looked at the internal switch ports.  From the logs he was able to find out that many of the core switches were at 100% CPU utilization, and that the message had something to do with multicast.  Rebooting the cores did not resolve the CPU utilization problem.

Eventually the problem was solved by process of elimination, turning off cabinet switches one by one until he was able to isolate the problem to a pair of switches.  Shutting the downlink ports of those switches off relieved the problem, at about 10 minutes after 10pm.

In a comment on Slashdot, Maz2331 gave some really good advice about why network engineers have to be in the data center:

It may be strange for those not in the networking field, but when things really go bad, the only place to be is physically in the data center.

That means looking at the LEDs on switches for traffic indications. If you see a single port is spewing a LOT of activity during an outage, disconnect it. No, don't make it "down" but pull the cable out of the port.  Then go downstream and repeat until the potential problem set is reduced to an understandable level.

What really sucks about these kind of outages is that you can't remotely log in to various hosts or switches - you have to pull wires out of ports to break the "spew" that is taking things down.

Well, obviously, you can remotely log-in to fix the problem, as Mr. Welcome did.  You’re just fighting against the same problem just trying to log in that you’re trying to fix by logging in.  It’s not a “can-opener-in-a-can” situation, but its close. 

The comments on the story are actually quite insightful, and are worth checking out – one user talked about his problems implementing a VoIP phone system broke the entire network in the factory.  Of course, another user criticized “armchair networking”. 

Back in September 2007, we wrote an article on Adblock Plus, an extension for Firefox that blocks advertisements from loading on Web pages.  It was one of our hits.

It came after some controversy about a Web developer blocking Firefox because of what he perceived were problems with Ad-block… robbing him of his revenue.  But we argued that enterprises should, in fact, encourage Adblock – maybe even to make it mandatory – because each ad which isn’t served means that there’s less bandwidth taken up, which frees up more bandwidth for other applications.

But we didn’t quantify exactly how much bandwidth blocking ads would actually save.  It took AdblockPlus themselves to actually do so, in a blog post that they entitled: “Anatomy of Ads.” Using Dilbert.com as a test bed, they found that:

• Ads triple the amount of data one has to download (over 450 kB increase).


• When downloading the page the first time (without caching) the delay due to ads is not notable. However, these measurements have been made with a very good internet connection, modem users will see something very different. Also, it doesn’t consider the time for server name resolution (all names were already cached) — resolving nine server names instead of just one usually makes a huge difference.


• Dilbert.com is really good at caching, often you can go back to the main page without contacting any server at all. Unfortunately, ads pretty much destroy this advantage by requiring almost every ad server to be contacted. This means that instead of showing up almost immediately the pages take more than 3 seconds to load.

Dilbert.com also includes 8 remote JavaScript files – so you’re loading eight different servers when you just want the content on one. 

But let’s not kid ourselves – while Adblock might have some performance benefits, it wasn’t invented to improve network performance.  It was invented because Internet ads can be intrusive and abusive.

If you look at the history of mass media advertisements, the trend has more or less become one of increased unobtrusiveness.  First came show sponsorships, then came the 30-second spot – a much less annoying form of advertising.  While there are some exceptions (Head On?) the advertisements of today are as a whole less annoying than early advertisements of the 1950s.

Additionally, the next step – product placement – is a less intrusive form of advertising than commercial breaks.  Certainly, used incorrectly, they can be tacky, but correctly used, they don’t interrupt the flow of the narrative. 

Web ads have defied this tradition – getting more annoying and intrusive, with pop-ups, pop-unders, flash ads, video ads, and the aforementioned eight separate scripts.  Even content providers are frustrated by the fact that advertisers they choose to block on their own sites (because they disagree with the content or the presentation) then do an end-run around those content providers, a move equivalent to theft-of-services.

I don’t want to get on a rant here, but as an advertiser, time is valuable.  The best online ads are usually something that either adds content or information or enhances the original Web page – things like “Whopper Sacrifice.” 

I’ve gone on a rant here.  Point is that until online advertisements are going to respect that they are, in effect, selling product on “borrowed bandwidth,” they’re just going to continued to get blocked. 

Steve Taylor and Larry Hettick at Network World recently talked to the CEO of VBrick systems, which sells video over IP appliances – and suggested that corporate networks should be engineered for video first and data second.  More importantly, he suggested that companies will start to engineer for video first and data second this year – 2009. 

“Last December (in 2008) I met with 30 CIOs and they were all planning for it to happen this year. CIOs view this as a low-cost way to increase productivity. Obviously there are travel cost savings, but increasingly [IPV] is being used for reassuring concerned employees [with improved communications].” Graziani also noted that the increased bandwidth capacity and improvements in video encoding / decoding technology have also contributed to making 2009 the year for a tipping point.

Don’t get me wrong, improvements in video encoding and decoding technology have made better quality video with smaller file sizes and requiring less bandwidth, and the importance of video in corporate communications is increasing.  Combine that with a desire to replace expensive travel with relatively inexpensive teleconferencing during an economic crisis that would make Cthulhu rising from the depths seem like a gleeful distraction, and you can see why the timing’s right. 

But considering that data applications for the network include such things as order processing, record retrieval, and report generation – I still have the feeling that companies will continue to see data as a priority for the foreseeable future.

Now, you could argue that when VoIP came around, it was more important that the CEO’s phone rings when he gets a call than processing X orders in Y milliseconds, when Y+100 milliseconds would do just fine.  But video over IP is, right now, a nice-to-have.

Now, I’m not saying that Video over IP isn’t important and effective at improving productivity and employee communications.  I’m just saying that there are other network aspects that are higher priority.  I mean, if you had to pull the plug on one of these services, which one would you choose?

A) VoIP
B) Data Applications
C) Video IP

I’d choose C, any day.  Granted, this is a false choice – a properly managed network should be able to handle all three.  Again, I’m not taking issue with the use of Video IP, simply the priority assigned to it in Graziani’s remarks.

The difference in priorities may be explained by the idea that there are some businesses out there that are so video-intensive that they might prioritize this differently.  However, we can’t imagine this being more than a small percentage. 

With that said, the leap from a Data/Voice network to a Data/Voice/Video network is not nearly as challenging as the leap from Data-only to Data/Voice.  For one, it’s simply the same problem – a new type of traffic on the network that’s latency sensitive.  The main difference from the networking perspective is that VideoIP is more like VoIP’s big brother, with larger bandwidth needs.  So I do agree with Graziani that we will see more VideoIP in 2009.  The timing’s right, the technology is here.   Now we just need to make sure that we have some way to monitor the new communications traffic.