I really hope that you enjoyed yesterday’s “Geek vs. Wild” post.  Partially because it was really fun to do and if I get enough feedback, I can justify going back out into the woods with Ben to film sequels. 

One of the things that I hope you noticed on “Geek vs. Wild” was that most of the audio came from a lavaliere microphone on Ben; the Sennheiser SK/EK 100 series, specifically. Don’t get me wrong, the audio isn’t perfect, but I like how it turned out. 

I’m an amateur filmmaker in my spare time – you can check out the short documentary films I’ve edited for GeeksAreSexy.net on different inventors – and the one thing that I was totally unprepared for when I got into this hobby was how important audio was to the whole experience.  I probably have about twice as much money sunk away in good audio equipment than I do in actual film and video equipment.  It’s just that important to communicating the story that you’re trying to tell. 

Still, I know I could do better – yesterday, for example, I used the company’s Canon HG10 camera – a nice high definition camera, no doubt about it.  However, one of the features it lacked was a way to do manual gain control.  On my home rig, the Canon HG20, I use both manual gain control and a third party mini-mixer from Juicedlink so that I get fine control over the audio.  So I’ve got lavalieres connected to transmitters connected to mixers connected to the camera. 

But you know what one of the most important parts of the audio setup is?  Connecting a pair of headphones to the camera – the last link in the chain – to make sure that what I’m hearing is what will be on the tape – and to make sure that the quality of the sound meets expectations. It wouldn’t work if I tried to listen in on the mixer, on the wireless transmitter – it’s only how good it is in the camera that makes any difference at all.  If I tested the audio, say, at the mixer, a simple disconnected cable between the mixer and camera would mean that I have no sound at all – but the test would sound crystal clear!

It’s the same way with VoIP and VideoIP – Sevcik and Wetzel over at Network World have written about the importance in the difference between Quality of Experience (QoE) and Quality of Service (QoS) – while good QoS policies can preserve and encourage good QoE, the two are not the same.  Additionally, using standard networking tools, designed for TCP/IP connections, may not be able to provide the type of information that you need to monitor QoE on UDP based VoIP connections.  As Sevcik and Wetzel put it:

At this point we can hear you say: "But my tools monitor packet loss so I am all set." You may think you are because you have a tool that looks for packet drops on network routers. The tool probably monitors for output queue drops on critical links--like where large numbers of links converge or at boundaries where traffic flows from a high speed link (say a LAN link) to a lower speed link (like a WAN link). Although this is useful information, it is not enough.

The reason is that packet loss can occur anywhere along the path. Loss often happens because a half/full duplex mismatch prompts a layer 2 error that remains undetected by the Ethernet collision mechanism because of the mismatch. The packet is dropped because it's incomplete or has a bad checksum so no router queue ever sees it. The router reports that there were no packet drops, but this packet never even got there! A noisy copper line, bad Ethernet cables or even a router interface that is unmonitored can cause similar results. The fact is that even though your tools tell you all is well, the application is still missing data and the voice quality will plummet.

So when detecting problems with VoIP, it’s important to use tools that can monitor Quality of Experience as well as QoS – that detects things like jitter and noise, rather than just “dropped packets.”  This is what makes VoIP such a difficult application to carry over the network. 

The last time we saw Ben Erwin, he was playing Ping Pong while explaining how to leverage Cisco’s NAM. Today, we went out into the woods and filmed his alter ego, “Network Survival Expert Moose Dentabling” out in the Texas wilderness, relying solely on what he can find inside old network gear for survival.

Really, I think it might just be that case of insanity that’s been making the rounds. We really can’t justify this other than we thought it would be cool, and it was 72 degrees with no humidity in the middle of February, so we were willing for any excuse to head out and still “work.”

So please enjoy “Geek Vs. Wild.” You can view it here, or you can check it out in full high definition at the YouTube page.

But while I’m on the subject – or rather off the subject, I wanted to mention a couple of things. First, when I was uploading the video to YouTube, I noticed this gem about a bunch of geeks who incorporated network monitoring (for fault) into their Christmas Lights setup.

Second, you’ll notice in the video that we used the iPhone “Campfire” app. For $.99, it was worth the laugh, but I had a heck of a time getting it onto the iPhone. See, the iPhone only allows you to sync to one computer at a time. You used to be able to hook up an iPod via USB to any computer and then use that to play music directly from the iPod, provided that you had iTunes. Not anymore – if I wanted to install the “Campfire” app, I would have had to delete all the songs on my iPod so that I could “sync” with the new computer.

Why couldn’t I have just bought the application and downloaded it over the 3G network? Because Apple/AT&T prevents the installation of applications over 10MB in size over the 3G network in order to preserve that bandwidth. A smart move, but combined with the sync problem, it created a major headache.

Why did you do that, Apple? Why can’t I use my iPhone at work and at home?

Eventually I did have to erase my iPhone music, which ticked me off, naturally. It just goes to show that you can be overzealous regarding security (in this case, the technology securing me from accessing the music I own), and overzealous regarding network performance and end up with ticked off end users.

By Brian Boyko

I wish my love life were as thrilling and fulfilling as the love life of NetQoS Performance Center.  The charming network monitoring suite just hooked up with EMC Smarts.  They not only hooked up quickly, they’ve started cohabitating: Through the NetQoS Connector for EMC Smarts, you can now view performance data from the EMC Smarts management console.

I’m insanely jealous.

When I wake up alone in the morning, I often find myself wondering what it would be like to settle down and find a nice special someone, or some management console, to settle down with.  Someone who cares for you deeply.  Someone who doesn’t overlook your faults, but rather, does everything they can to help you detect and correct them. 

I mean, I know NetQoS Performance Center is my friend, and I’m happy for him. Really.  I just – you know, I’m just kinda waiting for it to be my turn. Where’s my little network suite baboo?

It just seemed the other day that all NetQoS Performance Center was doing was single-direction, trap-level integration with Smarts.  But it came as a complete shock when NetQoS connector came out with a bi-directional orientation which allows the NetQoS Performance Center to alert the Smarts console to a problem, and allows NOC personnel to have direct access from the Smarts console to appropriate NetQoS performance data for quick analysis and troubleshooting. 

You gotta admit, they make a great pair.  Network operations groups relying only on Smarts sometimes didn’t know about an application slowdown until an end-user calls to complain, and then, since they lacked the performance tools to investigate the problem, it used to get escalated to the engineering group, causing all sorts of troubleshooting delays. Now the performance data is right there and NOC groups can start to get to the source of the problem much quicker.

Matt Sherrod, Vice President of Product Management for NetQoS put it this way: “The NetQoS Connector for EMC Smarts is really a shrewd matchmaker, resulting in a marriage between performance and fault that better arms NOC staff to proactively monitor application performance and take action without escalation or having to learn an entirely new product or discipline.”

I’m really bad at weddings.  I don’t know the protocol, don’t know how to address the hosts, and am really lousy at conversations.  Not so for NetQoS Performance Center and Smarts, who, simply clicked in the right way (with a right click) and now include protocol, host and conversation data for every interface to understand traffic composition across the WAN, and response time break-downs by network, server, or application for every TCP-based application to understand where performance issues originate. 

I think most of my performance issues originate from my lack of self confidence, really. Say – do you know if Smarts has a sister?

Last time Cisco came out with a game to market their routers, I pretended to knock off Yahtzee from Zero Punctuation. This time I’m not going to do that because my fake British accent from Australia stinks.

I will however talk about Cisco’s new Cisco Edge Quest 2 game – designed to market the ASR 9000, as the original Cisco Edge Quest promoted the ASR 1000. CEQ2 is a much better game than its predecessor. In the new game, you go around collecting packets (green squares) on 5 different tracks. You avoid network disruptions (red) and collect speed-ups (blue). If you pick up damaged packets (purple), and have 10 seconds to repair them (orange) otherwise you suffer a network disruption. Network disruptions lower your uptime, and if your uptime is less than 99.999 (it starts at 100.00000), you lose the game.

The mechanics are functionally similar to Audiosurf, and just as Audiosurf is a fun game to play, CEQ2 is as well. As a criticism, however, I’d have to say that CEQ2 may be more fun to play, but it’s less educational.

With the previous CEQ, it at least illustrated different technologies, like “parallel processing” within the metaphor of the game. There’s no metaphor here – it’s just a game. You can call the green squares “packets” instead of “green squares,” but they’re fundamentally without any networking context.

There are interstitial advertisements that try to talk up the ASR 9000, but they’re pretty much skippable; and people will probably skip them even on the first time through. This is mostly because the format that they take, “faux news reports,” typically doesn’t work. Even if it was working, these ads have nothing to do with the actual game itself. It seems like a really, really bad commercial, interrupting the flow of the good time you have with the game.

That said, CEQ2 is probably good for Cisco’s branding. Maybe we should look into making a game of our own someday. Hmm…

The Cisco EnergyWise solution, which we wrote about earlier when it was unveiled at CiscoLive! in Barcelona, is starting to gather some attention.  For example, the Forrester Infrastructure Blog recently wrote about how EnergyWise can change basic assumptions within the ruling theory of IT.  (And yes, I wrote that all out so that I wouldn’t have to write a phrase as clichéd as “paradigm shift.”)

Still, when the shoe fits – the idea of “Green IT” has always been about reducing the amount of power consumption of IT itself; lowering utility costs.  Using IT to reduce the amount of power consumption for the company as a whole is an abrupt step change, advancing and augmenting the capabilities and responsibilities of IT in a revolutionary manner.  (Or… I guess you could call it a “quantum leap” if you had to…)

(Darn.  I’ve been working in this industry long enough that I’m beginning to think in buzzwords.  But I digress.)

Point is that we too often have been thinking of energy efficiency and network performance as – if not inversely proportional, than at least two goals that can interfere with each other – in that sometimes companies may choose to sacrifice one of them in order to improve the other, though most of the time, it’s mostly about finding a happy middle ground. 

But what’s happening with Cisco EnergyWise is that energy efficiency is now being shifted from a concern to a networked application.  The network can become the core of energy efficiency, rather than just a cause of energy consumption. 

This places IT teams in the precarious position of being responsible not just for the computers but for the facilities – heating, cooling, and lighting.  In other words, IT teams have just been given a hell of a lot of – and forgive the bad pun – power. 

Bad pun, yes, but resistance to IT taking over responsibilities for facilities will likely be strong because in an organization responsibility often is power.   When bad times come, and layoffs with them, companies will be more hesitant if someone is the only person in the company, “who knows how to turn on the darn lights…”

In an unrelated note, Network Instruments just released a new filter for use with GigaStor and Observer devices specifically designed to track the Conficker/Downadup worm.  More information on GigaStor can be found here. 

Ben Erwin, Technical Marketing Manager at NetQoS, comes back for another Whiteboard Series video to explain how NetQoS’s integration into the Cisco NAM means that you need fewer data collectors and can lower your costs by leveraging your existing infrastructure.

He also, for some completely non-sequitur reason, plays ping-pong.

A high definition version can be found at the NetQoSVideo YouTube Channel, here.

We talked a bit about the SuperAgent/NAM integration earlier.

As far as enterprise technology companies go, Savvis is probably one of the more interesting ones. They provide managed services for enterprise IT functions, building “IT infrastructure as a service.” (This type of outsourced IT model is more typically found in Europe, though as Savvis shows, there are advantages to using this method in the United States.)

One of the caveats that we often pointed out is the idea that relying on an outsourced IT department means that it’s difficult to make sure that your network is being managed correctly; the corporate equivalent of trusting that your auto mechanic knows what he’s doing. And to that end, we’ve recommended that companies that rely on outsourced IT have some way to independently monitor network performance.

Savvis decided to skip that step and offer network performance monitoring on their “Global Network Solutions” themselves; using NetQoS’s tools. This way, their customers get insight into traffic composition, network utilization, and application performance assurance on Savvis’ MPLS-enabled Application Transport Services.

Or in other words, independent monitoring of network performance on managed IT is now standard on Savvis. It’s kinda like when Saab made seatbelts standard in 1958.

Here’s the obligatory “the economy stinks” paragraph: You can imagine that outsourced IT functions would be appealing for many companies as they refocus on priorities. A good IT department can help you better sell widgets, but if the widget company goes under, the IT department is going to go down with it. IT enables business. IT is extremely important to the business. But IT is not the core focus of the business (unless, of course, you’re Savvis or another outsourced IT provider). Outsourced IT means that companies don’t have to worry (as much) as they would if they tried to operate an internal IT department while figuring out how to make and sell the core product.

Companies do still have to worry that they’re getting the service they paid for, though, and that’s why network monitoring will remain important no matter what solution – insourced or outsourced – companies choose to handle their IT infrastructure.

Is it that time again?  Another media outlet suggests that the Internet is so clogged up with worms, spam, and people who can be broadly classified as “jerks,” that the only solution is to ditch it and come up with something else?

I suppose it is – although this time, it’s John Markoff of the New York Times presenting the argument, which makes things a little hairier. 

Bad enough that there is a growing belief among engineers and security experts that Internet security and privacy have become so maddeningly elusive that the only way to fix the problem is to start over.

What a new Internet might look like is still widely debated, but one alternative would, in effect, create a “gated community” where users would give up their anonymity and certain freedoms in return for safety. Today that is already the case for many corporate and government Internet users. As a new and more secure network becomes widely adopted, the current Internet might end up as the bad neighborhood of cyberspace. You would enter at your own risk and keep an eye over your shoulder while you were there.

Most of this anxiety is regarding the Conficker/Downadup Worm; a worm so deadly that just one drop of it on a dog’s tongue can kill the strongest man.   Okay, no – but seriously, the worm disables Windows Automatic Update, Windows Security Center, Windows Defender, and Windows Error Reporting before downloading and installing additional malware, while creating an HTTP server to distribute itself more effectively to other computer.  Finally, the worm starts a dictionary attack against administrative passwords.  It’s not a superworm, but it is nasty. 

The Conficker worm travels via infected laptops, infected USB flash devices, and any computer that it can guess the Admin password to.

The bad news/good news situation is that any client infected with Conficker will likely experience congestion on the network from malware downloads and uploads; the good news is that this congestion makes the worm easy to detect, with a very specific pattern.  Brett Roberts, of Microsoft New Zealand, pointed out that Conficker:

“… will try every three hours to connect to specific domains over HTTP (‘phoning home’) however, unlike many other worms which use a static list of domains, Conficker’s domain list is dynamically generated by an algorithm which has now been reversed engineered.

“Because of this, it may be possible to identify infected hosts on your network if you’re able to log outbound traffic and then analyse those logs. If you see an entry in your logs for one of your systems connecting to one of these domains, that system may be infected by Conficker.

“You can also use this information to block access to those domains at your network perimeter by adding these domains to any “block lists” you might have.”

With a situation like this, network monitoring and network security overlap – and  the same network management strategies, tactics, and tools are needed to provide both security and performance.

But going back to John Markoff, while you can debate the merits of having an information superhighway which requires drivers licences, does the idea of a “gated community” internet even begin to address these problems?   Conficker is mostly an Intranet, rather than Internet worm; it spreads through USB and corporate networks, which are already much more of a “gated community” than any reform to the Internet will ever be. 

Gated communities on the Internet have tried – and while some have succeeded as business models (Facebook, Myspace,) they cannot be said to be completely secure.  Perhaps the last time we saw a “gated community” model that actually was secure was back during the days of AOL, Compuserve, and Prodigy, which essentially provided all the content internally and did not allow users to generate their own.  Any move towards this model would be counter to the trends of user-created content which have made sites such as Flickr, YouTube, and Digg successes. 

There’s a maxim that any computer or program that prevents it’s users from doing stupid things also prevents it’s users from doing clever things.  I can’t imagine that an Internet that prevents stupidity would encourage intelligence. 

I’d rather risk it out here with the bad guys and the bugs, because among them are the truly brilliant.

In yesterday’s post, we talked about the problem with input/output delay on virtualized desktops, though to be fair, we only really talked about one type of virtualized desktop solution. That is, we talked about the challenges associated with server-side desktop virtualization, where the OS runs on the server and the end-user essentially sits at the end of a dumb terminal.

There is another method, called client-side desktop virtualization where the entire OS is downloaded over the network onto the hard drive (or RAM disc) of the client computer, where input and output are localized to the client computer, which means that you don’t get any added latency to the input and output.

However, this type of setup has its own problems. First, downloading an entire OS at once is likely to create bursts of traffic and spikes of high Internet activity. Specifically, what do you think is going to happen when everybody turns on the computer and logs in at 8:30? Or logs out at 5?

This method might be more applicable for work-from-home telecommuters, who can access the “work computer” from home during work hours, and log-off to use the “family computer” at night. There you run into a last-mile problem – will broadband be sufficient to download the OS each morning, and – more importantly, considering most personal broadband is asynchronous – be able to upload the OS each evening?

Now, none of these criticisms of virtualization – either form – should in any way constitute a criticism of the technology. We hope that someday, we’ll be able to realize the full benefits of virtualized desktops without network performance degradation. But do consider the impact of virtualized desktops on your network when considering whether they’re feasible.

Network World’s Jim Duffy’s latest article, “WAN critical to virtualization’s payoff” has me in knots. 

It’s well known that by virtualizing servers, you can consolidate them more easily in a single data center; but in order to maintain performance, you need high performing, low-latency WAN connections.  We’ve written about this before, and it’s only more relevant with VMWare announcing on Feb. 3^rd that they are providing an Open Source Virtual Desktop client.

But Duffy’s article seems to be focusing on the idea of virtualizing the desktop on backend servers, rather than virtualizing backend servers.  The rest of the article denotes all the challenges of doing so.

Now, there are benefits to virtualizing desktops – key among them the idea that software problems can simply be solved by replacing a user’s virtual desktop with a separate virtual desktop – instead of dispatching someone from IT down to the user’s physical location.  This is a laudable goal.

However, virtual desktops are prone to poor performance over the WAN, because, like video or VoIP, it’s an interactive, latency-sensitive service.  Specifically, let’s look at mouse input.

When you operate the mouse on your computer – feel free to do so right now, (unless you’re reading this on an iPhone, in which case, carry on being smug) -  it may seem like an instantaneous reaction.  You move and the mouse moves with you.  This is not true.  It is an illusion. The output, however, comes so quickly after the input that the human body can’t tell the difference.  This is because the input has to go from the mouse to the computer’s IO port (usually USB), get processed by the CPU, and painted on the screen by the GPU out to the monitor. 

However, in a virtualized desktop, the input goes through the user’s IO port, gets processed by the CPU, and then pumped out to the Ethernet connection to the WAN, is processed by the CPU on the server, is output back over the WAN, which is then processed by the CPU of the user and painted onto the screen via the GPU. 

Point is, you’re adding an entire round trip across the WAN to the process.  So even if there’s no congestion, if the round-trip latency due to distance between the user and the data center is 100ms, you’re adding a tenth-of-a-second delay.  That will probably be noticeable.  And there are other sources of delay on the WAN.

This does not even take into consideration the data involved in pushing video data out to the remote desktop.  Long story short, virtualized desktops are probably a good idea on the low-latency connections that you’d find on the LAN, but trying to operate them over the WAN probably won’t work that well.  Is the benefit of virtualizing the desktop (or of consolidating virtual desktops) so great that you’re willing to suffer through poor end-user experience?

Which means that the premise of the entire article frustrates me.  Yes, it’s obvious that you need to have low-latency performance with your WAN to provide for virtualization – and good performance is a good thing; but at some point you are going to hit the law of diminishing returns.  If you’ve been monitoring your network performance and have the capacity to try to serve remote users virtual desktops in under 150ms, feel free to give it a try, but ultimately, will virtualizing the desktop over the WAN provide the best benefits as a whole? 

What may bear more fruit is to have applications virtualized and sent over the WAN onto existing, local desktops, much like XWindows apps back-in-the-day.