Okay, hard decision time.  Do I go grab the very last ticket to see “Watchmen” at the last showing tonight alone, or do I spend a sleepless night tossing and turning, while having nightmares about blue men and conspiracy theorists as I wait to see “Watchmen” with my friends the next day? 

Ha Ha!  Just kidding!  I don’t have any friends!

It should not surprise anyone that I have read Watchmen, that I consider it the most culturally significant work of comic fiction produced since William Hogarth’s “A Rake’s Progress.”  But it also speaks to me on a personal level.  That is, the core themes of “Watchmen” have quite a lot to do with network performance.

Okay, maybe “network performance” isn’t as much on a “personal level” so much as it would be some sort of romantic relationship, or spiritual experience.  But it’s still important.  Anyway…

Who, indeed, watches the watchmen?  For example, carriers claim that by going to MPLS you would get better performance.  But there’s not a lot of data provided by the service providers to verify those performance gains.  Without that ability to quantify how the carrier network is performing, you have no idea of providers are living up to their service level agreements – or whether you’re better off switching to MPLS in the first place.

Additionally, without watching carefully, you can have well meaning IT teams affecting the network – changes which seem trivial to an application developer may actually cause major repercussions to the network traffic - generating and transmitting graphics required for a CAPTCHA, for instance, when previously you were only dealing with a text-based application.

The most extreme case we’re familiar with is when one team of application developers flipped a single flag in the database, making a graphics field available to the user, sending 1 MB worth of graphics per page on the application.   This one “little” change caused network traffic to balloon dramatically over a single night. 

It is not that carriers or application developers are mean or untrustworthy – just that they may not know the impact of what they do on the network; and as such, you should.

This means, of course, monitoring the overall round-trip time from end-to-end on an application by application basis, and monitoring for sudden changes in the network. 

Well, amidst all the bad news about the economy, some good news.  Denise Dubie points out that, assuming that the company in question hasn’t completely failed, “Bear-Stearns” style, high-tech talents such as network engineering remain in demand, according to a poll by Robert Half Technology and Bluewolf.

Network administrators were in demand by 65% of CIOs polled, and while it’s hard to believe, according to Bluewolf, salaries of those with networking expertise will spike in the coming months, with a salary increase of 14%.

Here’s the downside: Project managers will have salary decreases over the next year, according to the same poll.

I know this is a short post, but with so much bad news coming out about the economy, I just wanted to report on something economic that elicited a “Yay!” for once.

Over the past year or so, I have become enraptured with the possibilities of storytelling presented by reality television.  Last night, I wrote an e-mail to a pen-pal about how one reality show explored the themes of trust vs. mistrust, selflessness vs. selfishness, and rationality vs. instinct – like I was analyzing a novel for an English lit course.  And no, I didn’t do so ironically.

I know.  I scare myself too.

So I like television, don’t get me wrong; but Paul Graham is 100% right in his latest essay: “Why TV Lost.” 

It's clear now that even by using the word "convergence" we were giving TV too much credit. This won't be convergence so much as replacement. People may still watch things they call "TV shows," but they'll watch them mostly on computers.

For me, this has already happened.  Over the past year, there have been two shows I’ve watched “live” instead of waiting for the video stream to appear online – “The Mole,” and “The Amazing Race.” The only reason I’ve bothered doing so (and I use that verb because it is a bother) is simply because I want to be able to immediately join in the online conversations and social applications of my fellow fans after air.  To me, the television is the beginning of the whole entertainment experience – and the least essential part.  The most essential part is the network – the interconnections between people that makes everything fascinating.

For most shows, however, I don’t bother turning on the television at all – I watch them on the computer.  I have a 46” HDTV, but 99.404% of the time, it’s acting as a computer monitor – not as a television.  For me and others of my generation, “TV” ceases to lose any meaning – they’re the same bits as text, as pictures, as music, as games, and as computer programs. 

In Graham’s essay, he goes into the details of why the move from TV to computer is irreversible, but it mostly boils down to that A) broadband made video downloads possible, B) video downloads are more convenient, C) computers allow you to connect to other people and converse with them, something television has only ever been able to create a weak facsimile of. 

Now, as far as network performance goes, A and B are the most important.  We expect the ability from broadband connections to get video – whether YouTube or iTunes.  And one of the reasons that video communication is the “next big thing” in the enterprise is that because we’ve had years of experience with video IP through YouTube and other sites.  Video has become so ubiquitous on the Web that blocking video in order to conserve bandwidth often means that information is simply more difficult to get to, and we no longer think of network communication as “VoIP” – we now think in terms of unified communications.

From a network engineer’s standpoint, the technology in the home determines what people expect from technology in the workplace – anything less than that, and to the end-user, “the network is slow.”

It is C, however, that is the most interesting from a sociologist’s standpoint.  Remember how I said that I had become a fan of reality television.  One of the strange things about this genre of entertainment is that unlike the big sitcom star, or the talk show host insulated from people and press through an army of spokesmen and mail-openers, is that many of the “reality stars” and “pseudo-celebrities” are much more open and engaging with the audiences that watch them.  Check out a reality TV forum – stars appearing on TV are there, communicating with the audience directly, using the Internet to create interactivity. 

Decry reality TV as lowbrow if you want, but the shift from the tube to “YouTube” is outright fascinating.  Maybe Reality TV has gotten a foothold because it is extremely cheap to produce compared to other fare, but it, perhaps more than any other genre, starts to get what people want – they want interconnectivity and conversations.

And because people want interconnectivity and conversations, people expect sufficient network performance to telecommunicate – video and audio – with whomever.  The most dreaded foe in any online game is “lag,” because poor network performance kills our ability to communicate.

During the editorial process for this post, the theory was advanced that reality TV succeeds because people will watch whatever is on television, no matter what’s on television.  I don’t think that’s so – and if it might have been true at one time, I do not think that it is true today for those who prefer to get video entertainment via the computer.  We are the “conversation generation,” and we have created a new media to fill our needs.  The question is whether we can continue to keep the network supporting our conversations.

He thinks it sucks.

No, correction. According to his latest post in Network World, he thinks it “Sucks!!!” With three exclamation points.

What is it with NMS that feels like were are [sic] riding in the back seat from Wisconsin to Florida with our stinky second cousin Bert. Anytime, I sit in a vendor meeting and they are trying to hock their NMS off on me I can picture signs for "See Rock City" or "Wall Drug" for the east coast to west coast survivors.

As a blogger for a company that makes network management software, (well, technically, network performance monitoring and management software), I figured I should respond.

But still, NMS promises are kinda like being chased by a [sic] angry Shih Tzu with your friends watching. With networks becoming more and more application layer driven, we need something with a little more power. For example;

- Flow based management is cool but what if I need more then [sic] just the conversation, I need packet capture/inspection THEN correlate those together with a verifiable SLA? Now what?

- 1GB 10GB 40GB 100GB? How do I monitor that? Not with a plain Jane NIC for sure. Heck at 1GB my NIC buffer size is only 64K which is just fair for 100M. Plus add in fragmentation and CPU interrupts and you can see your accuracy goes down fast. What's that? Your are using jumbo frames also...oh man...

Tough call, but I think you’re probably looking at something like a combo of NetQoS SuperAgent to handle the flow data, and NetQoS Gigastor to do the packet capture/inspection and monitor the large links through storing the data so that problems can be examined for a short time after they’ve happened. On a 100GB link, you’re talking hours rather than days, but still.

Now, this isn’t a pitch, (I don’t trust Shih Tzu dogs,) but seriously, we were wondering about the same problems, and didn’t have a solution until we integrated Gigastor into our suite.

But that’s not really the point, the point, Jimmy Ray. (Do you mind if I call you Jimmy Ray?) The point is partially that what you’re saying about network management software doesn’t jive with the numbers we’ve got on our end.

We use maintenance renewal rates as a way to benchmark customer satisfaction. If customers think our product “sucks,” they’ll typically get someone else that doesn’t “suck.” Last quarter, the renewal rate was north of 90%. This is pretty good considering the fact that the economy is so bad that dogs are beginning to worry about inflation.

More than half of these customers own multiple products as well – no single metric is adequate, as you pointed out, but through a combination of metrics, you can get the data you need.

What I’m trying to say, Jimmy Ray, is not that you’re wrong about saying that Network Management Software sucks. What I’m trying to say is that if we do suck, we’re sucking in such a way that it’s a blind spot for us – we thought we were doing an excellent job.

Okay, you know the President’s got some serious chops when you’re out at Fry’s on a Tuesday night, frantically searching for a mini-plug to RCA adapter cable, and everyone – customers and staff, seems to be crowded around the big screen TVs watching a Presidential address.  Love or hate his policies, the guy can draw an audience.

We’re trying not to step on anyone’s toes here and take political positions that could needlessly alienate anyone in the audience that holds strong opinions.  (I, for example, remain a staunch supporter of the McGillicuddy Serious Party, despite the fact that has been completely disbanded.) But this blog is all about network performance, and we’ve talked repeatedly (here, here, here, here, and especially here), about improving U.S. broadband performance and availability.  If a politician comes out and says: “Hey, we need to improve the performance of the Internet,” there shouldn’t be anything wrong with giving them their due. 

Now, is the President’s way of improving broadband the best way to improve broadband?  Hell if I know. 

What I do know is that $7.2 billion of the $787 billion stimulus bill has been allocated for broadband expansion. 

Popular Mechanics has an in-depth article on the broadband improving funds – with some interesting conclusions.  For example, the Pew Research Center found that broadband adoption is around 55 percent, yet the cable industry claims that 92 percent of homes have access to its high-speed internet service. 

Glenn Derene at Popular Mechanics concludes, therefore, that “broadband stimulus in America has less to do with pushing cables out to rural areas than it does with finding a way to make broadband more affordable.”

“The biggest problem with broadband service in America is not a lack of availability, it's a lack of competition. Most users have only one or two options for service, and while prices have come down slightly, they are still relatively high for Americans who feel increasingly pinched. Pew's study found an average monthly broadband bill of $34.50, down 4 percent from the previous year, but it also showed a gradual migration away from cable service, which tends to be faster and more expensive, to cheaper and slower DSL service. So the broad language could end up defining an "underserved" area as simply an area without enough competition to make service affordable”

PM also points out that broadband is important for consumer devices, personal cloud computing (think Google Docs, Mapquest and the like), the filing of governmental forms online (cheaper for the taxpayer than paper forms) the dissemination of news via YouTube and blogs, the access of electronic medical records, the tracking of energy usage, etc.  Ultimately, anyone unable or unwilling to use these services essentially lose out on a first-world lifestyle – and ultimately, as PM puts it…

“…Americans who either cannot afford or are not capable of using reliable, robust Internet connections are at risk of becoming increasingly marginalized as citizens.”

Today was a frustrating day. I was hoping to have a video to show you but the best laid plans…

Essentially, I was thwarted because part of the video would have involved using one of the company’s projectors, and it seemed that the computers I had administrative access to didn’t have sound, and the computers that had sound didn’t grant me administrative access – access I would need to install Macromedia Flash.

(I’m appalled. I mean, why don’t I have root access to all the company’s computers? I mean, I’m the company blogger, for crying out loud. Without me, the company wouldn’t have a blog, and without the blog the company would… uh… erm… maybe have a Google Pagerank of 5 instead of 6? Anyway, the point is, I provide a highly demented demanded service for the company…)

Anyway, we’ll have to reschedule that video. But it does make me realize the relationship between network performance and network security – and that is, that if the application your end-user needs will not run, you’re essentially looking at an effective network performance of zero percent for that application. While there are tons of monitoring solutions for determining up/down status, and one really groovy monitoring solution for application performance, not being able to use the application in the first place is a frustration that probably won’t show up in the NOC, but is there nonetheless.

It also makes me think about the “cry wolf” scenario with security. Look at the dreaded UAC for Windows Vista. I have a Vista box. First thing I did was disable UAC. (Second was install Firefox.) Too often when people interact with computer security, it is preventing them from doing something useful, rather than protecting them from something harmful.

So any time that you can take security away from the forefront and put it in the server, the less people have to interact with it, the less they get pissed off and the less likely they are to ignore warnings when something bad does happen. This, of course, requires keen insight into the behavior of the network and ways to detect anomalous behavior in real-time.

As for my personal problem with computer security – it’s not really that bad. I consider myself lucky that that’s the worst I have to deal with. I’ll just send in an IT request to set up the audio on the projector next week.

Australia drops Internet filtering plans, New Zealand backtracks on S92, and in Sweden, the Pirate Bay is on trial. 

The world is interesting these days.

First, Australia: According to the Sydney Morning Herald, the Australian Government’s plan to introduce mandatory Internet filtering (which we’ve covered previously) has been effectively defeated when independent Senator Nick Xenophon switched to supporting the Opposition/Greens coalition.  And the Herald was not pulling punches on the measure’s failure.

The Communications Minister, Stephen Conroy, has consistently ignored advice from a host of technical experts saying the filters would slow the internet, block legitimate sites, be easily bypassed and fall short of capturing all of the nasty content available online… Even the trials have been heavily discredited… Senator Conroy originally pitched the filters as a way to block child porn but - as ISPs, technical experts and many web users feared - the targets have been broadened significantly since then….

This week, a national telephone poll of 1100 people, conducted by Galaxy and commissioned by online activist group GetUp, found that only 5 per cent of Australians want ISPs to be responsible for protecting children online and only 4 per cent want Government to have this responsibility.

As our interview with Mark Newton last October, a network engineer from Australia, showed, the problems with the filters were twofold; in addition to blocking legitimate content, it also degraded network performance severely. 

One of the problems was the mission creep of the blacklist, which started with “illegal content” but broadened to include mature but legal content, and last week, there was an uproar after it was discovered that an anti-abortion group Web page was placed on the blacklist. 

Meanwhile, across “the ditch,” in New Zealand, the National/ACT/United Future coalition government has delayed action on a piece of legislation known in shorthand as “S92” – Section 92 of the Copyright Amendment Act.  This law would cut off Internet access for people accused of copyright infringement.  The accusation would not need to be proven, and Section 92 contains no punishment for a mistaken or malicious accuser. 

In response, protestors started an “Internet Blackout” protest, with black protest signs, completely black Twitter profile icons, and black Web sites.  When the law was delayed, the protestors returned their icons.  Media 7 News Reporter Russell Brown wrote about the incident on his blog, “Hard News.”

The protest virtually came out of nowhere last week -- it was conceived in a schoolroom in Warkworth on the Saturday, and enacted on the Monday. The mobilisation involved was really remarkable.

The protest was also a success because it has fostered a new voice on copyright and creative issues; one that isn't an industry or technical body. The Creative Freedom Foundation's petition has 12,000 names -- and 8000 of the signatories identify themselves as artists. That's remarkable. It has attracted worldwide attention.

And finally, in Sweden, the Pirate Bay trial is ongoing.  The prosecution had dropped about half of the charges by the second day of the trial, and two days ago, the charges in the case were altered, making the case look more and more likely to go in the Pirate Bay’s favor.

The thing that interconnects all of these cases is that I think this – and I may be wrong here – I think this represents a trend that the public is becoming more informed about the impact of legislation on the Internet’s capabilities and on network performance specifically.  People are becoming informed – and more importantly, active and engaged in Internet issues, to secure a network that performs to the best of their capabilities. 

Of course, I could be wrong. 

I really hope that you enjoyed yesterday’s “Geek vs. Wild” post.  Partially because it was really fun to do and if I get enough feedback, I can justify going back out into the woods with Ben to film sequels. 

One of the things that I hope you noticed on “Geek vs. Wild” was that most of the audio came from a lavaliere microphone on Ben; the Sennheiser SK/EK 100 series, specifically. Don’t get me wrong, the audio isn’t perfect, but I like how it turned out. 

I’m an amateur filmmaker in my spare time – you can check out the short documentary films I’ve edited for GeeksAreSexy.net on different inventors – and the one thing that I was totally unprepared for when I got into this hobby was how important audio was to the whole experience.  I probably have about twice as much money sunk away in good audio equipment than I do in actual film and video equipment.  It’s just that important to communicating the story that you’re trying to tell. 

Still, I know I could do better – yesterday, for example, I used the company’s Canon HG10 camera – a nice high definition camera, no doubt about it.  However, one of the features it lacked was a way to do manual gain control.  On my home rig, the Canon HG20, I use both manual gain control and a third party mini-mixer from Juicedlink so that I get fine control over the audio.  So I’ve got lavalieres connected to transmitters connected to mixers connected to the camera. 

But you know what one of the most important parts of the audio setup is?  Connecting a pair of headphones to the camera – the last link in the chain – to make sure that what I’m hearing is what will be on the tape – and to make sure that the quality of the sound meets expectations. It wouldn’t work if I tried to listen in on the mixer, on the wireless transmitter – it’s only how good it is in the camera that makes any difference at all.  If I tested the audio, say, at the mixer, a simple disconnected cable between the mixer and camera would mean that I have no sound at all – but the test would sound crystal clear!

It’s the same way with VoIP and VideoIP – Sevcik and Wetzel over at Network World have written about the importance in the difference between Quality of Experience (QoE) and Quality of Service (QoS) – while good QoS policies can preserve and encourage good QoE, the two are not the same.  Additionally, using standard networking tools, designed for TCP/IP connections, may not be able to provide the type of information that you need to monitor QoE on UDP based VoIP connections.  As Sevcik and Wetzel put it:

At this point we can hear you say: "But my tools monitor packet loss so I am all set." You may think you are because you have a tool that looks for packet drops on network routers. The tool probably monitors for output queue drops on critical links--like where large numbers of links converge or at boundaries where traffic flows from a high speed link (say a LAN link) to a lower speed link (like a WAN link). Although this is useful information, it is not enough.

The reason is that packet loss can occur anywhere along the path. Loss often happens because a half/full duplex mismatch prompts a layer 2 error that remains undetected by the Ethernet collision mechanism because of the mismatch. The packet is dropped because it's incomplete or has a bad checksum so no router queue ever sees it. The router reports that there were no packet drops, but this packet never even got there! A noisy copper line, bad Ethernet cables or even a router interface that is unmonitored can cause similar results. The fact is that even though your tools tell you all is well, the application is still missing data and the voice quality will plummet.

So when detecting problems with VoIP, it’s important to use tools that can monitor Quality of Experience as well as QoS – that detects things like jitter and noise, rather than just “dropped packets.”  This is what makes VoIP such a difficult application to carry over the network. 

The last time we saw Ben Erwin, he was playing Ping Pong while explaining how to leverage Cisco’s NAM. Today, we went out into the woods and filmed his alter ego, “Network Survival Expert Moose Dentabling” out in the Texas wilderness, relying solely on what he can find inside old network gear for survival.

Really, I think it might just be that case of insanity that’s been making the rounds. We really can’t justify this other than we thought it would be cool, and it was 72 degrees with no humidity in the middle of February, so we were willing for any excuse to head out and still “work.”

So please enjoy “Geek Vs. Wild.” You can view it here, or you can check it out in full high definition at the YouTube page.

But while I’m on the subject – or rather off the subject, I wanted to mention a couple of things. First, when I was uploading the video to YouTube, I noticed this gem about a bunch of geeks who incorporated network monitoring (for fault) into their Christmas Lights setup.

Second, you’ll notice in the video that we used the iPhone “Campfire” app. For $.99, it was worth the laugh, but I had a heck of a time getting it onto the iPhone. See, the iPhone only allows you to sync to one computer at a time. You used to be able to hook up an iPod via USB to any computer and then use that to play music directly from the iPod, provided that you had iTunes. Not anymore – if I wanted to install the “Campfire” app, I would have had to delete all the songs on my iPod so that I could “sync” with the new computer.

Why couldn’t I have just bought the application and downloaded it over the 3G network? Because Apple/AT&T prevents the installation of applications over 10MB in size over the 3G network in order to preserve that bandwidth. A smart move, but combined with the sync problem, it created a major headache.

Why did you do that, Apple? Why can’t I use my iPhone at work and at home?

Eventually I did have to erase my iPhone music, which ticked me off, naturally. It just goes to show that you can be overzealous regarding security (in this case, the technology securing me from accessing the music I own), and overzealous regarding network performance and end up with ticked off end users.

By Brian Boyko

I wish my love life were as thrilling and fulfilling as the love life of NetQoS Performance Center.  The charming network monitoring suite just hooked up with EMC Smarts.  They not only hooked up quickly, they’ve started cohabitating: Through the NetQoS Connector for EMC Smarts, you can now view performance data from the EMC Smarts management console.

I’m insanely jealous.

When I wake up alone in the morning, I often find myself wondering what it would be like to settle down and find a nice special someone, or some management console, to settle down with.  Someone who cares for you deeply.  Someone who doesn’t overlook your faults, but rather, does everything they can to help you detect and correct them. 

I mean, I know NetQoS Performance Center is my friend, and I’m happy for him. Really.  I just – you know, I’m just kinda waiting for it to be my turn. Where’s my little network suite baboo?

It just seemed the other day that all NetQoS Performance Center was doing was single-direction, trap-level integration with Smarts.  But it came as a complete shock when NetQoS connector came out with a bi-directional orientation which allows the NetQoS Performance Center to alert the Smarts console to a problem, and allows NOC personnel to have direct access from the Smarts console to appropriate NetQoS performance data for quick analysis and troubleshooting. 

You gotta admit, they make a great pair.  Network operations groups relying only on Smarts sometimes didn’t know about an application slowdown until an end-user calls to complain, and then, since they lacked the performance tools to investigate the problem, it used to get escalated to the engineering group, causing all sorts of troubleshooting delays. Now the performance data is right there and NOC groups can start to get to the source of the problem much quicker.

Matt Sherrod, Vice President of Product Management for NetQoS put it this way: “The NetQoS Connector for EMC Smarts is really a shrewd matchmaker, resulting in a marriage between performance and fault that better arms NOC staff to proactively monitor application performance and take action without escalation or having to learn an entirely new product or discipline.”

I’m really bad at weddings.  I don’t know the protocol, don’t know how to address the hosts, and am really lousy at conversations.  Not so for NetQoS Performance Center and Smarts, who, simply clicked in the right way (with a right click) and now include protocol, host and conversation data for every interface to understand traffic composition across the WAN, and response time break-downs by network, server, or application for every TCP-based application to understand where performance issues originate. 

I think most of my performance issues originate from my lack of self confidence, really. Say – do you know if Smarts has a sister?