Websites tend to have a love-hate relationship with Slashdot.org.  A link on Slashdot can send a world of traffic to your doorstep.  It can also send so much traffic to your doorstep that your servers melt into piles of tapioca pudding. 

It is with a sense of irony that Slashdot slashdotted itself. 

Uriah Welcome, Sourceforge’s chief network engineer, wrote about the experiences in a Slashdot post.  His explanation is written in the dialect of GeekSpeak known as Engineeresque, so what follows is a paraphrase – the original text can be found on Slashdot.

At around 9pm, Welcome found that there were problems connecting to the site.  He tried to log in remotely, but had difficulty due to the network problems.  When he finally succeeded in logging in, he found that there was a massive amount of traffic, saturating 40Gbit/sec lines. 

The incoming ports showed very little traffic so he ruled out an external cause.  He then looked at the internal switch ports.  From the logs he was able to find out that many of the core switches were at 100% CPU utilization, and that the message had something to do with multicast.  Rebooting the cores did not resolve the CPU utilization problem.

Eventually the problem was solved by process of elimination, turning off cabinet switches one by one until he was able to isolate the problem to a pair of switches.  Shutting the downlink ports of those switches off relieved the problem, at about 10 minutes after 10pm.

In a comment on Slashdot, Maz2331 gave some really good advice about why network engineers have to be in the data center:

It may be strange for those not in the networking field, but when things really go bad, the only place to be is physically in the data center.

That means looking at the LEDs on switches for traffic indications. If you see a single port is spewing a LOT of activity during an outage, disconnect it. No, don't make it "down" but pull the cable out of the port.  Then go downstream and repeat until the potential problem set is reduced to an understandable level.

What really sucks about these kind of outages is that you can't remotely log in to various hosts or switches - you have to pull wires out of ports to break the "spew" that is taking things down.

Well, obviously, you can remotely log-in to fix the problem, as Mr. Welcome did.  You’re just fighting against the same problem just trying to log in that you’re trying to fix by logging in.  It’s not a “can-opener-in-a-can” situation, but its close. 

The comments on the story are actually quite insightful, and are worth checking out – one user talked about his problems implementing a VoIP phone system broke the entire network in the factory.  Of course, another user criticized “armchair networking”. 

Back in September 2007, we wrote an article on Adblock Plus, an extension for Firefox that blocks advertisements from loading on Web pages.  It was one of our hits.

It came after some controversy about a Web developer blocking Firefox because of what he perceived were problems with Ad-block… robbing him of his revenue.  But we argued that enterprises should, in fact, encourage Adblock – maybe even to make it mandatory – because each ad which isn’t served means that there’s less bandwidth taken up, which frees up more bandwidth for other applications.

But we didn’t quantify exactly how much bandwidth blocking ads would actually save.  It took AdblockPlus themselves to actually do so, in a blog post that they entitled: “Anatomy of Ads.” Using Dilbert.com as a test bed, they found that:

• Ads triple the amount of data one has to download (over 450 kB increase).


• When downloading the page the first time (without caching) the delay due to ads is not notable. However, these measurements have been made with a very good internet connection, modem users will see something very different. Also, it doesn’t consider the time for server name resolution (all names were already cached) — resolving nine server names instead of just one usually makes a huge difference.


• Dilbert.com is really good at caching, often you can go back to the main page without contacting any server at all. Unfortunately, ads pretty much destroy this advantage by requiring almost every ad server to be contacted. This means that instead of showing up almost immediately the pages take more than 3 seconds to load.

Dilbert.com also includes 8 remote JavaScript files – so you’re loading eight different servers when you just want the content on one. 

But let’s not kid ourselves – while Adblock might have some performance benefits, it wasn’t invented to improve network performance.  It was invented because Internet ads can be intrusive and abusive.

If you look at the history of mass media advertisements, the trend has more or less become one of increased unobtrusiveness.  First came show sponsorships, then came the 30-second spot – a much less annoying form of advertising.  While there are some exceptions (Head On?) the advertisements of today are as a whole less annoying than early advertisements of the 1950s.

Additionally, the next step – product placement – is a less intrusive form of advertising than commercial breaks.  Certainly, used incorrectly, they can be tacky, but correctly used, they don’t interrupt the flow of the narrative. 

Web ads have defied this tradition – getting more annoying and intrusive, with pop-ups, pop-unders, flash ads, video ads, and the aforementioned eight separate scripts.  Even content providers are frustrated by the fact that advertisers they choose to block on their own sites (because they disagree with the content or the presentation) then do an end-run around those content providers, a move equivalent to theft-of-services.

I don’t want to get on a rant here, but as an advertiser, time is valuable.  The best online ads are usually something that either adds content or information or enhances the original Web page – things like “Whopper Sacrifice.” 

I’ve gone on a rant here.  Point is that until online advertisements are going to respect that they are, in effect, selling product on “borrowed bandwidth,” they’re just going to continued to get blocked. 

Steve Taylor and Larry Hettick at Network World recently talked to the CEO of VBrick systems, which sells video over IP appliances – and suggested that corporate networks should be engineered for video first and data second.  More importantly, he suggested that companies will start to engineer for video first and data second this year – 2009. 

“Last December (in 2008) I met with 30 CIOs and they were all planning for it to happen this year. CIOs view this as a low-cost way to increase productivity. Obviously there are travel cost savings, but increasingly [IPV] is being used for reassuring concerned employees [with improved communications].” Graziani also noted that the increased bandwidth capacity and improvements in video encoding / decoding technology have also contributed to making 2009 the year for a tipping point.

Don’t get me wrong, improvements in video encoding and decoding technology have made better quality video with smaller file sizes and requiring less bandwidth, and the importance of video in corporate communications is increasing.  Combine that with a desire to replace expensive travel with relatively inexpensive teleconferencing during an economic crisis that would make Cthulhu rising from the depths seem like a gleeful distraction, and you can see why the timing’s right. 

But considering that data applications for the network include such things as order processing, record retrieval, and report generation – I still have the feeling that companies will continue to see data as a priority for the foreseeable future.

Now, you could argue that when VoIP came around, it was more important that the CEO’s phone rings when he gets a call than processing X orders in Y milliseconds, when Y+100 milliseconds would do just fine.  But video over IP is, right now, a nice-to-have.

Now, I’m not saying that Video over IP isn’t important and effective at improving productivity and employee communications.  I’m just saying that there are other network aspects that are higher priority.  I mean, if you had to pull the plug on one of these services, which one would you choose?

A) VoIP
B) Data Applications
C) Video IP

I’d choose C, any day.  Granted, this is a false choice – a properly managed network should be able to handle all three.  Again, I’m not taking issue with the use of Video IP, simply the priority assigned to it in Graziani’s remarks.

The difference in priorities may be explained by the idea that there are some businesses out there that are so video-intensive that they might prioritize this differently.  However, we can’t imagine this being more than a small percentage. 

With that said, the leap from a Data/Voice network to a Data/Voice/Video network is not nearly as challenging as the leap from Data-only to Data/Voice.  For one, it’s simply the same problem – a new type of traffic on the network that’s latency sensitive.  The main difference from the networking perspective is that VideoIP is more like VoIP’s big brother, with larger bandwidth needs.  So I do agree with Graziani that we will see more VideoIP in 2009.  The timing’s right, the technology is here.   Now we just need to make sure that we have some way to monitor the new communications traffic. 

Cisco announced Cisco EnergyWise at Networkers Barcelona; a software upgrade that allows users of Cisco Catalyst routers to control the energy consumption of pretty much anything that has an IP address. According to the promotional literature and video, you can limit power based on schedules, while allowing exceptions – for example, if a particular employee swipes an ID card to gain access to the building on a weekend, you can turn on the power only in his office until he signs out.

“Why does my toaster need an IP address?” is no longer a joke.

Rob Aldrich at Cisco’s datacenter blog wrote about it, and much of what he has to say is worth a look. If nothing else, the video is kind of cool.

One of the main benefits to data center consolidation and server virtualization is to decrease the power draw of the company overall – and it’s been argued that IT will have to handle electrical facilities in order to limit power consumption without creating problems with network performance. If facilities and IT do not communicate, you could end up with a real disaster, like, in a worst case scenario, someone trying to be too eco friendly and turning off the air conditioners in the server room over a long weekend.

While energy management via IP may seem like yet another thing – in addition to security, VoIP, and of course, network performance – that the network engineer will have to manage, the cost savings can be substantial over time. The trick is that energy management only occurs on things that have an IP address. Since it’s a pain in the butt to try to subnet every single light bulb, this may be an incentive for IPv6 adoption.

The headline read: “Bill Gates Unleashes Swarm of Mosquitoes on Crowd.”

I blinked, rubbed my eyes, and read it again.

The headline still read: “Bill Gates Unleashes Swarm of Mosquitoes on Crowd.”

It wasn’t the Onion.  This was too weird to be the Onion.  Bill Gates, one of the richest men in the world, and certainly one of the most well known, decided that in order to make a point about empathy, he would release a jar of mosquitoes at a talk he gave at the TED conference on malaria in developing nations.

Now, I’m all for Bill’s charity work; and you have to admire a guy who uses the money he’s made towards good deeds. But I’m a little bit worried that Bill Gates may have finally decided to make the full-fledged leap from eccentric billionaire to evil genius.

Speaking of Microsoft and full-fledged leaps, IT Business edge reported that while users of Windows XP will be eligible to upgrade to Windows 7, it only means that Microsoft will sell you Windows 7 for a reduced price.  XP users will have to reformat their hard drives to use Windows 7.

This is major. 

For various reasons, Vista has not been a popular operating system compared to it’s older brother, XP; in fact, by last summer, Vista had a grand total of 8.8 percent of the market.  XP is at 87.1% of the market.

Though it’s easy to blame Microsoft, there has to be a technical reason why there’s no upgrade path from XP to Windows 7.  You don’t last long as a company if you make things harder for 87.1% of your customer base to upgrade. 

If your company is in the 87.1% - and statistically speaking, that’s more than half – this will probably have a huge impact on companies IT, as companies repeat the processes of reformatting, installing Windows 7, reinstating data, and reinstalling apps for every XP computer.  Perhaps even network engineers may have to put projects on hold as they get drafted when the enterprise decides to upgrade. 

Second – backing up the data on all those computers before the reformat – especially in the post-Enron Sarbanes-Oxley record keeping era – will be crucial and easy to screw up. Additionally, backing up all that data to “the network” will increase the amount of traffic on the LAN. 

So network engineers are going to have to answer some hard questions: Can the hard drives be backed up via the LAN without impacting application performance?  In what ways can we use scheduling and packet prioritization to minimize the impact on application performance?  How much money will the company lose due to degraded performance; would it be cheaper to simply buy external hard drives for everyone in the company?

(And if you don’t know the answers to these questions, shouldn’t you?)

Shamus McGillicuddy, not to be confused with the McGillicuddy Serious Party of New Zealand, recently wrote at Searchnetworking.com about Don Lester, a senior network engineer with Wenatchee Valley Medical Center in Wenatchee, Washington.

Lester used the application response time and network traffic analysis modules of the NetQoS Performance Center to diagnose problems with performance of a medical records application, which “affects patient care because doctors won’t be able to do things they might normally want to do to help patients.”

The article shows our products at their best, and it would be a bit shameless of us to quote the bits we really like.

Shameless and fun!

“I popped up Reporter-Analyzer first because it's one of the quickest ways for me to look at what's going to a location," he said. "I was able to see that there was a whole bunch of traffic going to one of the locations from the workstation patching server."

Through some quick investigation, Lester was able to learn that the medical center's PC technicians were supposed to push out a patch in the middle of the night using their ability to turn on machines remotely for update. But something had gone wrong, and the PCs had never been turned on. Instead, the patching server waited until morning, when users started turning on their computers. The computers started pulling patches, which slowed down the WAN link at the remote location.

The problem at the second location was completely unrelated.

"There were no signs of anything," Lester said. "The link didn't have any significant traffic at all."

He pulled up another NetQoS tool -- SuperAgent, which analyzes TCP transaction -- and saw a high level of retransmission delay occurring at the second WAN link.

"So there was a problem with that circuit," he said. "We had to work with a third party who managed the circuit because it's not something we have a lot of eyes into. We told them to fix it. And we were able to use the same tool to tell when, in fact, they had fixed it and if it was as good as it was before the malfunction…

It kinda gives you a warm, squishy feeling in your heart, doesn’t it?

…As well as maintaining critical application performance, Lester's NetQoS tools control network costs.

"It isn't unusual for us to be negotiating a [WAN] contract based on a tier of service that is defined by the average amount of bandwidth consumed over a given period of time," he said. "There have been times when the vendor in question will inadvertently overstate our usage. When that happens, we will produce usage graphs over whatever time period is appropriate, detailing our actual usage -- and ultimately reclassify our tier level and negotiate a lower price."

Lester said a more common issue that comes up is dealing with application vendors. Often, when there is a performance problem with an application, the vendors will take a number of standard corrective actions to try to solve the problem. If none of these actions works, the application vendor will often blame the network and claim that the medical center needs more bandwidth.

"We are able to tell rather easily using NetQoS tools whether or not we really have a bandwidth issue and can then share that information in a concise graphical format," Lester said. "This not only saves us the cost of unnecessarily upgrading circuits, but it also results in better service to the end user since this re-engages the vendor. The troubleshooting escalates, and the root cause is isolated and repaired."

I think that the article illustrates one of the reasons that NetQoS focuses on performance rather than fault – that is, performance data is more valuable than fault data. You could use a quick and dirty program to check on fault and availability, but for more difficult performance problems, you need more sophisticated monitoring tools.

(I just got this idea for a comedy sketch in which you have a doctor that only checks for availability – all patients are either “alive” or “dead.”)

We contacted Don Lester for his thoughts. He said that:

“I have a lot more than two applications I worry about, but there are two that are more important than all the others. All in all though, it does detail what everyone should hope to have in their environment. Simply the ability to troubleshoot or analyze without having to guess or act on instinct or hunches. It is a lot easier to do this kind of work with solid, factual information than to sit around pinging everything and hoping you stumble onto a smoking gun.”

We just announced that NetQoS Unified Communication Monitor works with Microsoft Office Communications Server 2007 Release 2 [OCS 07 R2] this morning, and while it’s easy to get into the small details of how unified communications applications place great demands on the network, and how to handle those demands, I found myself pausing for a moment.

Where, exactly, are we going with this?

And by “we,” I don’t just mean NetQoS as a company but I mean – Us. The big Us. The human condition.

That is, unified communications applications do place more demands on the network than any other type of data that’s come before it. So, why then, do we even do it in the first place?

It’s because treating voice and video as data to be sent over the network allows us to do more with the communication-as-data than we could with the analog alternatives, even if this makes the network as a whole slightly less effective due to congestion – or if you have good performance monitoring information perhaps no less effective, but perhaps more complex. (We try to simplify the complexity as best we can, by using metrics directly from OCS 07 R2, but the necessities of a mixed communications and data network are simply more complex than the needs of a pure data network alone.)

Everything is becoming binary data, and this process is not likely to stop. To those of earlier generations, the New York Times is a newspaper; to many of the young, the New York Times is a news Web page with video and audio content; text and images being one of many offerings possible. A back-of-the-envelope calculation shows that it would be cheaper, over the course of a year’s subscription, to send every New York Times subscriber a free Kindle E-book reader – at retail prices, no less – and send the newspaper to them digitally, than it costs to print out and deliver all those physical papers to the subscribers.

It doesn’t take long to realize three things: Technology is getting cheaper, paper and distribution is getting more expensive, and the market of people who read the New York Times but will only do so if they have a physical piece of paper are dying out. The Grey Lady will become data, or there will be no Grey Lady.

Every advantage of the physical paper - portability, permanency, and simplicity, is being lost as technology becomes more portable, more permanent, and simpler. Our standards for digital technology to replace the more traditional equivalent are relatively low. Assuming it takes two and a half seconds to locate an article by flipping through the pages, any system that can serve up a Web page in less than 2500ms is an improvement. Even the sheer scale involved – that you would measure the time it would take to find an article in milliseconds rather than seconds – implies an entire quantum leap from the old way of doing things.

Those of us who work closely with the Web – bloggers, Web designers, media professionals – are aware of CSS, which removes content from layout, and RSS, which removes content from context. How far can we be from a society in which all content is completely removed from any sort of context or layout? A society where everything is abstracted? Where you could download the model of a basketball, and print it out on a 3D printer. Or even, if you wish, have the New York Times printed daily on a basketball, if you so chose…

But in that world, where everything is data, network performance suddenly becomes one of the most important things in the world. The bottlenecks once caused by the unfortunate limitations of pure physics suddenly give way to a single bottleneck – that of network performance.

Digitization is an awesome and powerful force… and while it has been mostly beneficial, I think that too often we do not recognize the power of this inexorable tide – this benevolent but gargantuan inevitability.

I don’t know if I’m ready for that world. I’m not sure I want my news to bounce.

An interesting post over at Slashdot pointed me over to a Network World story by Mitchell Ashley about how Windows 7 wouldn’t be a compelling upgrade from Windows XP. The interesting aspect is that Ashley goes on to suggest that perhaps the idea of OSes might become irrelevant if you move to thin clients connected to virtualized computers over the LAN.

The future Simon paints is one where these personal and business computing environments are virtualized onto the same computer, rather than intermingled as they are today. Businesses will deliver virtualized full OS plus apps, and stand-alone virtualized apps, to computers that users own. This maintains the security of corporate data and applications, and allows the business to viably deliver a computing environment they manage on computers they don't. Obviously this vision is in line where Citrix is going with XenServer, XenApp and their newly announced Project Independence, and given my own views about desktop and application virtualization I can see merit in a lot of Citrix's vision.

While we haven't seen much of this yet, I believe it would be wise for Microsoft to continue to improve Windows 7 as an easily virtualized OS and a platform for delivering virtualized applications. Microsoft has partially move Live Essential apps into the cloud. As they move Office and other apps online, the OS becomes thinner and thinner, less bloated with applications that entangle themselves into the registry and Windows folder of Windows 7.

You know what, I might be wrong on this – but I don’t see it happening.

This is not to say that thin clients and virtual desktops don’t have their place, but the problem with virtualizing the desktop is that offloading processing to a datacenter means, necessarily, more traffic on the LAN. Individual applications may be run remotely, sure, but operating systems – or specifically the graphical user interfaces of those operating systems – carry a lot of overheard. (If we were all using command line interfaces, the overhead would, of course, not be nearly as great.)

Additionally, companies have been moving more towards having consolidated servers connected to the end-user via the WAN – by bringing the servers closer to the datacenter, you’re also, in effect, moving the users further away. You can have virtualized desktops saturating your LAN, or virtualized servers saturating a WAN, but it would be extremely unlikely that you could have both on the same WAN.

What seems more likely is the use of virtual servers to serve up specific applications – applications that can be optimized to reduce overhead on the network. I just don’t see that happening with XP, nor Windows 7 – perhaps only the next generation – the Microsoft Cloud OS, perhaps – might be light enough to handle virtualized desktops. Then again, a computer’s what, $300 from Dell nowadays? And are you saving that much if you have to get a thin-client appliance for each user instead of buying a general purpose PC?

There have been some worthwhile attempts to do the WAN equivalent of having a cake and eating it too - a number of WAN Optimization vendors are putting in Windows services on the WAN Optimization blades themselves.  That is, one of the ways they optimize the WAN is to keep traffic off of it – and one of the ways to keep traffic off the WAN is to take care of Windows services on the branch LAN, before it even reaches the WAN.  In a way, it’s sort of the opposite of server consolidation – but since you have to run the blade for WAN optimization purposes anyway, you’re not adding additional hardware or sucking up much more power. 

This is pretty much speculation at this point – but speculation is fun, isn’t it? I’d love to hear your comments on this.

By Patrick Ancipink

So far, so good at Cisco Networkers in Barcelona this week. Despite spirits being a little tempered by the worldwide financial crisis, attendance seems to be quite good and we are noticing a radical shift in what enterprises and service providers are trying to accomplish with their networks.

One topic that’s a bit been very popular with this audience is the concern about VoIP and video quality on converged networks.  So you could say network pros are more jittery here about latency-sensitive UPD applications than they are about the macroeconomic situation.

Heh…

Moving on...

The primary concern about video has shifted from quashing recreational YouTube viewers to ensuring the network can carry video. We spoke with several different companies (with headquarters from Sweden to Qatar) that have requirements to stream video across the WAN as part of their mission. I was involved in several conversations and overheard several more where the main topic was using application-aware network management tools and techniques like IP SLA and how to determine, validate and assure QoS.

I have to say I was pleasantly surprised about the awareness of the Cisco NAM support we announced the day before yesterday. It seems like before the news even hit the wire we had several attendees asking how they could use their NAMs alongside NetQoS. There are some monster implementations of NAM out there so everybody’s happy when you can leverage the existing investment for better application response time monitoring and performance troubleshooting.

Tapas and Rioja are nothing to complain about either.

We recently praised Comcast for coming up with a solution to congestion while remaining platform and application agnostic. To recap: Comcast’s new policy is to lower the class of service on packets sent or requested for those users who are currently using more than 70% of their bandwidth over a 15 minute interval only during times of congestion. 

However, the FCC doesn’t necessarily agree – they’re asking why VoIP calls made on Comcast Internet might be choppy during times of congestion – but Comcast’s “Digital Voice” service won’t be choppy.

There are a number of issues; but ultimately, the reason why Comcast’s Digital Voice service is being treated better than other VoIP services is that the Comcast service travels on Comcast-owned pipes as far as possible before connecting to the general Internet.  Since Digital Voice traffic isn’t considered “Internet Traffic” it won’t matter when determining whether a particular user has been saturating his or her connection during times of network congestion. 

This argument may get Comcast out of trouble as far as net neutrality is concerned, but when making a voice offering distinct from that of the general internet, the FCC may rule that the service is a “telecommunications” rather than an “information” service.  That means being subject to different governmental regulations. 

Comcast isn’t the only ISP looking for ways to prioritize traffic and deal with congestion; Cox communications has announced that Internet users in Kansas and Arkansas who are using “time-sensitive” applications or services (we suspect it’s layman’s speak for “latency-sensitive” purposes like gaming, VoIP and teleconferencing) will get priority over downloading files during congestion periods. 

Though the coverage in TMCnet states that Cox “entered into the Net Neutrality arena” the solution to prioritize traffic seems sufficiently neutral to us – the prioritization is not based upon the particular application but on a class of latency sensitive applications (possibly determined at the transport layer [layer 4] rather than the application layer [layer 7]).

It will be an interesting study to see how the two different measures stack against each other; perhaps we might get some good data on how effectively the different policies tackle the same goal.