According to a post by Ann Bednarz’ on Network World, employers are beginning to understand that poor application performance can have an interesting impact on the bottom line. As more employees are working outside a central office, poor application performance impacts the productivity of branch offices and telecommuters.

A study by Harris Interactive (commissioned by WAN Optimization vendor Riverbed) suggested that employees that can’t be guaranteed application performance on the road are less likely to work offsite. As each employee who stays in the office costs money (water, electricity, office/cube space, and a parking space), telecommuting can be a powerful cost savings… if productivity can be kept the same.

But people who are frustrated by poor application performance are likely to stay in the office instead of working remotely. This might include those who probably should stay home and work, like, say, sufferers of H1N1. And for those who have no choice but to telecommute, they’re not nearly as productive as they could be.

Among employees surveyed, 40% say they would work offsite more often if their business files or software would load more quickly, and 33% report that accessing business files or software remotely negatively affects their productivity.

Now, there are some things out of company’s control – a bad public Internet connection on the employee’s end can’t be helped. But even so, applications can be optimized not just for in-house use but for WAN use as well, by lowering the amount of round trips the application requires, and by lowering the latency of the Internet connection. The latter is probably more difficult, the former requires recoding the app.

The only problem I have with the article is the title: “Mobile employees want speedy apps.” I thought everyone wanted speedy apps. I know I do.

I wanted to make a clarification regarding a blog post we recently published, called “FCC Weighs In On Network Neutrality.”  In that post, we talked about the FCC plan for network neutrality policies and legislation, and quoted FCC chairman Julius Genachowski extensively when he explained his reasoning for those policies.

What we did not make clear enough was that NetQoS does not specifically endorse those policies or agree with the FCC’s reasoning.  But the same token, NetQoS does not specifically not endorse those policies or not agree with the FCC’s reasoning. 

I know what you’re thinking: that we’re just another corporate company that can’t take a firm stand on a controversial issue.  But that’s not it at all.  The reason we don’t have a stance is that NetQoS isn’t just one person – it’s a collection of people, all with different viewpoints and different ideas of what makes good governmental policy regarding the public Internet.  We have every political viewpoint amongst our ranks: libertarians, neo-conservatives, paleo-conservatives, moderates, liberals, socialists, and one guy who votes for the candidate that looks most like Sam the Eagle. 

I don’t think we did anything wrong, per se, in our coverage, but I do think we could have done a better job.  When the article was written, the primary concern was answering three questions for the readership: What was going to be the FCC’s new policy? Why did the FCC come to its decision? How would the enforcement of these policies impact network performance?

What was in retrospect, misfortunate, was not asking a fourth question: Are there other alternatives, and what are they?

Where we are not ambiguous is on the ideal of a neutral network.  That is, that we believe that the primary goal of any governmental policies towards the public Internet should be those that encourage innovation from all comers and in all possible ways.  Tools such as traffic shaping, if used at all, should be used to preserve performance for everyone.  They should not be used to raise the barriers to entry for new technologies, new competitors, and new ideas. 

But as a collection of individuals, we're just divided on the best way to get there. 

The liberal viewpoint will tend to agree with the FCC’s reasoning for new policies.  If you trust the government’s ability to keep a neutral network more than the market’s ability to keep a neutral network, then in your view, it likely follows that you believe network neutrality policy needs to be drafted and enforced.  

But a libertarian viewpoint would suggest that a free market would be better able to keep network neutrality than the government, if only for the idea that the government can be bribed or intimidated into passing legislation that would make networks non-neutral as easily, or more easily, than they can be convinced to pass legislation that preserves network neutrality.  To the libertarian viewpoint, any governmental stance on network neutrality legislation is a bad idea because even good government policies can be reversed to favor bad outcomes by the next guy to come into the office, and that the market can exert enough pressure on the operators of public Internet gateways to favor a neutral stance. 

A moderate, of course, would suggest that it doesn’t really matter how we preserve a neutral Internet, so long as we end up with one.

So, with that in mind, please understand that the coverage of the issue on the blog will focus on the impact that policies, events, and decisions by all players in this space will have on network neutrality and on network performance.  That our goal will focus on trying to explain how the network will change under different plans, rather than arguing for or against any particular viewpoint.

Except for our unabashedly pro-puppy stance.  Anyone who doesn’t love puppies is a commie mutant traitor.

Here’s an interesting question – What’ll happen to application performance if the Obama healthcare plan passes?

I’m not just talking about the impact it would have on the networks of the medical industries, but across the entirety of the U.S. economy. 

Now, the Obama plan is both controversial and the coverage and interpretations are steeped in misinformation.  I’ve known supporters that believe that the plan will give them a free robot and puppy, and detractors who fear that they’ll have to install a crystal in the palm of their hands that will start blinking when they reach “lastday.”

But whatever the actual result of the plan is, people who think the public option would be sufficient for their needs (and who thought that independently purchased private insurance isn’t) might seriously consider quitting their jobs and starting their own businesses.  And among the many people starting a corner pizza store or barbershop or Spatula City franchise, there’s got to be a few talented people starting their own tech startups delivering cloud apps or Web apps, which have lower barriers to entry than desktop or server application development. 

Some of these applications may have compelling features – so companies may switch over to these Web apps.  Granted, this is already happening today, but the point I’m trying to make is that this single piece of legislation which has nearly nothing to do with networks may cause a very rapid jump in the number of cloud apps you have to support in your organization. Kinda freakonomicsy, but there you go. 

Virtualization is a good news/bad news technology. The good news is that you can consolidate your servers onto one piece of hardware, but the bad news is that you lose visibility into the overall network. Jim Metzler, of Ashton, Metzler & Associates, and Ben Erwin of NetQoS discuss how to preserve visibility into application delivery in this short Whiteboard Series Video

The FCC chairman, Julius Genachowski presented a speech to the Brookings Institution, the centrist think tank this morning, supporting FCC intervention to preserve Net Neutrality.

I’m going to quote a whole bunch from the speech, but you really should see it or read it yourself [PDF], as it’s interesting for those of us interested in the Net Neutrality issue. In fact, it’s a pretty decisive turning point.

“Notwithstanding its unparalleled record of success, today the free and open Internet faces emerging and substantial challenges. We’ve already seen some clear examples of deviations from the Internet’s historic openness. We have witnessed certain broadband providers unilaterally block access to VoIP applications (phone calls delivered over data networks) and implement technical measures that degrade the performance of peer-to-peer software distributing lawful content. We have even seen at least one service provider deny users access to political content.

This is particularly important to companies that have any sort of “work at home” or “work on the road” users using residential broadband connections in order to get their work done – telecommuters and “telepresencers” – to have video meetings with co-workers and clients.

This is also especially important for on the road workers, whose service providers change daily (or even more frequently.) It is bad enough for a salesperson to worry about whether hotel X’s broadband has enough “oomph” to be able to access Web-based applications without undue delay (and indeed, hotel reservations have been made or cancelled based on the availability of broadband.) It’s another thing entirely to be assured of broadband power but find that the hotel’s service provider has blocked or degraded that particular application – something one is not likely to find out until after you check in and try to log on.

And as many members of the Internet community and key Congressional leaders have noted, there are compelling reasons to be concerned about the future of openness.

One reason has to do with limited competition among service providers. As American consumers make the shift from dial-up to broadband, their choice of providers has narrowed substantially. I don’t intend that remark as a policy conclusion or criticism -- it is simply a fact about today’s marketplace that we must acknowledge and incorporate into our policymaking.

A second reason involves the economic incentives of broadband providers. The great majority of companies that operate our nation’s broadband pipes rely upon revenue from selling phone service, cable TV subscriptions, or both. These services increasingly compete with voice and video products provided over the Internet. The net result is that broadband providers’ rational bottom-line interests may diverge from the broad interests of consumers in competition and choice.

AT&T sells phone service and television and Internet, Time Warner sells television and phone service and Internet, Starbucks sells coffee and wireless Internet, which is important considering all those RFC 2324 (HTCPCP/IP) connections. The theory behind regulation is that it will prevent unavoidable conflicts of interest from affecting Internet service.

The third reason involves the explosion of traffic on the Internet. With the growing popularity of high-bandwidth applications, Internet traffic is roughly doubling every two years.

Indeed.

Technologies for managing broadband networks have become more sophisticated and widely deployed. But these technologies are just tools. They cannot by themselves determine the right answers to difficult policy questions -- and they raise their own set of new questions.

For example, deep packet inspection and traffic shaping are highly effective, very powerful tools that can manage broadband networks and preserve the performance of some applications while degrading others. The “difficult policy questions” Genachowski is probably referring to boils down to: “Which applications are preserved and which are degraded, and who gets to decide the answer to that question?” The answer to the latter half, apparently, is the FCC, which is the point of the speech:

In view of these challenges and opportunities, and because it is vital that the Internet continue to be an engine of innovation, economic growth, competition and democratic engagement, I believe the FCC must be a smart cop on the beat preserving a free and open Internet.

One of the other interesting things to take away from the speech is that the FCC is not opposed to specialized, non-neutral Internet service providers designed to cater specifically to individual market segments. If a company were to pop up promising optimization of business-critical cloud computing applications – that would be fine, according to Genachowski’s speech, but that this should only be a supplement, not a replacement, for general purpose broadband Internet.

I also recognize that there may be benefits to innovation and investment of broadband providers offering managed services in limited circumstances. These services are different than traditional broadband Internet access, and some have argued they should be analyzed under a different framework. I believe such services can supplement -- but must not supplant -- free and open Internet access…

The big thing, of course, to take away from the speech is what policy changes the FCC will bring forward. Essentially, Genachowski announced that the “Four Freedoms” articulated by former chairman Michael Powell in 2004 as principles he proposes as enforceable Commission rules, along with two other, additional principles. So the policy of the FCC towards Network Neutrality is:

• Network Operators cannot prevent users from accessing lawful Internet content, applications, and services of their choice.
• Network Operators cannot prohibit users from attaching non-harmful devices to the network.
• (New) - Broadband providers cannot degrade or discriminate lawful traffic over their networks, nor disfavor an Internet service just because it competes with a similar service offered by that provider.
• (New) - Broadband providers must be transparent about network management practices.

There are some caveats, and the FCC will make determinations on a case-by-case basis – in the speech, Genachowski mentioned that “during periods of network congestion, it may be appropriate for providers to ensure that very heavy users do not crowd out everyone else.”

Genachowski also mentioned the possibility of “supplemental” non-neutral managed Internet services (for example, an Internet provider which focuses on cloud computing applications, and degrades everything else to ensure good application performance,) but maintained that these should not be a replacement for general purpose Internet access.

As for the transparency policy, it makes sense. In addition to giving consumers the confidence of knowing that they’re getting the service levels that they paid for, it will allow companies that use cloud applications to gauge whether a particular Internet service is adequate for their needs, in markets with multiple services, it allows one to make informed choices when deciding which service to use, and will help cloud application developers determine whether there is enough broadband infrastructure to develop new features in their products.

(Ed. Note: Here, we have to make clear that NetQoS, and proposed acquiring company CA, makes network monitoring and management solutions that would likely be placed in higher demand if traffic reporting becomes required by the FCC.)

Finally, Genachowski made clear that the policies are designed to curb problems that they are already seeing, not to curb possible problems that non-neutral networks would pose.

“This is not about protecting the Internet against imaginary dangers. We’re seeing the breaks and cracks emerge, and they threaten to change the Internet’s fundamental architecture of openness. This would shrink opportunities for innovators, content creators, and small businesses around the country, and limit the full and free expression the Internet promises. This is about preserving and maintaining something profoundly successful and ensuring that it’s not distorted or undermined. If we wait too long to preserve a free and open Internet, it will be too late.”

What people are saying about the CA/NetQoS Acquisition:

“The value that NetQoS will bring to CA in terms of market leadership in NetFlow solutions as well as NetQoS’s broader capabilities for response time monitoring and packet-based monitoring is substantial…”
-“Net Management Shake-up: CA Acquires NetQoS”, Network World

“Network managers must now focus on how applications perform over the network infrastructures they manage. This trend is written all over CA Inc.'s $200 million acquisition of network and application performance vendor NetQoS Inc.”
- “CA-NetQoS deal: Network management = application performance”, Searchnetworking.com

“The combination of the two is a compelling proposition for network management.”
- “CA Buys NetQoS: A New Network Management Juggernaut?”, Forrester

“The merging of CA and NetQoS assets, said CA, will provide CIOs and network engineers and operations managers better visibility and control of many critical services.”
-“CA Acquires NetQoS for $200 Million”, InformationWeek

Because that’s what it’s about.  Network Management.  For Network Performance. 

So, any advice out there on marketing slogans for the merger?  The best we could come up with is: “We’re the company with a ‘CA+N Do’ attitude towards managing your network for performance.”  And, well, that’s horrible…

WAN Optimization solutions – assuming that they work for the applications you need them to work for – are like magic. Consolidating data centers, from a relativistic standpoint, actually moves users further away, so to consolidate data centers, and lowering costs, WAN performance needs to be good enough for the remote users to do their jobs.

But the irony is that as data centers are becoming more consolidated, users are becoming less consolidated. More people are telecommuting than ever before. (Even if the number of full-time telecommuters has gone down, part-time telecommuters rise). It makes a certain amount of sense – an employee too sick to come into work (and infect others) but not too sick to actually work might file some work from home, or sales teams might file reports from the road.

This creates a problem for most WAN Optimization solutions because most solutions require appliances at both ends of the WAN link. Telecommuters are usually accessing the applications from the public Internet. Software-based WAN optimization controllers (“Soft WOCs”) can do some of the work, but telecommuting requires high-performing broadband as well as optimization solutions.

The way that Soft WOCs work, is essentially to recreate a lightweight version of the client that normally sits at the remote end of the optimized WAN link in the software on the mobile computer. The Soft WOC then optimizes the stream between the telecommuter’s computer and the data center.

The problem is that WAN optimization is less efficient when you have a single user than when you have multiple users on the same stream. First, having multiple users accessing the same data means you can take advantage of caching. Caching is only useful on a Soft WOC link if the same user accesses the same data twice.

Secondly, in a normal optimized WAN link, there is only one TCP stream to worry about – the optimized one, with individual streams recreated only at the two ends of the transaction. Each SoftWOC essentially creates its own stream. For that reason, telecommuting solutions simply aren’t going to give you the same dramatic increase in performance you’d get from more traditional WAN Optimization.

On the other hand, any improvement is still improvement. Just be sure to baseline your performance and see if the value is there before deploying Soft WOC solutions.

While we could continue to talk about the CA’s intention to acquire NetQoS, there are other stories out there that we’d like to cover.  For example, Spawn Labs just announced the “Spawn HD-720,” which is essentially like a Slingbox for console video games. 

For those unfamiliar with the Slingbox, it’s a device designed to work with broadband connected homes that allows you to watch television, streaming from your home cable/satellite/fios/antenna/HTPC/DVR, etc., to a web page you can access from anywhere – including mobile networks.  You can switch between video sources, change channels, or access your DVR’s program selection as well.  Naturally, this requires a large amount of upload bandwidth (more for HD media) and download bandwidth on the other end to match. 

None of this was particularly difficult – SlingBox has been around – and operational – since mid 2006.  The biggest barrier, of course, is throughput.  Broadband simply had to hit a certain penetration point in people’s homes to make SlingBox feasible as a way to watch television content. 

The question is – if you can watch TV through Slingbox, or DVDs through Slingbox, or even PVRs through Slingbox, how come we haven’t been able to play video games through Slingbox?  It’s a matter of latency – and that is, that the Slingbox experience is not affected by high latency.  In fact, you could argue that delays on the order of seconds (rather than the normal measurement, measured in milliseconds) is not even a detriment to the Slingbox experience.  Because video watching is essentially one-way communication, getting all the information within a reasonable time frame is more important than getting most of the information now.

Video gaming is interactive, and with two way, time-sensitive communication (i.e, I press the button to make Mario jump, Mario jumps, and the computer displays Mario jumping… before the goomba runs him over), is sensitive to latency.  In this case, performance in the sense of low latency connections – on both ends of the connection, is more important than high-bandwidth connections. With latency, the controls are sluggish and, quite frankly, too annoying to be fun.  And any game that requires split-second timing, like, say “Splosion man” would be unplayable with too much round trip latency. 

Imagine if you will, that a goomba is heading towards Mario, and the player wants Mario to jump on the goomba.  Human reaction time is about 215 milliseconds.  That’s how long it takes you to press the button.  This is an easy feat for most practiced gamers. 

But now let’s imagine that’s happening on a connection that’s got 100ms of delay each way, making a total of 200ms round-trip latency.

• At 0ms, the goomba appears. It travels at – let’s call it 10 inches per second.   
• At 100 ms, the goomba is 1” closer to Mario.  But only now does the player see the goomba appear. 
• At 315ms, the player finally reacts to the goomba, and presses the jump button.  The player thinks he’s jumped when the goomba was 2.15” away from the start, but in actuality, the goomba was 3.15” away. 
• At 415ms, the computer gets the signal to have Mario jump, and Mario jumps – but he aims for a goomba that’s 2.15” from where it started, and by this time, the goomba is 4.15” inches away.  Mario obviously misses the goomba, but the player doesn’t notice it yet. 
• At 515ms, the player is shown that Mario missed the jump by a country mile. 

Spawnlabs’ new “Spawn HD-720,” which allows you to play the game consoles you have in your home from anywhere in the world, doesn’t eliminate this problem, though they have undoubtedly done everything they can on their end to reduce latency – especially in the time it takes to compress video.  Even then, however, the developers at Spawnlabs will admit to you that “distance matters” and that a fast (as in latency) connection is required for gaming.

"We plan to ship with an average end-to-end latency of approximately 100ms across a local area network, yielding a terrific and natural-feeling game play experience," says the company. "Playing across the Internet will typically add another 25-75ms of latency."

Mario missed the goomba in the example above, but in actual play 100ms is not that bad, and people can get used to the controls by learning to anticipate lag times – kind of like driving a different car with a little looser steering. Too much delay, however, and the game can be unplayable because the control just isn’t responsive. 

This is different from OnLive, by the way, in that Spawnlabs is providing a point-to-point solution using your own gaming hardware doing the backend processing (in other words, your Xbox) while OnLive wants to put gaming in the cloud.  Both require very low latency – better performance – rather than throughput. 

Monitoring your network is crucial to maintaining your network; but the two are obviously not the same. You can have all the data, have it presented in an easy to understand format, run report after report, and it won’t matter if, at the end of the day, the person whose job it is to interpret the data misinterprets it.

If you look for the wrong things – for example, if you’re still primarily concerned with availability rather than latency – you can miss the most important details and come to the wrong conclusions about your network.

It reminds me of this guy, who has analyzed the Mars Rover photos on the JPL Journal Web site, and believes that there is a vast conspiracy at NASA to trample tiny humans (about 5cm in height) under the wheels of the Mars Rover.

“Next three images shows [sic] typical areas on Mars where three sizes of humans and primates live a symbiotic lifestyle. Strangely, the primates appear to be sentient…”

“Next is the Tiny humans [sic] attempt to disable a Mars Rover. The reason; it is the machine that has cause numerous deaths among the smallest Humans who cannot detect or hear the Rover coming.”

“***Warning next 5 images show scenes of death by crushing.*** Americans have Constitutional rights to know this information I have discovered from public posted JPL images…. The second image is gruesome. It shows the Rover has driven through a thickly populated tiny human’s area, killing a great number of them…. We are not at war with them. Someone will answer for these deaths.”

The photos, obviously, contains blurry images of rock formations and dirt, the silhouettes of which may look vaguely human-like in a Rorschach-ian way. Personally, I don’t even think they look vaguely human.

I bring this up because it reminds me of the idea that network data can often be an ink blot test of sorts; if someone’s looking only for availability, they’re simply not going to see the problems that are caused by poorly performing (but still available) applications.

Wearing an eyepatch (I’m having a little eye trouble) has given me new appreciation for the importance of visibility.  There’s some good news coming out on that front; as company Ipoque has released an open-source deep packet inspection client.  It’s slower than commercial offerings – more of a tech demo than anything else, but the idea isn’t so much to provide DPI for the masses as it is to show the masses exactly what DPI is, instead of relying on rumor.  Ipoque doesn’t store or examine information being transmitted, for example, a common fear regarding DPI. 

Now, there are products that do packet capture and analysis – we even sell one for the enterprise IT environment.  But it would be chilling, at the very least, to use a product that inspects the content, rather than protocol, of information being transmitted for University or public ISP internet connections.  Still, knowing what protocols are running at any particular time is very useful for even public and university Internet connections.  If World of Warcraft connections and VoIP took a larger share of the network bandwidth than large Web/FTP downloads and YouTube, it’s an easy choice to focus network improvements on solutions that decrease latency rather than those that would increase latency but improve throughput. 

Of course, all the DPI and network monitoring in the world can’t help some networks. 

For example, Telkom, in South Africa, provides ADSL service.  Apparently, it’s not exactly the speediest service in the world, as Unlimited IT decided to transfer 4GB worth of data over 60 miles in one of two ways – via ADSL, or via carrier USB-stick connected to a carrier pigeon. 

The pigeon took two hours – including the time it took to load the data onto the computer from the USB flash memory stick. During that time, the ADSL transmission was about 4% complete.  It wasn’t even close.

Just doing some math here – 4% of 4GB is 163.84MB… that’s 81.92MB/hr, or 1.365MB/min, or 23.3kBps.  Yep, sounds about right.  That’s one of the major problems with ADSL… the “A” part.  That 23.3kBps is around 186kbps – which is actually not that bad compared to the 256kbps upload speed on most ADSL providers.  But the policy of providing download speeds vastly greater than upload speeds was created in an era where people overwhelmingly downloaded information from the Internet, and large uploads were rare, and usually done by large corporations.  Now, we have YouTube, Flickr, anyone in the world can contribute to open source projects, etc.  Might be time to consider changing those policies.

Or someone could make a ton of money providing an upload-only service to compliment the download-focused services of ADSL and cable providers.  Food for thought.