Noah Shachtman at Wired’s lede is hard to improve on, so I’ll quote him directly.

The Air Force wants a suite of hacker tools, to give it "access" to -- and "full control" of -- any kind of computer there is.  And once the info warriors are in, the Air Force wants them to keep tabs on their "adversaries' information infrastructure completely undetected."

This is why people like me have trouble getting to sleep at night.   The phrase “the military is trying to take over my computer,” is easily dismissed as the rantings of a paranoid delusional conspiracy theorist. It’s another thing when the military says: “We want to take over your computer.”

The program is called “Dominant Cyber Offensive Engagement” and the goal is to – well, in military parlance, the goal is to “Deceive, Deny, Disrupt, Degrade, [or] Destroy” computers deemed by the military to be hostile. 

One of the ways to “degrade” is through military botnets; another goal the armed forces are pushing forward.  Under the theory that the best defense is a good offense, Col. Charles W. Williamson III, (not related to Maj. Charles E. Winchester III, played by David Ogden Stiers for six seasons on M*A*S*H,) wrote in the Armed Forces Journal that “America needs the ability to carpet bomb in Cyberspace.”

It’s not hard to imagine how this could go horribly, horribly wrong for anyone caught in the middle of a “fight” between rival botnets.  Imagine an infected botnet zombie on your network – one whose botmaster, for whatever reason, terrorism, economic disruption, or “teh lulz,” decides to use that computer to attack a computer in the military.  The change from a defensive strategy to a counterattack means that instead of one botnet on your network – you now have two separate botnets.  Furthermore, what’s the likelihood the military botnet will call off the attack if you manage to contain the original botnet?  And of course, with a criminal botnet, you could always kick them off your network without impunity because what they’re doing is illegal.  Interefere with a military botnet and you’re “obstructing the interests of National Security.” 

Either way, both botnets are sending massive amounts of anomalous traffic back and forth – “degrading” performance if it doesn’t just bring the whole enterprise crashing down.

Of course, the military hasn’t been doing that well on cybersecurity defense.  Operation Cisco Raider revealed that over 3,500 counterfeit Cisco network components have been discovered, some of them in military installations. 

I’m going to have to call my doctor and ask him to increase my dose of Ambien.

by Brian Boyko
Editor, Network Performance Daily

Enterprises are seeing more adoption of 10 gigabit Ethernet according to a report by Network Instruments, and reported on their Network Observations blog that nearly one quarter of businesses are implementing 10G networks by the end of the year. The larger the company, the more likely a 10G rollout.

There’s certainly evidence of a trend, but is that evidence of a need-based demand? LAN technology at the gigabit Ethernet level typically has low latency – and I don’t see 10G Ethernet helping with that much if at all. Gigabit Ethernet is still a heck of a lot of bandwidth, especially compared to the bandwidth offered by WAN solutions. In any LAN/WAN/LAN traffic path, it’s almost always the WAN that proves to be the bottleneck.

But it is possible, with large VoIP networks, that you could be overloading the LAN capacity and decide to move to 10G for that reason. This could possibly explain why big companies are more likely to have 10G than smaller companies – because if you’re not hitting the bottleneck on the LAN, 10G doesn’t really help you deliver the applications any faster or effectively.

What I think is more likely is that 10G has hit a price point where it costs about as much to roll out 10G as it does the older technologies. Instead of 10G taking over the market from companies migrating from 1G, instead it seems that when companies choose to build new systems, they’re choosing to build them in 10G instead of 1G.

But again, it comes down to application delivery. And if we’re not delivering applications faster, the question is then asked – is there any application that is not feasible to execute on a 1G network for which a 10G network would be suitable?

Then I remembered that I’m a geek, and I like my toys.

Specifically, when I move into my new apartment next month, I’ll be back on my own router hardware. My current place has Ethernet built in – it’s a feature that saves me $50 a month, but the complex houses its own routers, which I have no capability to port-forward, which means that I can’t set up a remote desktop connection so that I can check on my home computer from work. And looking forward to being able to do that again reminds me that perhaps one of the new applications that could propel an adoption to 10G might be combining virtualization with remote desktop software – that is, making the end users work from their desk computers on a virtualized environment on a server. This means that you get more life out of older but still usable desktop hardware. According to the FAQ from RealVNC, at 100Mbps per connection, “most tasks will be indistinguishable performed remotely from if they were performed locally” Still, 100Mbps fills up a 1Gbps LAN pretty quickly. However, a 10Gb LAN might be able to accommodate this new application.

There are limitations – anything using full screen video or animation (a movie, or a 3-D environment) where there are rapid changes of every pixel will require even more bandwidth before it gets “choppy” – which will probably sink my plans of playing Half Life 2 on my Mac via a remote desktop connection to a PC. But this is certainly one of those “think about it” half baked ideas that may become reality in the near future.

By Ben Erwin

Have you ever read the Surgeon General’s warning on the side of your packet capture probe or appliance?  Look closely and you’ll find it:

SURGEON GENERAL’S WARNING:  PACKET CAPTURE PROBES MAY CAUSE IT BUDGETS TO VANISH AND MAY COMPLICATE PREGNANCY

IT budgets are being needlessly consumed by over indulgence in packet capture and analysis capabilities.  People seem to be somewhat addicted to packet capture.  Why is this happening?

Don’t get me wrong, I’m not anti-packet capture.  Packet capture has an important place in managing application delivery.  That place, however, is the data center.   Applications will occasionally misbehave (even though everyone will blame the network) and you will need some packet capture and analysis capabilities to find out why.  NetQoS provides the capabilities with NetQoS SuperAgent and GigaStor.  SuperAgent’s end to end application response time capabilities can isolate the issue from the network to the application, and GigaStor’s retrospective analysis can help you analyze raw packets for root cause – all from the confines of your data center.

Monitoring the edge is when packet capture becomes hazardous to your IT budget’s health.  While most IT shops have not pushed their application servers to the edge, technologies like MPLS and VoIP have decentralized network communication and increased the need for visibility at the edge. This is when packet capture junkies get out of control.

At $5 to 10K each, putting packet capture probes throughout the edge is simply not worth it. If you manage a large enterprise network, close your eyes and imagine having to deploy, monitor, and manage hundreds of boxes throughout your network just for edge visibility.  What’s my TCO of having all of the probes on your network?  What’s my impact to manpower?  Sure, they work fine for troubleshooting issues in the data center but can I really afford one at every site?  But I still need visibility…is there is a cure to my packet capture addiction? 

As a matter of fact, there is a cure – NetFlow.  Enabling NetFlow on edge routers provides cost-effective application visibility without the need to deploy probes.  Why is it cost-effective?  Because NetFlow itself is essentially free.  It already exists on your existing infrastructure, just waiting for you to enable and collect it. 

While NetFlow is not as granular as packet capture, robust NetFlow reporting capabilities will help you solve almost every issue at the edge.  NetFlow provides several IP layer metrics (including port, IP address, and ToS bit just to name a few) to help you troubleshoot network problems. 

Currently, NetFlow and NetQoS ReporterAnalyzer are monitoring over 250,000 WAN links for IT shops worldwide, and over 65 companies use NetFlow ReporterAnalyzer to monitor at least 1,000 WAN links.  All of these IT shops have blissfully discovered NetFlow is the only cost-effective solution for monitoring the edge.

So if you find yourself trying to scratch the packet capture itch just make sure you’re buying in moderation.  Save yourself a ton of time and money and keep packet capture in the data center.  When it comes to obtaining visibility into the edge, NetFlow is the way to go.

We’ve mentioned numerous times about broadband penetration and speed lagging behind countries more rural and less populated – in other words, countries the U.S. has no excuse lagging behind.

Ars Technica recently put out an article detailing what differences in national broadband policy exist that have enabled other nations to surpass the U.S.’s broadband capability. Japan and France have local loop unbundling – that allows for more competition among ISPs.  They also both deploy fiber instead of copper even if it doesn’t show an immediate profit, and competing ISPs are rolling out new fiber infrastructure instead of just leasing lines. 

Japan, France, Sweden, and Canada all treat broadband as a “core infrastructure element” – that is, it is treated as vital to the functioning of the national economy as good roads, bridges, tunnels, and electrical grids.

In all fairness, the U.S. can claim the same thing.  The U.S. may have no broadband policy, may be looking to traffic shaping to solve problems that would be better addressed by more fiber rollouts (oh, and by the way, there’s a new $800,000 deep packet inspection device on the market today to help service providers monitor and shape traffic), and may be relying on increasingly obsolete technologies – but at least we treat it the same as we do our roads, bridges, tunnels, and electrical grids. 

Which is to say, not very well at all.  The American Society of Civil Engineers gave the United States infrastructure a “D” in 2005, down from a score of “D+” in 2003 – and to fix those problems would require $1.6 trillion over five years.  Since then, not much has been done, according to this CBS video reposted on RawStory.com.

Instead, the government is considering plans to lease highways to private companies – using tolls to provide a “free market” solution to the infrastructure problem – but which will ultimately be a government sanctioned private monopoly over certain sections of blacktop. It is difficult to see how this would improve infrastructure, rather than simply allowing private companies to charge the maximum people will pay for the minimum infrastructure service people will put up with.

So, as far as treating broadband infrastructure like the rest of America’s infrastructure, it seems we already do that.  But what needs to be clear is that broadband infrastructure is infrastructure – that is, it is just as important for the rural area to get good broadband as it was for them to get good roads back during the Eisenhower administration. 

In a macabre way, this limited broadband is good for vendors; if broadband was plentiful there wouldn’t be so great a demand for WAN Optimization tools, for example.  Sure, WAN Optimization is a good idea anyway but it is the high cost of bandwidth that spurs demand forward.  It is becoming harder to maintain performance not just because of the various new demands on the network but also because the infrastructure across the country is simply inadequate – thus the demand for network performance monitoring tools.  Increasing bandwidth doesn’t always solve the network problem but insufficient bandwidth always creates one.

The one thing I hate more than anything else is seeing people get the blame for something that they didn’t do because the people in charge are ignorant about technology. No, wait. Paper cuts. I hate paper cuts more than anything else. But that whole “travesty of justice” thing – very close #2.

This (possibly true, possibly not, definitely plausible) story from The Daily WTF had me ticked off. “Cam” – apparently a pseudonym - had set up a Web cam so he could prove to his bosses that he was working from home instead of just saying he was working from home. But on that day….

During a quick lunch break, Cam got a panicked call from his boss's boss, Ron. "Cam, do you still have your webcam on?"

"Yeah, wh-"

"Turn it off. NOW," he said in all caps over the phone.

"Uh, ok." Cam flicked the switch on the webcam off. "So, why exactly is it so urgen-"

"Can'ttalknowbigproblems-" *click*…

See, it seems that there was a brief but major hiccup in a router somewhere between the bank's data center and their T3 provider, causing a dramatic slowdown in outbound network performance, which rippled out into hundreds of branches and affecting thousands of online banking customers. In the troubleshooting process, the lead network engineer caught wind that Cam had been "streaming live video" over the network, and was going to tell! He complained loudly to Ron that Cam had caused the issues and lost some revenues for the bank in the process. Adding to this theory was the fact that the issue had apparently resolved itself close to the time that Cam turned off his webcam.

One week later, Cam is sitting with his boss Joel to discuss the issue. "Cam, I'm going to need you to sign this disciplinary action report before we file it with HR," Joel said weakly.

Appealing to reason, Cam began, "Joel, you know exactly what happened. You know that all that was coming across the network was a static web page with a new image every so often. I never had more than five HTTP sessions at a time. It would take thousands, if not hundreds of thousands of simultaneous users accessing my web site at the same time to consume the bandwidth that it says I consumed on this report."

"I know," he said as his expression sank. Clearly, he'd fought for Cam and been overruled.

"Besides that," Cam continued, "I'm hosting my site at my house. My upstream connection is capped at 360 kbps. There's literally no physical way that anything I did from my house could even make a dent in our massive T3 lines, even if my upstream connection was 100% saturated!"

"I know," Joel said as his face slipped into his hands. At this point, it dawned on Cam that he was lucky that all that was happening to him was a writeup. It sounded as though upper management would prefer to see him hanged. Still, it was absolutely unfair that he'd be made to take the fall.

"Furthermore," Cam pressed, "what about our QoS policies? Surely internal users browsing external web sites have lower priority than-"

"I know," Joel said again. "Look, I've fought them on this. You know I trust you, and that I know you wouldn't ever — that you couldn't ever — do something like this. I'm saying this as a friend; you're better off just signing this. It's not just you; management is pissed at me now, too. It's not fair, but it's how it is."

In the end, there wasn't much Cam could have done.

Of course, the network engineer who latched onto the “streaming video” theory should have gotten the blame for misdiagnosing the problem using the same kind of “If she weighs the same as a duck, she’s made out of wood and therefore a witch!” logic that can destroy the best laid plans of IT.

Maybe, if the moment the router started having huge performance problems, there was some sort of alert delivered to the network engineer – one detailing the problem, how bad it was, and where it was originating from, that might have helped. Someone might want to look into making something like that. Or – or, bear with me - if there was a way to look at the traffic patterns and Netflow data to see exactly how much bandwidth the Webcam was taking, providing exculpatory evidence for Cam, that might be nice. Someone should get on that.

In the meantime, I’ve just set up my own Network Performance Daily webcam to give you an idea of what my job is like. Hope you enjoy it!

The Network Systems research group of the Max Planck Institute for Software Systems recently published a cool little online tool called Glasnost. It tests for BitTorrent traffic manipulation.

I’m not providing a link to the tool mostly because the institute – recently popular from Slashdot – seems to have been hijacked by malware that is causing pop-up windows to appear. Some of the pop-ups are pornographic – so I wouldn’t go checking out the site at work. Still, the basic idea is pretty damn cool.

In addition to testing for BitTorrent blocking, you can also get a pretty accurate bandwidth and latency reading. I have no idea if this program can be modified to keep WAN service providers honest and get a real measure of latency on a WAN, but the source code has been released to the public and anyone can use it.

But most people will just use it to check to see if their ISP does any BitTorrent traffic manipulation.

There is a bit of irony to the project as well; Glasnost is named after the well known economic and political reforms of Mikhail Gorbachev, the last General Secretary of the Communist Party and de facto last ruler of the Soviet Union. He opened up Russia to criticism from within.

The Planck Institute’s Glasnost has been gathering data on which ISPs are blocking or throttling BitTorrent transmissions. A copy of a map on which that data was plotted is found below – the black dots are tested connections that have no throttling, the red dots are tested connections that have throttling. I think I’ll just let the map speak for itself. (Click on the map for a larger version)

The raw numbers on the site confirm what is on the map. 889 total ISPs were tested. 14 of those had some sort of BitTorrent blocking. 10 of those were located in the United States. That’s 10 out of 199 – or a little over 5%.

The only other countries that have any sort of BitTorrent blocking ISPs are Canada (1 out of 99), Ireland, (1 out of 7), Malaysia (1 out of 2), and Singapore (1 out of 6). All the countries that were part of the former Soviet Union, and tested, came out with no blocking whatsoever.

Glasnost seems to be an appropriate name.

By Ben Erwin

Even the WAN optimization vendors are starting to realize that WAN optimization’s utility is diminished without visibility into the network.

Riverbed’s recent announcement shows that they, (and other WAN optimization and acceleration vendors), are recognizing the dependency between better network management and successful WAN optimization projects. The partner vendors mentioned in the announcement all provide some level of network management capabilities, from network configuration to application delivery monitoring. These partnerships are part of a growing trend since network management capabilities are critical to understanding WAN optimization’s ROI and impact on network performance.

In addition to third-party partnerships, WAN optimization vendors have also developed their own technology to improve integration with network management vendors. Riverbed, Juniper, Cisco, and Blue Coat (Packeteer) have all developed traffic flow export capabilities within their WAN optimization appliances to help customers better understand WAN optimization’s impact on application flows. For example, a large insurance brokerage company exports traffic flow records from Riverbed appliances to NetQoS ReporterAnalyzer to help visualize changes to volume, rate, and bandwidth utilization for every application on the optimized link. This flow export alongside ReporterAnalyzer provides the customer with continued visibility across the link for future troubleshooting, traffic analysis, or capacity planning efforts.

While traffic flow and per application bandwidth utilization information is critical to managing application delivery, it’s only part of the story. The other part is measuring latency of mission critical applications between remote sites and the data center – a more tangible metric of WAN optimization’s ROI and impact on the end-user experience. This measurement can be nearly impossible to obtain since WAN optimization appliances obfuscate application transactions between clients and servers, breaking up the TCP stream.

In order to get around the broken TCP stream problem, we at NetQoS entered into a partnership with Cisco to provide unique technology which measures end-to-end latency over Cisco Wide Area Application Services (WAAS) optimized WAN connections. By integrating our NetQoS SuperAgent technology, WAAS users can get client and server-side data collection and reporting capabilities.

To our knowledge, Cisco and NetQoS currently provide the market’s only solution for accurate latency measurements for client and server communication over optimized links.

WAN optimization is all about improving application delivery. Collecting volume, rate, and bandwidth utilization on optimized applications is only part of the solution. Truly understanding ROI on WAN optimization requires accurately measuring network latency and the end-user experience.

Ben Erwin is technical marketing manager at NetQoS and on Tuesday, May 27, 2008, he will be presenting a Webinar on Building Performance-first Application Delivery Networks with Cisco and NetQoS.

Editorial
by Brian Boyko
Editor, Network Performance Daily

Today (May 6, 2008) the House Telecommunications subcommittee is meeting to debate network neutrality legislation; the ramifications of which are likely to be far reaching and quite controversial no matter what the conclusions are.

Neither this blog’s editorial stance nor the position of NetQoS is to endorse or denounce any particular piece of network neutrality legislation. But that does not mean that there are not deeper issues that the debate over Net Neutrality is part of, nor does that mean that where aren’t some more fundamental truths that we cannot agree on.

Richard Stallman at MIT
--Credit: Sam Ogden

The only problem with this is that this isn’t a software development blog – we deal with networking. So I wrote an essay to him about what I felt were the moral issues behind the network neutrality debate – something I personally feel has a moral component.

I’m printing some of our conversation below. Now, this is not a debate in the classic sense – we agree on many things and disagree on others; and what I am trying to do is not so much to convince our readership of a position, but rather to convince our readership to think about this issue philosophically, and to join into our conversation – whether via the comments section here or elsewhere.

EDITOR BRIAN BOYKO: …Just as you believe that free software is a moral right, I happen to think that effective and efficient networks are a free speech issue. Throughout history, improvements in the quality of life – whether through technology or social activism – have been proportional to the abilities of people to communicate. Europe suffered 1000 years of the dark ages until Arabic scrolls allowed them to recapture the lost wisdom of the Greeks. Technological development booms with every innovation in communication; the telegraph, the telephone, the Internet.

Metcalfe’s law states that the value of a telecommunications network is proportional to the square of the number of users of the system; I believe that Metcalfe’s law can be applied to humanity as a whole – that the value of us as a species is proportional to the square of the number of us who are in communication with each other.

For these reasons I believe that open and effective communication is a fundamental human right. Now, as I believe communication is a human right, the only limit one should have on their ability to communicate should be when that communication harms someone else’s right to communicate.

RICHARD STALLMAN’S RESPONSE: It is hard for me to accept that, as stated, because it would imply that until the 1990s all governments were acting unjustly no matter what they did. That cannot be justice.

I think it that the term "human right" can only properly apply to matters of not hurting other people. Thus, it is abuse of language to speak of the "human right" to have food to eat. I think states have a duty to provide food to the hungry, and more generally, to operate a welfare system to help the poor and disadvantaged. Perhaps we have reached the point where wealthy states also have a duty to provide broadband to everyone. But that is a different kind of duty from that of respecting rights.

It is easy to imagine a situation in which there is insufficient supply of food for everyone to eat. But there cannot be an insufficient supply of freedom of speech to go around.

BRIAN BOYKO: But bandwidth is a limited resource. It is entirely possible for some types of traffic to overwhelm others, and this is not an exaggeration; at NetQoS we see this happening on corporate networks all the time.

Right now, Network Neutrality proponents believe in the idea of a “dumb” network. Yet, this does not reflect the realities of the situation; if UDP traffic (VoIP, Gaming, Streaming Live Media) is on the same pipe as TCP traffic without some sort of limitations on the traffic in place, the UDP traffic will eventually overwhelm the TCP traffic entirely, blocking it out.

The scenario that Net Neutrality opponents trot out of heavy users degrading the quality of communication for light users is entirely plausible. Overhyped, to be sure, but plausible.

On the other hand, this does not in any way make the anti-neutrality position in any way correct. Neutrality detractors often argue for solutions that are worse than the problem. Some want to block certain types of traffic – BitTorrent is seen as the perpetual scapegoat – others want to limit the amount of data that people can download, or charge them more for more data. But data is unlimited. Given enough time and enough reliability, I could download a GNU/Linux distribution over a 2800 baud modem. Data is not the issue; bandwidth – or the amount of data that anyone could download at any one time – is.

RICHARD STALLMAN: I think it is legitimate to give small transfers priority over big ones. I do not understand why UDP traffic would overwhelm TCP traffic, but I have no objection to giving TCP priority over UDP if that is useful -- because anybody could, feasibly, use either one to talk with you.

I also see nothing wrong with charging you as a client more for more bandwidth.

What I object to is that your ISP privileges some sites over others when you, as a client, access them -- either explicitly, or indirectly as a consequence of something else. If your ISP does that, it is not working honestly for you.

As mentioned above; this is not so much an endorsement of any position as a hope that we can start talking about these philosophical issues openly. (If you have any problem with the CAPTCHA, feel free to e-mail me directly at brian.boyko@netqos.com and I will be happy to publish your comments.)

Here's the keynote presentation of Carl Duhnoski, CIO of PSS World Medical, from Symposium 2008.

The Information Technology and Innovation Foundation released their 2008 report comparing countries around the world in broadband access, speed, and price; the United States comes in at 15 out of 30. 

Every year since 2001, the United States has fallen further behind in broadband access.  We are now being beaten by Australia.

Last month, we released an editorial entitled: “U.S. falling behind in broadband; enough is enough.” We ended that column with the following words:

What I’d like to see are articles talking about how Americans are trying to solve the broadband problem – not articles dwelling further about how bad things have gotten. 

I don’t have any panaceas, but if you know of something – or have an idea, feel free to leave a comment below.

The report from the Information Technology and Innovation Foundation actually does propose a number of solutions to increase broadband adoption in the U.S.  Here are some takeaways from the report:

• The United States poor performance is roughly about 25 percent to blame on poor policy, about 75 percent on environmental issues like the distribution of homes in suburbia and exurbia, as well as the very long copper loop lengths that such an arrangement necessitate.


• The culture of Wall Street also plays a large part – Japan’s NTT faces less pressure to continually post quarterly profits; as such they can plan on a more long-term basis.  The United States focus on quarterly performance discourages investment in infrastructure that will not show a profit in 90 days.


• Support at the highest levels of government for broadband correlates positively to broadband adoption, speed, and low pricing.


• Competition between providers both inter and intra-modal usually correlates positively to  broadband adoption, speed, and low pricing – but not always.


• There is an upper ceiling on broadband penetration in the U.S.; as only two thirds of American households have computers, the maximum broadband penetration can be is 66%. 

And here are some of the recommendations:

To encourage the development of broadband infrastructure (supply) in the United States, we recommend that U.S. policymakers take the following steps:

1. Enact more favorable tax policies to encourage investment in broadband networks, such as accelerated depreciation and exempting broadband services from federal, state, and local taxation.

2. Continue to make more spectrum, including “white spaces,” available for next-generation wireless data networks.

3. Expand the Department of Agriculture’s Rural Utilities Service Broadband Program and target the program to places that currently do not have non-satellite broadband available.

4. Reform the federal Universal Service Fund program to extend support for rural broadband to all carriers, and consider providing the funding through a reverse auction mechanism.

5. Fund a national program to co-fund state-level broadband support programs, such as Connect Kentucky or North Carolina e-NC Authority.

6. Promote the widespread use of a national, user-generated, Internet-based broadband mapping system that would track location, speed, and price of broadband.

7. State and local governments should take action to make it easier for providers to deploy broadband services, including making it easier to access rights-of-way.

To encourage the growth of consumer demand for broadband, we recommend that U.S. policymakers take these steps:

8. Support initiatives around the nation to encourage broadband usage and digital literacy.

9. Fund a revitalized Technology Opportunities Program, with a particular focus on the development of nationally scalable Web-based projects that address particular social needs, including law enforcement, health care, education, and access for persons with disabilities.

10. Exempt broadband Internet access from federal, state, and local taxes.

11. Support new applications, including putting more public content online, improving e-government, and supporting telework, telemedicine, and online learning programs.

What are your thoughts on these recommendations?