For months, we’ve been waiting to see what the fallout would be from the sub-prime mortgage crisis.

Apparently, the results are not unlike a hefty bag filled with chili con carne, dropped from the top of a skyscraper. Only instead of a hefty bag, it’s the U.S. economy.

So, as Wall Street explodes like an explosive so explosive it could explode and create a massive explosion, technology turnaround times will probably extend a couple more years as CIOs try to figure out how to use existing tools to solve network management problems and improve performance. How do you do that?

Luckily, there are ways to do that – Cisco routers and switches already have “application-aware” technologies and don’t require any additional purchases – including IP Service Level Agreement (IP SLA), Class Based Quality of Service (CBQoS), and Network Based Application Recognition (NBAR).

Managing Application Response Times with Cisco IP SLA

Now, measuring real application transactions is the most accurate method for measuring response times. But, failing that, you can use Cisco IP SLA to create synthetic transactions. This is not only useful when on an IT budget crunch but can also provide useful data when assessing whether or not to roll out a new application, or measuring a service provider’s SLA edge-to-edge.

IP SLA operates by sending synthetic transactions between two network devices or between a network device and a server. It can be configured to send different types of synthetic transactions based on port, packet size, type of service, and even more advanced characteristics, as is the case with Voice over Internet Protocol (VoIP) tests. When it gets a response, the sender then calculates the response-time metrics appropriate for the test type, and then repeats multiple times.

Some SNMP polling products can collect data automatically, store it in a database, display the results in a GUI, and provide analytical function beyond data collection, such as calculating baselines, displaying trends, and triggering threshold alerts based on collected IP SLA data. There’s also the possibility of simply getting the information from the CLI, but extracting the IP SLA response-time metrics and copying them to a spreadsheet can be difficult and tedious. However, for the extremely budget-conscious, it can be done.

Deploying Quality of Service with Cisco CBQoS

QoS is a blanket term for network policies and practices that help to manage different types of data traffic that share network links. Effectively, QoS determines how different types of traffic, with different priorities, are handled whenever tradeoffs that are likely to impede performance must be made.

Now, within any enterprise, the end-user experience with certain applications will always be more critical than it is with others. Strategies to avoid (or at least manage) congestion could include dropping traffic, adjusting application responses, and building packet queues. CBQoS is one way to do this – and comes with the CBQoS Management Information Base (MIB) to collect statistics about the traffic traversing the router and reports how the QoS configuration is being applied.

Here, an SNMP polling product with application-aware capabilities can get information on input and output QoS class map utilization, drop percentage, and packet counts. It can also get information on pre-versus-post QoS traffic volume, rate, and packet count. It can also point out traffic marked in conformance, in excess, and in violation of defined policies.

Without CBQoS, network managers don’t have a whole lot of evidence to verify that their QoS settings are actually improving network performance – in fact, they may even be inadvertently harming performance. CBQoS prevents network managers from flying blind with QoS deployments. And, like IP SLA, it’s built into Cisco IOS.

Gaining a New Level of Visibility with Cisco NBAR

From within the network device operating system, Cisco NBAR can inspect packets traversing the device and identify the corresponding application – for example, TCP traffic running on port 80 could be labeled as Google, SAP, SharePoint, SalesForce, etc. NBAR can also provide utilization, volume, and rate metrics on a per-application basis relative to the network circuit carrying the traffic.

It’s similar to NetFlow, but NetFlow identifies protocol traffic mixes – not application-layer visibility. NBAR identifies by application – which is important in setting proper QoS policies. And because NBAR is part of Cisco’s IOS, and the data can be collected with an application-aware SNMP poller (which many of you already have), it can be a more cost-effective solution than application discovery hardware.

While on Wall Street, banks were collapsing, IT pros were in New York as well for Interop.

We were a bit concerned, what with the economic downturn and all, as to what would happen with spending in IT in the upcoming year. So NetQoS ran a survey polling 112 respondents who attended Interop New York about how much they would spend on network management disciplines and other IT initiatives in 2009.

Here’s what we found.

A plurality of respondents, 46 percent, said that their spending on network management disciplines would stay the same. Only 15 percent of respondents said that they would spend less on network management disciplines and 39 percent actually said that they would spend more on network management disciplines.

Considering the economic woes on everyone’s mind – that’s pretty huge. And it implies that network management is seen as a necessity rather than a luxury. For example, a plurality of 28 percent of survey respondents indicated that the least likely to see an increase in spending in 2009 was change management. This makes sense: No money, means no new projects, which means no change, and no need for change management. Plus, change management has been a heavy investment area over the past few years so more competency has been built in this discipline at the expense of others.

Overall, 34 percent of survey respondents actually plan to increase overall IT spending in 2009, with 54 percent keeping it the same level. Only 12 percent plan to cut IT spending in 2009.

Does this mean that the economy is better than perhaps we had thought? Unlikely. Instead, what I think this means is that either A) IT is seen as such a vital part of the company that companies aren’t likely to cut corners, B) the corners have already been cut so far that there isn’t much left to cut without hitting something vital, or C) IT is finally starting to make the case that spending there can reduce costs elsewhere in the company.

Look at the big trends in IT: Server virtualization, datacenter consolidation, WAN application development, teleconferencing – all of these are designed to reduce cost. To some extent, IT has always been about leveraging technology to do more with less money, but there’s definitely more of a pronounced emphasis on the “less money” part of that equation than the “do more” part.

If you’re interested, we have a press release about the survey on the NetQoS main Web site.

It is a great selling point for many networking vendors to point out exactly how much money you lose when networks aren’t performing to peak efficiency – and there are real savings from faster round trip application response times.

But as Mark Gibbs at Network World points out, when you start to equate worker productivity to network performance, it gets a little hairy.

The problem with these kinds of analyses is they aren’t identifying real costs because you can’t equate a solid hour of an employee’s time with an hour of his time that’s broken up into chunks of minutes or even seconds over a long period. 

If you are calculating the value of an employee it has to be on the basis of actual productive work done and revenue derived from that work.

So, for example, shaving 2 seconds off login times each day may make people slightly happier, but those two seconds really don’t “add up over time.” In many occupations, it is not the volume of transactions that determines the value the employee brings to the company – the creatives in marketing, the go-getters in sales, the brainiacs in R&D, and the psychos in management typically aren’t affected by the extra minute of time that it takes to log in each month.

This type of mentality – that all employees earn X dollars per second, and any second they are not working costs the company money – is a bit alien to me. And by “alien” I mean the kind of alien that enslaves the human race to make them build statues to their leaders and orbital brainwarp lasers. Yes, work ethic is important. But micro-managed employees are stressed and unhappy, and stressed and unhappy workers make mikstakes.

Of course, if you’re waiting 1 or 2 seconds for transactions that you do repeatedly, that does save time – and this is where latency actually produces a serious problem. Delay, the way a human normally thinks about it is a function of latency times the number of transactions. Focusing on latency is good, but focusing on the end-user experience is better – a lightning fast pipe doesn’t mean much if you’re sending data across it 30 to 40 times more than you have to.

For example, any automated system, like, say, algorithmic trading, will receive tons of value from lowering latency considering that robots, like interns, are well suited for mindless, repetitive tasks because they have no souls.

But back to the point; there are some jobs in the organization where network delay actually does affect productivity. I remember working at a medium sized supermarket retailer in the Northeast straight out of college as a data entry clerk. We used a piece of Java-based database software that was slow as hell – it would take seconds just to switch fields between different pieces of data entry on the same form. If I knew then what I know now, I’d probably say that the problem was that the software was designed for a low-latency LAN and had tons of connections through a higher-latency WAN. This caused tons of problems – what could have taken minutes took hours, what could have taken hours took days.

Of course, back then I was just a data entry clerk. I decided not to bring this up with management, considering that I didn’t want management to think about ways to improve network response times. After all, I had figured out by week three that my employment was preconditioned on management never figuring out that since we were just entering data that someone else printed out from Excel spreadsheets sent in by regional managers, the entire department could have easily been replaced by a very small shell script.

Reprinted from TheDailyWTF.com:

At my company, there's a bit of a wall between Application Development and Network Operations. All "network and network-service related issues" must be reported through the porthole, a.k.a. Helpdesk. Quite often, this leads to interesting results.

"Helpdesk, Jerry speaking."

"Hey Jerry," I said, "this is Paul over in app dev. Our TerraTrade system has a defective ForEx feed that needs to be fixed right away. It's causing a bit of an outage, so if possible, can we open the ticket as 'Urgent'?"

"Not a problem," Jerry responded, "let me just have your name and number, and we'll take care of it."

I gave him a few more details and felt pretty happy that helpdesk was actually helpful. Five minutes after I hung up, an email message came in:

[URGENT] TICKET #71248 HAS BEEN ASSIGNED TO YOU

Not a moment later, my phone rang. I hesitantly picked it up.

"Hello is this Paul," the caller asked before I could even say Hello. I affirmed that it was me.

"Paul," he said, "this is Steve over at help desk. We've got an Urgent trouble ticket that just came in. It's for a, uh, Fortix feed defect. We just wanted to make sure you're on it?"

It took me a few minutes to explain to Steve that he was, in fact, assigning me the defect I had just reported.

Before you laugh too hard at the above story – it’s not that far removed from what many IT departments do daily – play the blame game. The user calls a problem into the help desk, then assigns it to the network. The network calls the helpdesk to tell them that it’s a problem with the server, and the server team calls the help desk to tell them it’s the application. If you’re lucky, someone along that chain will know how to fix the problem, but even if you are lucky, it’s still a lot of wasted time and energy.

This line from Manish Chacko’s article, “God Help the Help Desk” sums it up:

Imagine a man walking into a hospital, saying that he doesn't feel good, and doctors around the country are immediately called in, starting with the cardiologist, who rules out heart trouble. The man is next wheeled to a podiatrist, who rules out any problems with his feet. He's then wheeled to a gynecologist (But I'm a man... Ma'am, I'm a doctor. I think I should make that determination - and only after the tests come back.) If your diagnostic process is trial by error, you're not, technically, diagnosing.

This is why Dr. Jim Metzler recommended time and time again that application development and network operations merge into a single “application delivery” team. The primary job of IT is to deliver an application. Focusing on the performance your single group misses the point – it’s how the applications perform that is most important, and hardware, software, and networking are all part of that performance equation.

I can’t believe I missed this the first time around.

I was so focused on how the online Olympic video was getting through the last mile, that I completely forgot to ask: How the heck are they getting it from Beijing to the U.S.?

Douglas Gourlay at Cisco has been blogging about how NBC’s been using Cisco’s Wide Area Application Services (WAAS) for WAN optimization, so that NBC’s video editors can use three 155Mbps OC-3 pipes, combined and load-balanced (with, of course, Cisco gear) to get the files directly from Beijing. While I’m not 100% sure on “as if they were stored locally,” holds true, it’s clear that WAAS is capable of some amazing stuff – we know because NetQoS has SuperAgent integration on WAAS devices and ACE load balancers. We track stuff like that all the time.

“This reduces operating costs of housing, air travel, transportation, and food. Avoiding 800 airplane trips also supports NBC’s green initiatives for the Olympic Games.”

It also probably makes the video editors a bit grumpy that they didn’t get to go to Beijing.

What I’m curious about is what will happen after the Olympics. Just as Olympic stadiums still stand – and are used – in every host city, I’m wondering if the infrastructure that NBC has to Beijing to deliver high definition video will remain after the Olympics. As China starts to become a new superpower, more news and information is bound to come from Beijing, after all.

And if this can be done for one series of events in one major city, is it that far off from having video-heavy WANs in every city to cover every major event?

While we’ve talked quite a bit about what impact the Olympics may have on an enterprise network’s performance, we haven’t talked much about the performance of the NBC site hosting the live streaming of the Olympics. 

According to Jason Perlow at ZDNet, Limelight networks (which hosts the streaming videos) deployed the videos by going to the public internet by hosting the content more locally – at the ISP.  That means you’re viewing the Olympics through your ISP’s internal network, and the broader internet doesn’t even enter into the connection. 

This is smart thinking, it appears to be working, and by all measures this should be applauded.  Perhaps even duplicated – if you know that multiple employees will download the same content, local hosting on the LAN is preferable to duplicate download streams tying up the more expensive, slower WAN lines.

From the enterprise end of the equation, the fact that Limelight is delivering Olympics video more effectively just means that IT managers cannot count on their servers going down from being unable to handle the demand – IT managers still need to monitor their own networks for performance problems when a big event like the Olympics come up. 

However, it would be wrong to assume that Limelight’s strategy is the only reason why Olympic live-streaming hasn’t slowed to a trickle.

First of all, the site blocks 95.44% of visitors from accessing the content – because it limits the content only to those in the United States.  That’s a lot of people.

Secondly, the site requires Microsoft Silverlight. Most people don’t have Silverlight installed.  Some can’t even install it on their systems.  And there are certainly going to be a quite a few people who just didn’t think installing Silverlight was worth the bother to watch five minutes of Olympic footage they may be mildly interested in. 

And finally – none of the really popular sports are being streamed.  Gymnastics, Women’s Beach Volleyball, Swimming (with the exception of synchronized) and most of the track and field events aren’t available live. So you’re left with judo, fencing, and the decathlon.

So while it is a true technological wonder that the lights have stayed on and the site performs admirably – it is important to recognize that Limelight has not found a magic bullet to deal with extremely high internet video demand. 

by Brian Boyko
Editor, Network Performance Daily

USA Today publishes “CyberSpeak” from columnist and radio talk-show host (not to be confused with “talk radio show host”), Kim Komando. For over a decade now she has been helping people become more comfortable with digital technology and the Internet. She has won the 2007 Gracie Award, and is a journalist I greatly admire.

I give her that introduction, because I’m going to rip her latest USA Today column, entitled “Web Delivers New Worry for Parents: Digital Drugs,” to shreds, turn the shreds into mulch, and turn the mulch into compost.

We all know that music can alter your mood. Sad songs can make you cry. Upbeat songs may give you an energy boost. But can music create the same effects as illegal drugs?

This seems like a ridiculous question. But websites are targeting your children with so-called digital drugs. These are audio files designed to induce drug-like effects.

All your child needs is a music player and headphones.

The article goes on from seizing the “maternal fear gland” by the throat to explain that she’s talking about binaural beats, which supposedly affect your brain waves and give the listener a high not unlike taking a drug. If this sounds familiar, it’s a lot like the plot behind the William Shatner-created “Tekwar” series of novels.

(Continued...)

Continue reading "Won’t somebody think (better) of the children?" »

The Wall Street Journal has an article out about “Why the Olympics Scare Tech Pros.”  But really, should this even be scary anymore? 

We’ve known for quite some time that major cultural events, such as the Olympics, can increase recreational traffic on the network as people tune in to catch sporting events.   These events can generate enough traffic to push many enterprise networks to the limits and adversely affect business-critical application performance.  And NBC plans to stream footage of the Beijing games over the Internet.

There are a number of solutions including QoS policies, limiting bandwidth to certain subnets… I particularly like the approach that Brunswick (the bowling guys) are taking.

[Cathy] McClain [divisional chief information officer at Brunswick] can’t just block streaming videos. Some Brunswick employees, the marketing department for example, have to watch the Olympics for work reasons. And blocking sites doesn’t fit with the company culture. Instead, she’s letting workers do whatever they want. But if the network becomes strained, a message will pop up on employees’ computers asking whether they’re watching the video for work-related reasons, and if not, could they please wait until off-peak hours.

The messages explain that Brunswick is trying to save money and McClain includes her phone number so that anyone who has a question can call for an explanation. And they don’t block the video – they just ask workers if they have to watch right now.

It’s a backlash-free way to protect the network. “My community is polite,” McClain tells us. “They get it.”

So, yes, there needs to be policies in place for this sort of thing. But it’s not like this is any sort of big surprise.  We’ve had four years to prepare for this.  Four.  Years.  And chances are if you’re reading this you know about what streaming video can do to your network if left unchecked, you’ve probably lived through a few March Madnesses and Super Bowls and World Cup and World Series and Shriner Bowls…

Besides, the Olympics are crap.

What?  They are! 

First, and to the chagrin of those guys at Brunswick, there are no bowling events.  They just completely ignore the sport.  How can you even take the Olympics seriously if they don’t include bowling?  We’re talking about a franchise whose winter version has included curling.  Curling is practically the same thing, only colder and with brooms. 

Secondly, the International Baseball Federation (IBAF) is changing the rules of baseball at the last minute.  You can’t do that!  You can’t really even call it baseball if you change the rules.  Call it… I don’t know.  Whinyball. 

And of course there’s the whole China/human rights thing. 

Worst of all, the Olympic games in Beijing is pretty much dominated by sports.  Seriously, someone should talk to their marketing department.  I feel pretty confident based on informal polling of myself and my friends at the Linux User Group, the guys at the comic book store, and my LARP buddies – and they pretty much agree that the Olympics has to have some sort of draw other than sports, because really, who likes watching that stuff? 

Recently, we did an unscientific (and really, I cannot stress that word enough) but real-world test of performance using the Ono plugin for BitTorrent client Vuze/Azureus. Our results were inconclusive.

David Choffnes, the author of the Ono plugin, responded to the test in our comments section of that article:

Regarding your results, it is difficult to run controlled experiments because even when you download the same torrent, you're doing it at different times with necessarily different swarms. My research group's evaluation is not limited by this, and we showed that performance improves *on average*.

Also note that if Ono doesn't find any nearby peers, it can't help your performance. You can see if Ono found nearby peers (and is using them) in the "Ono" plugin view … Also, the plugin's effectiveness is limited by the fact that "only" 180,000 users have installed Ono. The more people use it, the more likely you'll find nearby peers.

One last point -- even when Ono doesn't dramatically improve performance, it encourages better "Internet citizen" behavior. Why transfer data from halfway across the world when you can get the same data and (potentially better) performance from peers nearby? Ono makes it easier to do the latter, which should eventually help everyone using the Internet.

Ono is part of Aqualab, a Northwestern University computer science project researching large-scale distributed computing. Choffnes, a doctoral candidate, will present his findings at SIGCOMM next month, and his paper on the subject can be found here – which is great if you like trigonometric functions in your technical literature.

There’s also a telling paragraph which may explain why we got the results we did for our tests (other than just the random variability of different BitTorrent swarms), instead of a massive throughput boost.

In our analysis, we compare statistics from peers located by Ono (referred to as Ono-recommended peers) to those from all peers selected at random by the BitTorrent protocol, which also includes those located by Ono.

In Network Performance Daily's analysis, we compared statistics from peers located by Ono combined with peers selected at random from the BitTorrent protocol, against only peers selected at random from the BitTorrent protocol.

To determine the cosine similarity value for a peer, Ono must be able to compare its ratio maps with those of other peers. The latter information can be obtained in a number of ways: through direct exchange between peers, from distributed storage and from trackers. Ono currently supports the first two options. With direct exchange, when two peers running the Ono plugin perform their connection handshake, the peers swap ratio maps directly… Though Ono enjoys a large user base, it is still a small fraction of the total BitTorrent population. Thus Ono also attempts to perform DNS lookups on behalf of other peers that it encounters, to determine their ratio maps. This enables Ono to perform biased peer selection over a much larger set of peers, including those not running the Azureus client. From both direct exchange of ratio maps and DNS lookups, our Ono clients locate over 180, 000 peers per day using our CDN-based approach.

When Ono determines that a peer has similar redirection behavior, it attempts to bias traffic toward that peer by ensuring there is always a connection to it, which minimizes the time that the peer is choked. Due to limitations of the Azureus plugin API, we are currently unable to bias other aspects of peer connections, e.g., the bandwidth allocated to each connection.

In addition to Ono, Aqualab also does other projects that are designed for improving Internet performance in a number of other areas. Choffnes’s advisor, Dr. Fabian Bustamante, has been working on "sustainable scalability in distributed systems,” called the 3R project. Many P2P and internet VoIP systems are built in a way that routing is controlled at the application layer, and that in order to identify better paths and faster throughput, the application probes the network environment repeatedly, as the application has no quick way to determine whether a particular peer or node is performing well except by trying to connect to it. The 3R project seeks to decrease probing by re-using the view of the network gathered by long-running, ubiquitous services.

While enterprise networking and Internet networking are two different beasts, performance advances in one usually lead to advances in the other, and with cloud computing promising to make enterprise networking a hybrid of LAN, WAN, and Internet connectivity, these advances remain important.

Series Summary and Editorial
Part One: Interview with Texas State Rep. Joe Driver
Part Two: Interview with Matt Miller, Institute for Justice
Part Three: Interview with Capt. RenEarl Bowie, Texas Private Security Bureau

by Brian Boyko
Editor, Network Performance Daily

We’ve written three stories and conducted three interviews regarding HB2833.  The first was with the author of the law, Texas State Representative Joe Driver, the second with Matt Miller of the Institute for Justice, and the third with Texas Private Security Bureau Captain RenEarl Bowie.

Here is our editorial summary:

HB2833, the law designed to make changes to laws regarding private investigation but has PC and Network techs worried that their work may now be illegal, has caused confusion and worry from normal people doing normal jobs in a normal manner.  Whatever the original intent of the law, it is clear to see from its effects that the law itself is poorly written. 

Ultimately, words like “open to interpretation” and “case-by-case basis” are not words you want to use when describing either the meaning of, or enforcement of, the law.

So, where did things go wrong?  I think the man problem was a key misunderstood concept by Texas State Rep. Driver when he wrote the law.  It is clear from the interview with him that he believes that there is a clear and well defined line between “retrieval of data” and “investigation.”

“’Review, analyze, and investigate’ are the three key words, in my opinion, that drive the need for people to have some kind of license. Because if they're doing some of that, then they don't need to be - it doesn't need to be just anybody able to do that - they need to have somebody that has a security license. But if someone's just retrieving information and providing information for someone who is going to analyze, to use one of the words, then that's just a regular computer repair person.” – Rep. Driver.

But what Rep. Driver simply did not realize is that in the practical realities of IT, no such line exists. Any and every interaction that any IT person has with a computer requires some sort of “review, investigation and analysis,” whether it’s simple troubleshooting or complex network latency optimization. 

I can see where Rep. Driver was going with the law and what his intent was when writing it – rooting around through someone’s Windows Recycle Bin can be just as invasive as rooting around in somebody’s trash. 

But rooting around in the guts of a computer to discover the cause of a malware infection is different from rooting around in the guts of a computer to discover infidelity.  However, instead of making the criteria of “investigation” the purpose and use to which the information could be put, the law makes the criteria the way that the information is stored – “computer-based data not available to the public.”  The end result is that the net was cast too widely. 

Compounding this problem is the interpretation provided by the Texas State Private Security Bureau of the law – a literal one.

“Computer repair or support services should be aware that if they offer to perform investigative services… they must be licensed as investigators” – Texas Private Security Bureau Opinion Summary.

Unlike the law itself, the opinion summary is an unambiguous statement, and while Capt. Bowie may say that the law will be interpreted on a “case-by-case” basis, that is not what is in the official statement of opinion. 

As for the court case brought by the Institute for Justice – unfortunately, the Institute for Justice seems to want to fight this case on Constitutional grounds.  However, that will be a hard sell, as qualifications and licensing are clearly powers that states exercise, from state bar associations for lawyers, and state medical boards for doctors.  If the state of Texas wants to make a PI license a requirement for PC repair techs, it certainly has the power to do so.  It may be absurd, but absurdity is not unconstitutional. 

So, where does that leave technical practitioners like network engineers and PC repair gurus?  As a practical matter, I think most people are going to continue going about this, “business-as-usual” style and make a stink only after the law is enforced on some, most likely unsuspecting, tech somewhere in Texas.

The good news is that I think that it is indeed possible to clarify and change the law through the legislative process – Rep. Driver has stated that he would indeed make changes to the law if it needs clarification or amendment. 

It clearly does.