What do YOU think of the NetQoS Performance Center?

Peter Sevcik and Rebecca Wetzel, analysts with NetForecast, would like to know (and of course we at NetQoS are always looking for feedback). They have published an article about NetQoS for their “App Performance View” blog on NetworkWorld.com. This is the result of a series of blog posts they are writing about “tools that monitor application performance in real-world environments.” They ask for customers to comment about their experience with the NetQoS Performance Center at the end, and we would like to encourage all of you who have experience with any of NetQoS products to respond: What do you think of the NetQoS Performance Center?

You may respond anonymously if you wish, and the comment can be as short or as long as you want to make it. Feedback is encouraged on this blog as well.

And if you are just thinking about deploying any of our products, check out the customer comments already posted at the end of the Network World post!

For months, we’ve been waiting to see what the fallout would be from the sub-prime mortgage crisis.

Apparently, the results are not unlike a hefty bag filled with chili con carne, dropped from the top of a skyscraper. Only instead of a hefty bag, it’s the U.S. economy.

So, as Wall Street explodes like an explosive so explosive it could explode and create a massive explosion, technology turnaround times will probably extend a couple more years as CIOs try to figure out how to use existing tools to solve network management problems and improve performance. How do you do that?

Luckily, there are ways to do that – Cisco routers and switches already have “application-aware” technologies and don’t require any additional purchases – including IP Service Level Agreement (IP SLA), Class Based Quality of Service (CBQoS), and Network Based Application Recognition (NBAR).

Managing Application Response Times with Cisco IP SLA

Now, measuring real application transactions is the most accurate method for measuring response times. But, failing that, you can use Cisco IP SLA to create synthetic transactions. This is not only useful when on an IT budget crunch but can also provide useful data when assessing whether or not to roll out a new application, or measuring a service provider’s SLA edge-to-edge.

IP SLA operates by sending synthetic transactions between two network devices or between a network device and a server. It can be configured to send different types of synthetic transactions based on port, packet size, type of service, and even more advanced characteristics, as is the case with Voice over Internet Protocol (VoIP) tests. When it gets a response, the sender then calculates the response-time metrics appropriate for the test type, and then repeats multiple times.

Some SNMP polling products can collect data automatically, store it in a database, display the results in a GUI, and provide analytical function beyond data collection, such as calculating baselines, displaying trends, and triggering threshold alerts based on collected IP SLA data. There’s also the possibility of simply getting the information from the CLI, but extracting the IP SLA response-time metrics and copying them to a spreadsheet can be difficult and tedious. However, for the extremely budget-conscious, it can be done.

Deploying Quality of Service with Cisco CBQoS

QoS is a blanket term for network policies and practices that help to manage different types of data traffic that share network links. Effectively, QoS determines how different types of traffic, with different priorities, are handled whenever tradeoffs that are likely to impede performance must be made.

Now, within any enterprise, the end-user experience with certain applications will always be more critical than it is with others. Strategies to avoid (or at least manage) congestion could include dropping traffic, adjusting application responses, and building packet queues. CBQoS is one way to do this – and comes with the CBQoS Management Information Base (MIB) to collect statistics about the traffic traversing the router and reports how the QoS configuration is being applied.

Here, an SNMP polling product with application-aware capabilities can get information on input and output QoS class map utilization, drop percentage, and packet counts. It can also get information on pre-versus-post QoS traffic volume, rate, and packet count. It can also point out traffic marked in conformance, in excess, and in violation of defined policies.

Without CBQoS, network managers don’t have a whole lot of evidence to verify that their QoS settings are actually improving network performance – in fact, they may even be inadvertently harming performance. CBQoS prevents network managers from flying blind with QoS deployments. And, like IP SLA, it’s built into Cisco IOS.

Gaining a New Level of Visibility with Cisco NBAR

From within the network device operating system, Cisco NBAR can inspect packets traversing the device and identify the corresponding application – for example, TCP traffic running on port 80 could be labeled as Google, SAP, SharePoint, SalesForce, etc. NBAR can also provide utilization, volume, and rate metrics on a per-application basis relative to the network circuit carrying the traffic.

It’s similar to NetFlow, but NetFlow identifies protocol traffic mixes – not application-layer visibility. NBAR identifies by application – which is important in setting proper QoS policies. And because NBAR is part of Cisco’s IOS, and the data can be collected with an application-aware SNMP poller (which many of you already have), it can be a more cost-effective solution than application discovery hardware.

First off, congrats to Nick Carr – we’ve talked with him (and disagreed with him!) often on the blog and we’re thrilled that he managed to go toe-to-toe with Stephen Colbert on last night’s show.

And, thanks to the Colbert Report’s online presence, here’s an embedded player with that interview.

Although the book plugged is “The Big Switch,” the majority of the interview talks more about the implications of dwindling attention spans due to the Internet’s “hyper” hyperlinked nature – a topic not covered in “The Big Switch,” but instead in the cover article Carr wrote for the Atlantic Monthly, “Is Google Making Us Stoopid?”

The idea, as we’ve mentioned before, is that Carr believes the end result of the attention getting behavior of the Internet is that it will “scatter our attention and diffuse our concentration.”

“When the Net absorbs a medium, that medium is recreated in the Net's image. It injects the medium's content with hyperlinks, blinking ads, and other digital gewgaws, and it surrounds the content with the content of other media it has absorbed. A new e-mail message, for instance, may announce its arrival as we're glancing over the latest headlines at a newspaper's site. The result is to scatter our attention and diffuse our concentration.”

During the interview, Colbert made a play of ignoring Carr to check his iPhone. Now, that that does happen in real life, but I’d say that’s more an indication of individual rudeness then of culture spinning on a dime over the concept of hypertext.

The same criticisms that Carr makes of the Internet could be made of the newspaper – you’re trying to read one thing but it’s broken up, put next to all these other interesting articles, and ads designed to catch your attention… with all these… analog gewgaws, how is one supposed to be informed?

We’ve mentioned before that the limits on network performance limit the ability to communicate complex thought back when the Atlantic Monthly article first came out. But, we missed an opportunity to get less academic, more practical, and closer to the issues in a corporate network environment.

While we disagree with Carr’s diagnosis that the Internet causes short attention spans, (I’m a pro-blogger at a tech company, raised on Nintendo and MTV – I’m the poster-child for the 21^st century digital boy, and still I managed to summon the concentration to read the book Nick Carr wrote…) we do agree that human attention spans are short.

When I worked at a supermarket retailer, back in the early 2000s, as I’ve mentioned (and complained about) we were using a java-based networking app that took one to two seconds to input each number and move to the next field, and processing the entire report took minutes. The network performance was absolutely horrible, and as I pointed out before, we would have mentioned it in the hopes of having the performance improved somehow, except that we all realized that our jobs were essentially superfluous anyway and that we could all be replaced by a very small shell script that could parse the orders as they came in instead of printing them out and having us enter in all of them by hand.

Of course the lot of us at the data entry farm had CNN.com, Slashdot, and All Your Base Are Belong To Us and Hamsterdance open while we waited for the pages to load. (It was a simpler time back then.)

Of course, if we didn’t have outside Internet access, we could very well have distracted ourselves offline with desktop toys or conversation. We did that often anyway – as I said, it just took forever for those fields to come up.

I’ve also heard, second and third-hand, stories of other companies who are shocked to find that employees are going on to do other tasks while they wait for reports to generate, fields to come up, and pages to load – so if you’re honestly worried about dwindling attention spans, it might be better to not curse Google or the Internet, but to go in and actually improve things where you can.

Michael Cote of Redmonk just recently spoke with us and some other vendors here in Austin, and talked about it in a podcast he put up on his (very well read) blog.

Mostly, he focused his attention to our partnerships with Cisco, Microsoft, F5, and EMC, although there was a head-scratcher when Cote said that we were “getting into some configuration stuff” – which, I guess, if you mean we provide tools to diagnose many problems including router misconfiguration, sorta applies…

The other was a minor slip, which he quickly corrected, when he referred to “Quality of Assurance,” instead of what he meant to say, “Quality of Service.”

However, just to set the record straight – NetQoS provides some of the highest quality assurance possible, with daily affirmations, a self-esteem lab, positive-thinking modules for the Cisco routers, and assertiveness training for passive monitoring.

And if you work here five years, they give you a puppy.

From Cote’s podcast:

I have to admit I’m not an extreme expert in network performance tools and things like that. But what was interesting was to see the approach to spreading around the technology that they [NetQoS] have - just to various, sort of, partners, whether it's Cisco or Microsoft or F5 or EMC or even different geographies.

The position that NetQoS is in, is that, as with a lot of IT management stuff, different primary vendors, I guess you could say, people like Cisco, or people who are selling actual devices or applications being monitored often put out a lot of different information about that advice, and just kind of have it sitting there. And even, when you get into things like Cisco, obviously, there's a lot more advanced things like NetFlow and stuff like that that you can use to dig into an troubleshoot these problems.

So it creates this interesting sort of third party market, where other people like NetQoS can come in and do something with that data and plumb into those systems. And usually, you have to have to have a pretty good relationship with the primary vendor you're integrating with, which NetQoS seems to have. And so, what they’ve managed to do with their various platforms is sort of layer in to - like, peering into the network, and just helping people out with, as their name implies, ensuring quality of service.

And, what I found interesting, I was starting to say, about them, was that they're clearly in the stage where they're comfortable enough with their technology that what they're trying to do now is trying to spread it to different partners and geographies and things like that. So, I would guess, what you would see from them is - establishing more relationships like the one they have with Microsoft and EMC and Cisco, and so forth and so on.

Ken Church, Albert Greenberg and James Hamilton of Microsoft recently put out a paper on “Delivering Embarrassingly Distributed Cloud Services.”[PDF] Like most papers of this type, it’s a dry read, but informative. It looks at the tradeoff between mega-data center size and micro-data center diversity from the both the viewpoints of total cost of ownership and of performance.

The most important line in the entire report, of course, is “The trade-offs vary by application.” However, they make the argument that applications with little need for server-to-server communications will show benefits in cost, scale, reliability and performance through geo-diversification – in other words, lots of little datacenters as opposed to one big datacenter.

This seems to fly in the face of the trend in data consolidation, but there is a point to it: For any data center, there needs to be redundancy, but in a centralized data center, there needs to be more redundancy than having multiple small data centers. As Church, Greenberg, and Hamilton put it, “the more geo-diversity, the better. N+1 redundancy becomes more attractive for large N.”

The part that really interested me, though, was the networking section. (Section 3, in case you want to skip right to it.) Church, Greenberg, and Hamilton point out that in a large, centralized datacenter, you can have end-to-end control and assure a particular level of performance through supported service level agreements. On the other hand, they argue:

“[with distributed data centers] the cloud service provider has ceded control of quality to its Internet access providers, and so cannot support (or even fully monitor) SLAs on flows that cross out multiple provider networks, as the bulk of the traffic will do. However, by artfully exploiting the diversity in choice of network providers and using performance sensitive global load balancing techniques, performance may not appreciably suffer. Moreover, by exploiting geo-diversity in design, there may be attendant gains in reducing latency…”

“Many large analysis applications are best run centrally in mega data centers… Interactive applications are best run near users… [they] can be delivered with better QoS (e.g., smaller TCP round trip times…) via micro data centers.”

The argument’s sound, especially when you consider that interactive applications are probably the most latency sensitive because they need to make multiple trips to and from the client and server with every interaction.

But reducing the propagation delay (or distance delay) is merely one part of the performance equation. By ceding control over router performance and transmission, you have no way of diagnosing network round trip time problems if they occur, and wouldn’t be able to fix them – short of the messy step of changing service providers – even if you did. If something goes wrong, it could negate the speed increases by diversifying servers, so moving to this model more of a gamble than a guarantee of improvement. Granted, it’s a gamble that might make sense for some apps and some organizations – some apps, apparently, can get away with less than 100% uptime.

It’s easy to hit Microsoft for its ad campaigns; between the misleading nature of the “Mojave Experiment,” followed by the theatre of the absurd campaign starring Bill Gates and Jerry Seinfeld but which actually failed to explain absolutely anything about anything Microsoft does, and finally it’s “I’m a PC” campaign which can be seen as a reaction to the Apple campaign – a reaction years too late, and which can be easily negated by Apple asking John Hodgman to appear in jeans, a scuba suit, and an evening gown.  He’ll do it, too.

When ads from counter-culture surrealist animator Bill Plympton make more sense than your multimillion dollar ad campaign, which amounted to a celebrity endorsement of nothing, you’re doing something wrong.

But Microsoft isn’t the only offender. 

Cisco is launching “The Human Network Effect,” which demonstrates how Web 2.0 tools shape the way consumers live today.  Of course, I’m a consumer and I already know how Web 2.0 already affects me.

This is a continuation of the “Welcome to the Human Network” campaign where, among other things, humanitarian aid workers use a Cisco teleconferencing setup to communicate with parents back home from a village which, presumably, needs humanitarian aid.  That’s some good prioritizing there.  (To be fair, the advertisement also showed that Cisco has managed to somehow solve the speed-of-light latency problem from a satellite truck. That’s impressive.)

It makes me long for the advertising genius of “Philip Fill,” the marketing guru alter-ego of Kermit the Frog, in Muppets Take Manhattan, who comes up with the idea: “Ocean Breeze Soap Will Get You Clean.”

One of the weird things about IT is that people often talk about the “relationship” between “IT” and “The Business.” The word “relationship,” however accurately describing the circumstances is a bit odd. Rarely do people talk about a “relationship” between “marketing” and “the business.” You don’t describe a relationship between “accounts receivable” and “the business.” That’s because we think of marketing and accounts receivable as parts of the business – and are as inseparable to the business as a human body is to its liver. And it’s rare to say that I have a “relationship” with my liver. (Well, sometimes I have an abusive relationship with my liver.)

But it does seem that IT and “business” are separate somehow. Whether it’s because “business doesn’t get IT culture,” or “IT culture doesn’t get business” doesn’t matter – the two are thought of as separate entities in a relationship with one another. Specifically, they are in a multualistic symbiotic relationship – IT cannot exist without the core business writing the checks, and the core business cannot compete without well performing IT.

Now, we’ve often pointed out that IT needs to be more in tune with business needs, and that there should be more coordination between IT and business. That hasn’t changed. IT functions best when the individuals in IT departments both are aware and can make it known to the C-level executives that they are responsible, in the business, for increasing production and reducing expenses. But the reality is that there is some sort of disconnect, and the recent financial crisis shows part of it.

Because a symbotic relationship means that the two organisms can’t function without each other. So, when business is good, the mood in the IT department is often concerned more over the small details within IT that they have control over - incremental improvements and measuring the relative value of one generation of routers to the next. When business is bad, IT people start to become more concerned with the business.

It’s counterintuitive; but IT teams really need to focus on things the other way around. If business is bad (and I don’t just mean a non-profitable quarter, but when something in the core of the business has fundamentally failed – like we’ve seen with Lehman Brothers and AIG), nothing IT can do at that point can really save it.

It is at the other times – when business is doing well - that IT can have the most impact on the business, by reducing costs and increasing productivity. For example, we laugh about the “irrational exuberance” of the late 1990s dot-com boom, but infrastructure developed during that period helps companies to this day. If I’m recalling correctly, FedEx (and I’m not choosing them for any reason other than it’s the first company I thought of off the top of my head for which this metaphor applies) brought in package tracking infrastructure and electronic sign-in during that period – projects that never would have been approved during the next few years as the boom turned to bust – but which undoubtedly helped FedEx to cut costs during that bust.

If IT really is in a symbiotic relationship with the businesses in which they reside, then at the very least, IT should be doing everything it can to ensure the survival of its symbiotic “partner” in business.

In some cases, IT “survives” the death of the symbotic partner but they do not survive unscathed. The data centers of Lehman Brothers were picked up by Barclays. That may be reason to hope for some of Lehman Brothers IT staff, but the needs of a bank are very different than that of a security trading company – Banks care more about batch-processing – being able to store ultra-large queues and the power to process them, while securities trading companies on ultra-fast network speeds to make trades using the FIX protocol.

Bloggin’ from Brisbane – Cisco Networkers 2008
From Ben Erwin

It’s not all vegemite and tim tam slams.  Network managers from all over Australia decended on Brisbane to discuss their network management challenges and business goals with NetQoS.

NetQoS Booth at Networkers Brisbane 2008
--Source: Network World Cisco Subnet Blog -- Credit: Brad Reese/Network World

The NetQoS booth was busy during Cisco Networkers Brisbane 2008, with network managers from financial institutions, universities, and service providers looking for a monitoring solution that closely aligns with their Cisco infrastructure.  Several service providers stopped by to say their end customers are clamoring for performance-based network management capabilities, especially around existing deployments of VoIP.  Without a doubt, VoIP was the most heavily discussed technology at the show.

Everyone else has two things on the mind:  QoS and WAN optimization

Attendees asked for an enterprise dashboard to manage their QoS deployment, instead of relying on SNMP-only statistics, command line interfaces, or free tools.  They wanted to understand how NetFlow can be leverage for QoS, in addition to understanding the QoS impact to end-to-end response times of mission critical applications.  Everyone claimed that configuring QoS is a challenge and more visibility is needed to validate QoS configurations and their impact on the network.  For WAN optimization, attendees looked for reporting solutions that better integrate with Cisco WAAS from device performance of the WAE appliances to their impact on optimized applications.  Similar to QoS, attendees clamored for WAAS dashboard that can report on all aspect of WAN optimization in their environment.

We spent much of our time at the NetQoS booth discussing how you can manage your network for application delivery… and setting up attendees up with a neat flashing guitar pin. 

That’s all for now – time to readjust to the home time zone.

Video Thrilled the Networking Star
From Brian Boyko, Editor, Network Performance Daily

I know you guys love the Whiteboard Series, and we’ve always wished we can do more of them – well, now we can.

By day, I’m a blogger – but by night, I’m a video geek, making indie movies, that sort of thing.  So in addition to being the guy people go to when they ask me what blogging platform to use, I’m also the guy who gets a lot of video questions.  And, as it turns out, we’ve been using my personal video camera and equipment in order to do much of our Whiteboard Series videos as well as some internal training videos. 

And so we’ve been doing more with video as time went on, it was about time for NetQoS to get its own high definition video camera.  We went with the Canon HG10, for those film geeks who care enough.  Yeah, the HV20’s probably got better picture quality, but the tapeless workflow is going to be a timesaver, not to mention that compared to the HF10… crap, I’m filmgeeking here. 

Point is, what we are hoping that readers of the blog could do is – now that we have the fancy new camera – if you could send us your ideas for Whiteboard Series videos that we can then film, put up on the blog, and justify the money we spent on the new film equipment. Thanks!

While on Wall Street, banks were collapsing, IT pros were in New York as well for Interop.

We were a bit concerned, what with the economic downturn and all, as to what would happen with spending in IT in the upcoming year. So NetQoS ran a survey polling 112 respondents who attended Interop New York about how much they would spend on network management disciplines and other IT initiatives in 2009.

Here’s what we found.

A plurality of respondents, 46 percent, said that their spending on network management disciplines would stay the same. Only 15 percent of respondents said that they would spend less on network management disciplines and 39 percent actually said that they would spend more on network management disciplines.

Considering the economic woes on everyone’s mind – that’s pretty huge. And it implies that network management is seen as a necessity rather than a luxury. For example, a plurality of 28 percent of survey respondents indicated that the least likely to see an increase in spending in 2009 was change management. This makes sense: No money, means no new projects, which means no change, and no need for change management. Plus, change management has been a heavy investment area over the past few years so more competency has been built in this discipline at the expense of others.

Overall, 34 percent of survey respondents actually plan to increase overall IT spending in 2009, with 54 percent keeping it the same level. Only 12 percent plan to cut IT spending in 2009.

Does this mean that the economy is better than perhaps we had thought? Unlikely. Instead, what I think this means is that either A) IT is seen as such a vital part of the company that companies aren’t likely to cut corners, B) the corners have already been cut so far that there isn’t much left to cut without hitting something vital, or C) IT is finally starting to make the case that spending there can reduce costs elsewhere in the company.

Look at the big trends in IT: Server virtualization, datacenter consolidation, WAN application development, teleconferencing – all of these are designed to reduce cost. To some extent, IT has always been about leveraging technology to do more with less money, but there’s definitely more of a pronounced emphasis on the “less money” part of that equation than the “do more” part.

If you’re interested, we have a press release about the survey on the NetQoS main Web site.

Management consulting firm Janco Associates estimates that the flurry of news around Lehman Brothers, Merrill Lynch and Bank of America -- and separately HP with its layoffs planned around the EDS integration -- could cause the loss of some 13,000 IT-related jobs.

I was digesting that information when I came across this item about how the US is losing its competitive lead in the IT industry:

Workforce: US technology firms, like those elsewhere in the world, are experiencing shortages of skilled talent and will be adversely affected by slower increases in the number of science and engineering graduates from domestic institutions. Easing or at least avoiding further tightening of immigration restrictions on skilled workers would help US competitiveness.

So I have to ask this question: Can massive layoffs actually help US IT competitiveness by solving some of the talent shortage? Surely some percentage of these workers has the highly prized skills that are in short supply.

And in case you were wondering about the value of non-personnel IT assets to the financial sector, check out this assertion on the value of data centers:

What assets retain value in the midst of a financial panic? Data centers. When assets of bankrupt Lehman Brothers were sold to Barclays Tuesday for $1.75 billion, Lehman's data centers and headquarters accounted for $1.5 billion of the value in the deal. That echoes the JPMorgan-Bear Stearns fire sale, in which Bear's two data centers and HQ represented much of the sale price. Amidst financial turmoil, Wall Street's high-tech data centers become the crown jewels for buyers of distressed assets.

There’s a lot of discussion about the veracity of these claims, but one thing to keep in mind is that data center proximity to exchanges helps reduce latency in trading transactions, and that difference can mean millions.