by Ben Erwin

NetFlow or NetFlow-esque technology (Jflow, Cflowd, NetStream, IPFIX, etc.) has been around the network management world for quite some time.  Thousands of IT shops worldwide leverage its capabilities to analyze traffic flowing across the network. 

Recently, some vendors have recently made somewhat misleading statements about NetFlow’s capabilities.  There are very good reasons why NetFlow is a de facto standard (and through IPFIX, soon to be an IETF standard).  Here are some quick reminders on why NetFlow is still the king:

• 100% visibility across all network links.   A common misconception about NetFlow is that it samples traffic.  Netflow exports every transaction it sees, and provides a full picture of what traffic is flowing across the network.  Now, it is true that sFlow samples traffic for flow export, but NetFlow exports every transaction it sees.


• Enabling at network aggregation points.   Instead of enabling NetFlow on every router, most NetFlow aficionados are able to enable NetFlow only on those aggregation routers that see the majority of network traffic.  This way, network managers can visualize their network traffic while not having to go overboard with router configuration. 


• Granularity versus TCO.  It’s true that NetFlow does not provide Application Layer (Layer 7) information.  But even so, remains the best bang for the buck for network visibility – yes, you could deploy probes all over the network to gain Layer 7 visibility – but there’s a significant opportunity cost in time and manpower for deployment, configuration, and ongoing monitoring, and the total cost of ownership for a probe solution for Layer 7 visibility simply isn’t worth it.  Many IT shops have dumped probes altogether and gone with NetFlow despite this limitation. 


• Free (if you use Cisco).  NetFlow is free on all Cisco routers.  All you have to do is enable it.  This makes it a very cost-effective solution compared to alternatives. 

These are all reasons why NetFlow will continue to be top dog for network visibility.  And while there are improvements to be made, certainly (there is no such thing as a “perfect” machine,) right now some of the best solutions for network visibility take advantage of the capabilities that NetFlow provides. 

WAN Optimization solutions – assuming that they work for the applications you need them to work for – are like magic. Consolidating data centers, from a relativistic standpoint, actually moves users further away, so to consolidate data centers, and lowering costs, WAN performance needs to be good enough for the remote users to do their jobs.

But the irony is that as data centers are becoming more consolidated, users are becoming less consolidated. More people are telecommuting than ever before. (Even if the number of full-time telecommuters has gone down, part-time telecommuters rise). It makes a certain amount of sense – an employee too sick to come into work (and infect others) but not too sick to actually work might file some work from home, or sales teams might file reports from the road.

This creates a problem for most WAN Optimization solutions because most solutions require appliances at both ends of the WAN link. Telecommuters are usually accessing the applications from the public Internet. Software-based WAN optimization controllers (“Soft WOCs”) can do some of the work, but telecommuting requires high-performing broadband as well as optimization solutions.

The way that Soft WOCs work, is essentially to recreate a lightweight version of the client that normally sits at the remote end of the optimized WAN link in the software on the mobile computer. The Soft WOC then optimizes the stream between the telecommuter’s computer and the data center.

The problem is that WAN optimization is less efficient when you have a single user than when you have multiple users on the same stream. First, having multiple users accessing the same data means you can take advantage of caching. Caching is only useful on a Soft WOC link if the same user accesses the same data twice.

Secondly, in a normal optimized WAN link, there is only one TCP stream to worry about – the optimized one, with individual streams recreated only at the two ends of the transaction. Each SoftWOC essentially creates its own stream. For that reason, telecommuting solutions simply aren’t going to give you the same dramatic increase in performance you’d get from more traditional WAN Optimization.

On the other hand, any improvement is still improvement. Just be sure to baseline your performance and see if the value is there before deploying Soft WOC solutions.

Wearing an eyepatch (I’m having a little eye trouble) has given me new appreciation for the importance of visibility.  There’s some good news coming out on that front; as company Ipoque has released an open-source deep packet inspection client.  It’s slower than commercial offerings – more of a tech demo than anything else, but the idea isn’t so much to provide DPI for the masses as it is to show the masses exactly what DPI is, instead of relying on rumor.  Ipoque doesn’t store or examine information being transmitted, for example, a common fear regarding DPI. 

Now, there are products that do packet capture and analysis – we even sell one for the enterprise IT environment.  But it would be chilling, at the very least, to use a product that inspects the content, rather than protocol, of information being transmitted for University or public ISP internet connections.  Still, knowing what protocols are running at any particular time is very useful for even public and university Internet connections.  If World of Warcraft connections and VoIP took a larger share of the network bandwidth than large Web/FTP downloads and YouTube, it’s an easy choice to focus network improvements on solutions that decrease latency rather than those that would increase latency but improve throughput. 

Of course, all the DPI and network monitoring in the world can’t help some networks. 

For example, Telkom, in South Africa, provides ADSL service.  Apparently, it’s not exactly the speediest service in the world, as Unlimited IT decided to transfer 4GB worth of data over 60 miles in one of two ways – via ADSL, or via carrier USB-stick connected to a carrier pigeon. 

The pigeon took two hours – including the time it took to load the data onto the computer from the USB flash memory stick. During that time, the ADSL transmission was about 4% complete.  It wasn’t even close.

Just doing some math here – 4% of 4GB is 163.84MB… that’s 81.92MB/hr, or 1.365MB/min, or 23.3kBps.  Yep, sounds about right.  That’s one of the major problems with ADSL… the “A” part.  That 23.3kBps is around 186kbps – which is actually not that bad compared to the 256kbps upload speed on most ADSL providers.  But the policy of providing download speeds vastly greater than upload speeds was created in an era where people overwhelmingly downloaded information from the Internet, and large uploads were rare, and usually done by large corporations.  Now, we have YouTube, Flickr, anyone in the world can contribute to open source projects, etc.  Might be time to consider changing those policies.

Or someone could make a ton of money providing an upload-only service to compliment the download-focused services of ADSL and cable providers.  Food for thought. 

One of the things holding back the rollout of new applications (like VoIP, Video, and Unified Communications) is the fear that the new applications will cause network performance problems; according to Network World’s Denise Dubie, citing a survey from Apparent Networks.

Nearly 61% said that they had delayed a VoIP implementation due to network performance concerns. Some 35% postponed a video rollout for the same reasons and 26% put a unified communications project on hold. The survey also showed that network managers can’t always validate their service-level agreements (SLA) with external service providers. More than one-quarter of respondents don’t have the capability to validate SLAs.

It would be instructive to know if decision makers are “concerned” that new apps will reduce their performance because they have baselined performance and know that the network cannot handle new application rollouts… or if they’re concerned because they have no idea whether the network can handle it or not.

It’s the difference between being stopped by practicality and being paralyzed by fear.

And if you’re being paralyzed by fear, it’s costing you money.

For example, Cisco decided to “eat it’s own dogfood” and estimated that they saved $277M from bringing in their own virtual office telecommuting technology – a new application (based on their “Cisco Virtual Office”) for the network that leads to cost savings. If Cisco didn’t know that their network was capable of supporting the CVO application, they would have been out $277M.

Of course, the reason you don’t roll out an application that might save you millions when you don’t know whether those applications will negatively affect network performance is that poor network performance can cost more than whatever you’d save by the rollout.

You can know, or you can be paralyzed by fear of the unknown. I know which I’d rather be.

When Savvis promises “proximity hosting,” they mean it – according to this New York Times Magazine article. In Weehawken, New Jersey, right outside of the Lincoln Tunnel, there’s a data center that houses the Philadelphia Stock Exchange’s computers. (The PSE is now part of Nasdaq.) Firms compete to have their computers located close – physically and in the networking sense – to the trading exchanges in that data center. Milliseconds of latency are unacceptable in this environment.

“It used to be that things were done in seconds, then milliseconds,” Varghese Thomas, Savvis’s vice president of financial markets, told me. Intervening steps — going through a consolidated ticker vendor like Thomson Reuters— added 150 to 500 milliseconds to the time it takes for information to be exchanged. “These firms said, ‘I can eliminate that latency much further by connecting to the exchanges directly,’ ” Thomas explained. Firms initially linked from their own centers, but that added precious fractions of milliseconds. So they moved into the data center itself. “If you’re in the facility, you’re eliminating that wire.” The specter of infinitesimal delay is why, when the Philadelphia Stock Exchange, the nation’s oldest, upgraded its trading platform in 2006, it decided to locate the bulk of its trading engines 80 miles — and three milliseconds — from Philadelphia, and into NJ2 [in Weehawken, NJ], where, as Thomas notes, the time to communicate between servers is down to a millionth of a second. (Latency concerns are not limited to Wall Street; it is estimated that a 100-millisecond delay reduces Amazon’s sales by 1 percent.)

Back in March 2008, electronic trading made up 60-70 percent of the daily volume of the NYSE. (I’m sorry I don’t have more recent numbers, but they might have been artificially affected by the credit crisis anyway.) And when you remove human beings from trades; the only thing that matters is the speed of a sale; whichever seller’s computer connects first to the buyer makes the sale, whichever buyer connects to the low-bidding seller first gets the bargain. Speed, while not everything, is not underestimated – and it’s one of the reasons you need to identify immediately any problems with network performance in financial applications. Every second a problem doesn’t get fixed – even problems that are imperceptable to the end user, like an added 3ms of delay - means more money is lost.

Now, if your company is over-leveraged and built on shaky investments, network performance won’t save you – we’ve seen a lot of companies with very good network infrastructures go downhill these past few months.

If you want to learn more about the topic of monitoring trading applications for performance, you might want to check out Alex Malone, Software Engineer Manager at NetQoS, who will be speaking at the Securities Industry and Financial Markets Association Technology Management Conference & Exhibit on June 23-25 in NYC. Alex is scheduled to speak June 24, at 2:35pm. You can also look us up at booth #1822.

He thinks it sucks.

No, correction. According to his latest post in Network World, he thinks it “Sucks!!!” With three exclamation points.

What is it with NMS that feels like were are [sic] riding in the back seat from Wisconsin to Florida with our stinky second cousin Bert. Anytime, I sit in a vendor meeting and they are trying to hock their NMS off on me I can picture signs for "See Rock City" or "Wall Drug" for the east coast to west coast survivors.

As a blogger for a company that makes network management software, (well, technically, network performance monitoring and management software), I figured I should respond.

But still, NMS promises are kinda like being chased by a [sic] angry Shih Tzu with your friends watching. With networks becoming more and more application layer driven, we need something with a little more power. For example;

- Flow based management is cool but what if I need more then [sic] just the conversation, I need packet capture/inspection THEN correlate those together with a verifiable SLA? Now what?

- 1GB 10GB 40GB 100GB? How do I monitor that? Not with a plain Jane NIC for sure. Heck at 1GB my NIC buffer size is only 64K which is just fair for 100M. Plus add in fragmentation and CPU interrupts and you can see your accuracy goes down fast. What's that? Your are using jumbo frames also...oh man...

Tough call, but I think you’re probably looking at something like a combo of NetQoS SuperAgent to handle the flow data, and NetQoS Gigastor to do the packet capture/inspection and monitor the large links through storing the data so that problems can be examined for a short time after they’ve happened. On a 100GB link, you’re talking hours rather than days, but still.

Now, this isn’t a pitch, (I don’t trust Shih Tzu dogs,) but seriously, we were wondering about the same problems, and didn’t have a solution until we integrated Gigastor into our suite.

But that’s not really the point, the point, Jimmy Ray. (Do you mind if I call you Jimmy Ray?) The point is partially that what you’re saying about network management software doesn’t jive with the numbers we’ve got on our end.

We use maintenance renewal rates as a way to benchmark customer satisfaction. If customers think our product “sucks,” they’ll typically get someone else that doesn’t “suck.” Last quarter, the renewal rate was north of 90%. This is pretty good considering the fact that the economy is so bad that dogs are beginning to worry about inflation.

More than half of these customers own multiple products as well – no single metric is adequate, as you pointed out, but through a combination of metrics, you can get the data you need.

What I’m trying to say, Jimmy Ray, is not that you’re wrong about saying that Network Management Software sucks. What I’m trying to say is that if we do suck, we’re sucking in such a way that it’s a blind spot for us – we thought we were doing an excellent job.

Ben Erwin, Technical Marketing Manager at NetQoS, comes back for another Whiteboard Series video to explain how NetQoS’s integration into the Cisco NAM means that you need fewer data collectors and can lower your costs by leveraging your existing infrastructure.

He also, for some completely non-sequitur reason, plays ping-pong.

A high definition version can be found at the NetQoSVideo YouTube Channel, here.

We talked a bit about the SuperAgent/NAM integration earlier.

As far as enterprise technology companies go, Savvis is probably one of the more interesting ones. They provide managed services for enterprise IT functions, building “IT infrastructure as a service.” (This type of outsourced IT model is more typically found in Europe, though as Savvis shows, there are advantages to using this method in the United States.)

One of the caveats that we often pointed out is the idea that relying on an outsourced IT department means that it’s difficult to make sure that your network is being managed correctly; the corporate equivalent of trusting that your auto mechanic knows what he’s doing. And to that end, we’ve recommended that companies that rely on outsourced IT have some way to independently monitor network performance.

Savvis decided to skip that step and offer network performance monitoring on their “Global Network Solutions” themselves; using NetQoS’s tools. This way, their customers get insight into traffic composition, network utilization, and application performance assurance on Savvis’ MPLS-enabled Application Transport Services.

Or in other words, independent monitoring of network performance on managed IT is now standard on Savvis. It’s kinda like when Saab made seatbelts standard in 1958.

Here’s the obligatory “the economy stinks” paragraph: You can imagine that outsourced IT functions would be appealing for many companies as they refocus on priorities. A good IT department can help you better sell widgets, but if the widget company goes under, the IT department is going to go down with it. IT enables business. IT is extremely important to the business. But IT is not the core focus of the business (unless, of course, you’re Savvis or another outsourced IT provider). Outsourced IT means that companies don’t have to worry (as much) as they would if they tried to operate an internal IT department while figuring out how to make and sell the core product.

Companies do still have to worry that they’re getting the service they paid for, though, and that’s why network monitoring will remain important no matter what solution – insourced or outsourced – companies choose to handle their IT infrastructure.

By Patrick Ancipink

So far, so good at Cisco Networkers in Barcelona this week. Despite spirits being a little tempered by the worldwide financial crisis, attendance seems to be quite good and we are noticing a radical shift in what enterprises and service providers are trying to accomplish with their networks.

One topic that’s a bit been very popular with this audience is the concern about VoIP and video quality on converged networks.  So you could say network pros are more jittery here about latency-sensitive UPD applications than they are about the macroeconomic situation.

Heh…

Moving on...

The primary concern about video has shifted from quashing recreational YouTube viewers to ensuring the network can carry video. We spoke with several different companies (with headquarters from Sweden to Qatar) that have requirements to stream video across the WAN as part of their mission. I was involved in several conversations and overheard several more where the main topic was using application-aware network management tools and techniques like IP SLA and how to determine, validate and assure QoS.

I have to say I was pleasantly surprised about the awareness of the Cisco NAM support we announced the day before yesterday. It seems like before the news even hit the wire we had several attendees asking how they could use their NAMs alongside NetQoS. There are some monster implementations of NAM out there so everybody’s happy when you can leverage the existing investment for better application response time monitoring and performance troubleshooting.

Tapas and Rioja are nothing to complain about either.

Nortel filed for bankruptcy protection today, according to the Globe and Mail. At the peak of the tech boom, Nortel was Canada’s largest company, accounting for nearly a third of the total value of the Toronto Stock Exchange. Now, it’s a fire sale, as the company will have to sell of key networking assets.

In addition to selling the Ethernet unit, divisions that could go on the auction block include the carrier networks unit, which sells gear to phone companies, and the enterprise division that sells telecom equipment to businesses. Potential buyers for these units are all foreign [to Canada]: Nokia Siemens Networks, Telefon AB LM Ericsson, Cisco Systems Inc. and China's Huawei Technologies Co. Ltd.

The biggest problem is that Nortel suffered a large drop in orders from phone company clients as the economy cratered.

The question then remains: Where does this leave companies that bought Nortel’s networking and unified communications equipment? What may end up happening is that monitoring and maintaining Nortel’s equipment may have to come down to vendor neutral products, as it is unlikely that Nortel will continue to support equipment from a division that they no longer own.

Tracking network response times and quality of service of unified communications becomes more important when dealing with a defunct vendor’s equipment – you need to be proactive and head problems off quickly, because there’s no external vendor to go to for help – or to assign blame.

There’s another point in here: relying on network equipment providers for management has generally been a troublesome approach because network equipment providers typically only manage their stuff and heterogeneous environments are the norm. Secondly, management is not the network equipment provider’s core competency.

The result can be a collection of sub-par management tools and no standardization. Large enterprises have known this for a while but smaller companies that have managed to stay with a single network equipment provider and make do with their management tools should now be taking a harder look at diversification at vendor neutral management strategies and products.

This article was created with the input of Patrick Ancipink and Ben Erwin.