Network World reports on a survey by security software vendor Symantec which talks about data center staffing. Specifically, half of the respondents said that their data centers were either extremely or somewhat understaffed.

And of course, there’s always the usual suspect to trot out – the economy – forcing IT workers to do more with less, with cutbacks and layoffs hitting IT hard. But there’s also another factor – that it’s not just that the IT staffing budgets are decreasing but also that the job of the network engineer is becoming complex, thus increasing the overall workload.

This is especially true in mid-sized enterprises where new technologies which can save money but which are extremely complex, like virtualization, WAN optimization, and cloud computing are being implemented at a faster rate than either smaller enterprises or larger ones.

Well, if you don’t have enough manpower in your data center, there are three solutions I can think of off the top of my head. The first is to hire more people. This may not be feasible given current budgets.

The second is to decrease workload. In short, taking the approach that instead of trying to do “more with less,” that it’s okay to do “less with less.” Five nines of uptime give way to three nines, and applications previously handled in-house are outsourced to a cloud services provider. There are some disadvantages to doing things this way, of course.

The third is to find a way to decrease the complexity of your network – perhaps by using management tools that provide a broad overview of the network and how the applications are performing. The only downside there is that if you don’t use these tools correctly, instead of making the job easier, an additional manager could just end up increasing the complexity of the network that much more.

All three of these solutions have the possibility of being disruptive – at least in the short term – and monitoring your network for those disruptions is the quickest way to get to the root cause of them.

Though CA and CA|NetQoS are vendors of aforementioned management and monitoring tools, I’m pretty comfortable suggesting that if you can hire more people, that it might be a good idea to do that first, if you’re making decisions about where to spend the budget money. There’s a couple of reasons for this.

First, no diagnostic, monitoring, or management tool can replace a network engineer with a good head on his or her shoulders. All a tool can show you is where the problem lies; the engineer has to come up with the solution.

Second, if you have engineers who know what they’re doing, they’ll be the ones to suggest the tools that they need, rather than buying tools first and then trying to train engineers on the proper use of the tools chosen on their behalf. A good engineer with a mediocre management tool is better than a mediocre engineer with the best stuff in the world, after all.

(Not that we don’t want you to buy the best stuff in the world - which, if you haven’t guessed our particular bias, is our stuff…)

Monitoring your network is crucial to maintaining your network; but the two are obviously not the same. You can have all the data, have it presented in an easy to understand format, run report after report, and it won’t matter if, at the end of the day, the person whose job it is to interpret the data misinterprets it.

If you look for the wrong things – for example, if you’re still primarily concerned with availability rather than latency – you can miss the most important details and come to the wrong conclusions about your network.

It reminds me of this guy, who has analyzed the Mars Rover photos on the JPL Journal Web site, and believes that there is a vast conspiracy at NASA to trample tiny humans (about 5cm in height) under the wheels of the Mars Rover.

“Next three images shows [sic] typical areas on Mars where three sizes of humans and primates live a symbiotic lifestyle. Strangely, the primates appear to be sentient…”

“Next is the Tiny humans [sic] attempt to disable a Mars Rover. The reason; it is the machine that has cause numerous deaths among the smallest Humans who cannot detect or hear the Rover coming.”

“***Warning next 5 images show scenes of death by crushing.*** Americans have Constitutional rights to know this information I have discovered from public posted JPL images…. The second image is gruesome. It shows the Rover has driven through a thickly populated tiny human’s area, killing a great number of them…. We are not at war with them. Someone will answer for these deaths.”

The photos, obviously, contains blurry images of rock formations and dirt, the silhouettes of which may look vaguely human-like in a Rorschach-ian way. Personally, I don’t even think they look vaguely human.

I bring this up because it reminds me of the idea that network data can often be an ink blot test of sorts; if someone’s looking only for availability, they’re simply not going to see the problems that are caused by poorly performing (but still available) applications.

That sound you’re hearing is the screaming of my soul being crushed.

All great journalists can maintain complete objectivity in the most trying of circumstances. I am merely a good journalist.

Well, I’m adequate.

But I think I should disclose my biases here, as this is an issue that affects me personally.

DSLReports.com reported that Time Warner has decided that Austin will be one of four “trial cities” to test out their new “Tiered Bandwidth” plans – which are essentially paying roughly the same amount of money you were before (give or take about $5) plus $1 for each gigabyte of data you transfer, over a set cap. The highest plan will be 100GB. Omar Gallaga over at the Statesman worries that the top plan it might not be enough for him and his family – I know it certainly won’t be enough for me.

The New York Times once estimated the wholesale cost of bandwidth to Time Warner at something like $0.10 per GB. At $1/GB, that’s a markup of 1000%

So I got upset. The Internet isn’t just how I make my day-job livelihood, it’s also what I use to transfer and upload the high definition videos that I put out on the Web for my moonlighting. It’s how I send raw footage to documentary collaborators. Since YouTube is the only affordable outlet for my work at this time, a bandwidth cap will make it cost prohibitive for me to continue as an independent filmmaker.

Best estimate yesterday was that I was I was using 400GB a month, and that my Internet service would increase 650%. Since then, I’ve actually looked at my home traffic data for the past 15 days. I’m happy to report that I’m only using 300GB a month, and at $1/GB, the bill would only go up about 500%.

(I could go for go for business service, which has no caps, but the equivalent of my current “Turbo” service, which provides 15Mbps down and 2Mbps up, would cost $280/mo. There are plans at $120, which is merely a 200% markup, but the $120 business class service provides only 10Mbps down, and 512kbps up.)

But what was even more upsetting was that I couldn’t figure out what was going on at first. I first found out about the story from DSL Reports. Calling up Time Warner customer service, they told me there was no plan for tiered bandwidth in Austin. Chatting with Time Warner tech support, they said it would start this month.

The actual truth is that monitoring end-user bandwidth will begin this month in Austin, but we won’t be charged for bandwidth until a couple months down the road.

But I didn’t know that until I literally had Alex Dudley, VP of Public Relations at Time Warner Cable on the line.

Time Warner is a monopoly in my apartment complex. I have checked – AT&T, Grande, Verizon – none of them offer service to my apartment.

I like my apartment. My apartment is very close to work, it has a nice swimming pool, and there’s a 100% lack of cockroaches. Essentially, I can’t justify staying there if I’m going to be paying $300/mo (or even $120/mo) for Internet service, compared to $60/mo elsewhere. In essence, I’m being kicked out of my home by Time Warner. (If one of your bills shot up 500%, you’d move too.)

I’m just lucky that my lease ends around the same time that TWC will be instituting these caps. It would have been cheaper in the long run to break my lease otherwise.

So journalistic impartiality? Well, I give it my best shot, that’s all you can ask of me in this situation.

One last thing before we get to the interview: We’ve covered bandwidth caps from Time Warner and others before in this publication. I really hope that if you’re interested in this subject, you read “Bandwidth Caps and the Cognitive Surplus.” In short: The Internet has finally given people something better to do than watch “whatever’s on” TV, and it’s creating a more participatory culture. Bandwidth caps are an attempt to stuff the genie back into the bottle. There’s more to it, of course, which is why I suggest you take a look at the full article.

A transcript of the interview with Alex Dudley – which at times seemed more like a debate – is below.

[The following transcript is mostly verbatim. A few words have been changed (ums, some interruptions, confused & corrected technical terms such as saying “gigabyte” when meaning “megabyte” and vice versa, etc.) so that the interview reads more clearly. Reporters following up on this story who need to verify the accuracy of the transcript are welcome to contact brian.boyko@netqos.com for details.]

---------------

Brian Boyko, Network Performance Daily: As you know, I'm a customer of Time Warner in Austin. I have the biggest bandwidth plan, I think the Turbo...

Alex Dudley, VP of Public Relations, Time Warner Cable: Turbo, yes. You like it?

NPD: Well that entirely depends on the answers to some of the questions that I'm about to ask you!

Dudley: [Laughter] It always does, it always does.

NPD: It does, you know? We've actually covered a lot of this stuff, like for example, when Beaumont, Texas came out with the 40 gigabyte plan. I actually did a couple of back-of-the-envelope calculations, and - I can't give you an exact number, but I'm pretty sure I would probably go over the largest limit - which was 40 gigabytes - just myself personally. So, I'm wondering - let's just talk about the largest plan, 40 GB - how much after that, per gigabyte does the Beaumont plan cost?

Dudley: A dollar.

NPD: A dollar per gigabyte. Here's one of the big things. I called up Time Warner in Austin, through the [customer service] number - that's the customer service for Austin - everybody I've talked to says there are no plans to do monitored bandwidth, or metered bandwidth, or data caps, or however you want to call it - there's no plan to charge per gigabyte. I called up - or rather, I chatted online with a representative - a technical support representative for Road Runner, and he said that the plan would begin for Austin residents in April. So this is conflicting-- the first question is obviously, what is the story with regards to Austin, Texas, and I assume the other plans in Greensboro, and San Antonio - is it Richmond? Let me check my notes.

Dudley: Rochester, New York. So, the plan is a little bit different in some of those places. In Austin, we are going to begin metering usage in April. But we are - let me back up. The best way to think about this is that we kind of do it in three steps. The first is that we get our technical house in order so that we can actually do this. And we start metering bandwidth. And we give customers a three month grace period, so that they can see what they're consuming, get used to the new plan, figure out the right plan for them without being charged. After which, we begin to mail the first bills. So - we are going to be technically ready to meter usage in Austin in April, and we're going to begin doing so, but we are not going to begin the grace period for a couple more months. Probably early Summer. So the first time that people in Austin see a bill is likely to be Fall. Septemberish.

NPD: I was a little concerned about that - when I heard that metering would begin in April, I thought that you would start charging in April. And - I guess not. And that was exactly what I was hoping that you'd say, actually, and I didn't expect you to say it. Because, I called up, I tried to find out my usage over the past three months exactly. I couldn't do that.

Dudley: Yeah - that's - you know, look, that's one of the keys to this. We understand that this is basically a wholesale change in the way that people talk about the way they use the Internet. We get that, that this is hard. So, what we want to do is make it as clear as possible. So - in addition to the three month grace period, we've also created what we call a gas gauge that will rest on your RoadRunner home page, so you can get instant up-to-the-minute calculation of how we are measuring your usage. And that will launch at the same time that we begin metering.

NPD: The only problem is, and as I've said, these are all back-of-the-envelope calculations - I don't have hard numbers, but personally, I'm not only a journalist, I'm also a filmmaker. I make short documentaries for the Web. And the idea is that-- Essentially, I'm sending 250 megabyte files over and over and over, so one video could take about 2 gigabytes themselves. And I'm downloading stuff like the Prelinger archives - for stuff to use - public domain footage to use in my video, I wouldn't be surprised, if you add in things like Netflix, (I'm a Netflix subscriber,) YouTube, Hulu, things like that - I wouldn't be surprised if I ended up being one of the highest usage customers in Austin. Probably - if I had to guess, maybe 400 Gigabytes - and that would cost around $400 a month - which is about... my rent.

[Ed. Note: I’ve since checked my Internet usage over the past 15 days based on what Vista’s Network connection reported during that time. A more accurate estimate would be 300 GB/mo.]

Dudley: Yeah, I think there's a number of different things that you can do. First and foremost, we are going to introduce a 100 GB plan. Kind of as a response to the folks who feel that the caps are low - we're going to offer that. We haven't finished pricing it yet, so I don't know what it's going to cost. That said, I mean, if you are using, consistently, 400 GB a month, then clearly, you're a target of what's going on here. And basically, what we're saying is - because of consumers that are using amounts like this, what we're seeing is a need for network expansion. Basically what we figure is that the top 25% of users use 100 times more network bandwidth than the bottom 25%.

NPD: Well that's just standard bell curves.

Dudley: I’m sorry?

NPD: Well, when you put any system on a graph like that, I mean, that actually seems a little low because of the 80/20 rule or the Pareto Principle or whatever it's called. When you put something on the bell curve, of course the top 25 are going to use the most bandwidth because they're the top 25. The lowest 25 are going to use the least amount of bandwidth. It seems like - funny numbers.

Dudley: Let’s use a metaphor then. You live in a small apartment. And, you're on the same electric grid as a very fancy neighborhood with big houses. And the guy in the big house leaves every light in the place blazing all night long - has a bunch of lights outside, spotlights on his beautiful house. And you have to subsidize his electric bill, just because you happen to share the network with him.

NPD: Right... but that doesn't...

Dudley: You conserve energy; you live in a small apartment, that's what we're saying. As we need to make improvements in the network to accommodate the increased demand, we can do one of two things. Either we can just charge everybody more, and we let the smaller user subsidize the top users, or we could create a plan that has a consumption element to it that asks people to pay for what they use.

NPD: Right, but that's kind of a false choice in that-- haven't you considered using -- it's kind of a false choice because what you can do is - first of all, when you talk about something like that, with congestion, the problem with the electric grid - using that as a metaphor - is that there's a finite amount of oil in the world. There's a finite amount of output. With a cable company, what you're really talking about is bandwidth, and bandwidth is simply a measure of how much you can have over time. So when you're charging for the data, basically, to use your metaphor, I think it's particularly unfair to charge more for the person who is using 40 gigabytes after letting a download go off overnight, compared to a guy whose using, maybe, under his cap, but he's doing it the most congested part of the day. And, what I'm thinking might be a solution without caps - and I was wondering if you ever considered this - is simply tracking the high-end users, and when they're downloading a lot and the line is congested, and only when the line is congested, then perhaps, throttling back their service using QoS priorities. Giving them...

Dudley: That’s exactly what Comcast did about a year ago, and it caused a complete outrage and the FCC hauled them before the committee and told them they had to stop doing it.

NPD: Actually, I covered that. That's actually the result that Comcast applied after the FCC asked them to choose a different system. You're talking about the Sandvine stuff that was sending forged RST packets and the issue there was that the RST packets looked like they had come from the sender itself, which was essentially kind of a classic "Man In The Middle" attack. A kind of a fraudulent thing.

Dudley: So here's what I'd say about that, then. What I'd say is that there are a number of ways you can address this problem. And the way that you've mentioned is certainly a possibility. We think this one also has some merit, and we're going to test it. And we'll see what happens.

NPD: Well, I can tell you right now that it probably won't work as far as congestion is concerned, because you're not attacking congestion. You're attacking data, while data is unlimited, while bandwidth is finite. And I'm already paying, as I said, I'm a Turbo customer - I'm already paying more for more bandwidth. If you want to charge me more for that bandwidth, I can go down to a lower tier in order to have it remain affordable. But, one of the things that I really can't do is, I really can't cut back on my consumption, because I do need this for both of my jobs - both this job as a journalist and as a filmmaker.

Dudley: Then it sounds like you should be a commercial customer, then, which is also possible. You know, for $140 bucks a month, you could be a commercial customer. And then there's no cap.

NPD: And can you give commercial customer service to residences?

Dudley: Yes.

NPD: That's good. That's a little bit more reasonable. Here's what I'm wondering. If a bill for a month goes consistently over, 140 GB a month - or rather the bill ends up being more than what it would cost for a commercial customer - let me start over. Basically, will customers be notified when it seems like they're going over and maybe they should upgrade to a bigger plan? Like a lot of the cell phone companies do, if they see you have like $30 worth of overage charges in a month, they'll tell you, "If you're going to keep doing this, you really should upgrade to the higher plan?"

Dudley: In the early parts of the trial, we are going to try to do that - try and alert folks when they're getting close, get them into a proper plan. But look, someone in your scenario, if you're really consuming 400GB a month, then that's not - we'll work with you on ways to curb your consumption, but clearly, if you're dependent upon it for work, you're not going to be happy with any of those discussions. So I think that for someone like yourself, a commercial account is probably the best option.

NPD: What about families that perhaps have multiple users. For me, I understand that I'm a high user, because I do videos. But the thing is, especially with services like Hulu and Netflix, and things like that - a lot of families could end up paying for one account, and everybody thinks that they're fine because they are only downloading 20 gigabytes which is half their cap, but if there's four people in the family, downloading 20 gigabytes - you start to have a problem. Will this negatively impact those larger families that need the Internet more?

Dudley: Well, I think that what you're failing to account for is that even in Beaumont, where the trial is active, 86% of our customers are unaffected.

NPD: Right, but 14% are!

Dudley: And by customers, I mean households, I don't mean individual people. So, the assumption is that the family may need more, but we haven't found that. Basically, we're targeting the highest end-users, to pay their share. So, I think that what we found is that for the majority of customers - for the overwhelming majority of customers, it's not even an issue.

NPD: Right, but 14% is a significant minority, and, I mean, my publication's called "Network Performance Daily," Chances are that my readers fall into that 14 percent. A great deal - and as time goes on, more users use the Internet for different services, and people get more Internet savvy, that 14% is just going to keep on growing. I mean, to me...

Dudley: But there's nothing to say that the plan couldn't grow with it, either.

NPD: Well, can I get your promise on that? That the plan will grow over time as Internet consumption increases overall?

Dudley: [long pause] That's a tricky spot for me. No. You can't. I don't make that decision. But what I'm saying is that there's nothing inherent in the capped levels that prevents us from doing that from an engineering perspective.

NPD: Alright. I just have to ask this stuff, and I'm sorry for putting you on the spot like that, but you know...

Dudley: Nah, this is okay. This is what I do. I'm happy to have a debate, so, it's not - no big deal.

NPD: Well, there is another aspect to this, and that is - Time Warner is a cable company that not only sells Internet service but also sells cable service - and I've mentioned services like Hulu, YouTube, AppleTV - those services - couldn't this be seen as anti-competitive? That all of a sudden it costs - you not only have to pay a dollar for a movie rental, but you also have to pay Time Warner a dollar and a half on top of that, for the extra bandwidth to make the movie rental - can't that be seen as anticompetitive?

Dudley: [pause] Only as much as it's anticompetitive for ExxonMobil to charge you the gas to drive down to Blockbuster to rent your video.

NPD: Right, but if you walk to Blockbuster, you're fine. I mean...

Dudley: That’s right, and if you stay under your cap, you're fine.

NPD: I - well you're still, if you stay under your cap, you're still charging per gigabyte - it's like... if you

Dudley: No we're not. We're charging for an allotment of gigabytes. We're charging for a monthly plan.

NPD: Right, but what I'm trying to say is...

Dudley: It has a limit. Much like you're - look, I don't know why this is such - why this is foreign to folks. You know, you're either paying for consumption... I mean, the concept of paying for what you consume is not a foreign one. I understand that it's different from the way we've charged for the Internet in the past, and we admit that. But the concept that you pay for what you use is how you buy just about everything.

NPD: Yeah, what I'm trying to say is that there's a way to pay for what you use and tackle consumption, without a data cap which has all these other side-effects.

Dudley: It's not a cap - you're thinking about it wrong. I mean we're calling it a cap, but it's not a cap. We don't stop you from consuming after you go over that cap, we just charge you differently. It's the same thing with a cell phone plan.

NPD: Yes, but it's effectively - it's a de facto cap. The argument is entirely that you are doing this entirely to get users to change their behavior. And what I'm--

Dudley: No we're not. We don't care - use as much as you want. All we're asking is that you pay for what you use. We're happy when you use - I mean, if you want to use 400 GB a month, and pay for it, we love you.

NPD: So this isn't a congestion solving problem?

Dudley: It is in -- it's a congestion solving problem in one of two ways. Either it will provide us with the revenue stream needed to beef up the network, or folks will change their consumption habits, which is entirely possible. But - it's not - that's not to say that-- it's completely content and protocol agnostic. We don't care what you download from where.

NPD: And that's great. I love that.

Dudley: I'm sorry?

NPD: I love protocol agnostic solutions - I think they're great - I just think that there are other protocol agnostic solutions that, perhaps, would be better than what you're doing.

Dudley: And if there are, and we will certainly look at all of them, then we will naturally be incorporated into this. And again, this is just a trial, so... I understand that as a heavy user, you're concerned - and your readership maybe heavy users too, and they're concerned about their personal skin in this game, and that's understandable. But I think that basically, it would be hard for anyone who consumes 400 GB a month to say that that doesn't cost us as your network provider a lot to service you. And it doesn't impact the levels of service on those with whom you share your bandwidth.

NPD: Right, but the point that I'm trying to say is that I'm perfectly fine for you charging me more, but charge me more based on the bandwidth I use, not on the data I download. You can use QoS policies - because I can't filter how congested the lines are on my end. You guys have to do that on your end. I'm just - what I'm trying to say is - yes, I'm a heavy user, but the heavy user isn't making problems for his neighbors if he's doing it when none of his neighbors are using the Internet.

Dudley: Hmm - and I don't disagree with that point, that that doesn't impact... I mean, you can't impact service on someone who is not online. And I don't disagree with that. But it still costs us money - the increased usage still costs us money to make the network able to accommodate that. And that's just a fact of business.

NPD: But you've already charged me more for my Turbo plan... You've already established, that "if you want this level of service, if you want this level of bandwidth, you're charging me for that Turbo plan." Now you're also saying, "In addition to that, you're going to also charge me for consumption?"

Dudley: That's exactly what we're saying.

NPD: Alright. No disagreement here. I know. I'm being rough. I don't like to be rough.

Dudley: No, it's okay. Look, I understand. This is a very passionate issue. It's very close to people like yourself that are heavy users. I get it. It's fine.

NPD: This corporate plan that you're talking about, how is it different from a personal plan? So, let's say that I do decide to go to the $140 a month plan.

Dudley: Just call us and ask for a business class account. And basically we come and hook up a corporate connection. It's a different customer service queue too. There are other advantages for you.

NPD: This is interesting - can we resell that? I know that residential services, you are not allowed to resell that, there was an article about someone who was trying to pull a fast one and resell 35 cable modems - with business class, am I restricted? Because I'm thinking - if it's going to be like $140 a month - that's still a pretty penny, and if I can get my neighbor to go with me on it, if he's got a Wi-Fi - if we can work out some sort of sharing deal, I have the landline so I'll maybe pay a little bit more - sort of like, "Internet Roomies."

Dudley: I don't know if that's possible or not. I'm not familiar with the service agreement on that. But they'll - just give us a call and the people that sell that service can answer that question. I mean, look, we're not interested in folks reselling our services, so I think that's what we're trying to prevent, but whether you guys could link as one business account, I don't know. I don't think it's outside the realm of possibility, but I could be wrong.

NPD: Well, thank you very much for talking - I know that I talked a lot - is there anything else you'd like to add?

Dudley: No, I think - for your readers it's different, but for the vast majority of our customers, they won't even notice a difference, and I think that - and I know that's probably not music to your ears, but at the same time, what we're doing is trying to ensure that we're maintaining a level of service that folks are happy to pay for, and if we don't make some sort of investment in this, or if we don't at least acknowledge that there's an issue here that needs to be addressed, then by the time we need to do it, it'll be too late. So this is an experiment. One of the things we're going to measure is customer reaction, so, you know, it's an important part of the process and we're happy to listen to our customers. But, you know, so far, the reaction has been what we've expected, fairly even across the board. So we'll see what happens and we'll make decisions based on what we see.

NPD: Alright, well again, thank you for your time.

By Patrick Ancipink

So far, so good at Cisco Networkers in Barcelona this week. Despite spirits being a little tempered by the worldwide financial crisis, attendance seems to be quite good and we are noticing a radical shift in what enterprises and service providers are trying to accomplish with their networks.

One topic that’s a bit been very popular with this audience is the concern about VoIP and video quality on converged networks.  So you could say network pros are more jittery here about latency-sensitive UPD applications than they are about the macroeconomic situation.

Heh…

Moving on...

The primary concern about video has shifted from quashing recreational YouTube viewers to ensuring the network can carry video. We spoke with several different companies (with headquarters from Sweden to Qatar) that have requirements to stream video across the WAN as part of their mission. I was involved in several conversations and overheard several more where the main topic was using application-aware network management tools and techniques like IP SLA and how to determine, validate and assure QoS.

I have to say I was pleasantly surprised about the awareness of the Cisco NAM support we announced the day before yesterday. It seems like before the news even hit the wire we had several attendees asking how they could use their NAMs alongside NetQoS. There are some monster implementations of NAM out there so everybody’s happy when you can leverage the existing investment for better application response time monitoring and performance troubleshooting.

Tapas and Rioja are nothing to complain about either.

Technically, it’s a powerpoint presentation, not a whiteboard sketch, but here, Robert Webb, Principal Network Consultant at NetQoS, brings a short sample of the type of training he does for the NetAnalyst program – in this case discussing TCP Slow Start.

The embedded version is low quality – you can head to the appropriate YouTube page for a high definition version of the video.

Part 7 in a series adapted from Joel Trammell’s Keynote Speech at NetQoS Symposium 2008

Think back three years ago. Back then, how many of you had an MPLS environment?

The carriers have been busy. And in that MPLS environment you lose visibility.

So how do you quantify how the performance has changed and how that carrier network is performing? After all, carriers claimed that by going to MPLS you would get better performance. But since they don't really monitor their own networks, how would they know? With MPLS, you don’t have a lot of data to validate whether it has improved performance. The ability to quantify how that carrier network is performing is critically important.

Additionally, how do you gain traffic visibility into the traffic flows, such as voice, that can now, instead of being in a traditional hub and spoke design, go from any location to any other location? The ability to understand the traffic flows without having to put devices out at each of your local sites to get visibility back into that traffic is crucial.

Finally, anomaly detection is important because often we see with MPLS routing changes that “just occur” that often affect performance. Suddenly the route changes on a major protocol, unknown to you, and suddenly performance is dramatically affected. So, you'll detect that in response time but you'll also see it in changes in traffic flows that will show up in anomaly detection.

EMA analyst Dennis Drogseth had a column in Network World yesterday talking about end-to-end application management. In it, he had this to say:

You might believe, and with some real justification, that the term “end to end” is only used by vendors who custom-fit the definition to the scope of their particular product.

Does “end-to-end” application management, for instance, include the mainframe? You bet it does if you’re a vendor that manages the mainframe environment! Does it include capturing the end user experience at the end station, desktop, or mobile device? Once again, the answer is a definitive “yes” if you’re a vendor that has strong QoE (Quality of Experience) roots. Or how about insights into the code and design of the application itself? If you’re one of the few vendors that does this, you’re proud of it and wouldn’t have it any other way!

And this concerned me because, if you do a google search for: [site:networkperformancedaily.com “end-to-end”], you get 122 results. The phrase, “end-to-end” appears in a little more than 1 in 5 posts we’ve made to this blog.

So, what do we mean by “end-to-end?”  We’re usually using the phrase in connection with network response times and the end-user experience at the end station; NetQoS is a “vendor that has strong QoE roots.”

Now, we do have some insight into the code and design of the application.  But that isn’t the focus of our tools; the focus is to tell you whether the problem is in the network, server, or application, and if it’s in the application, give you a good idea of where to start your investigation.  (For example, an application that is slow due to unnecessary round-trip transactions behaves differently from an application that is slow due to a memory leak on the server where it is being run.) 

Drogseth is right when he says that no one vendor is optimized to do it all.  In the future, there could be, but then you run into the quality vs. quantity problem.  Is it better to do it all adequately or to do a few things extremely well?

EMA defined five major technology spheres, and last June, they polled more than 400 respondents to find out which of them they believed “most critical to end-to-end application management in 2008.”  The answer was “Network Application Management,” focusing on application flows and end-to-end (as we define it) transaction capabilities. 

For more information on this, I recommend you read the original article up at Network World.  Additionally, Drogseth promises to follow-up in his next two columns. 

by Brian Boyko
Editor, Network Performance Daily

I remember my first computer.  I remember deleting files to save space on it – files like “config.sys” and “autoexec.bat.” 

I remember my second computer.  I remember forcing a RAM chip in too hard and breaking off a piece of the slot. 

I remember my third computer.  I remember not grounding it properly. 

Around that time I started working in a mom-and-pop computer repair shop.  I remember accidentally reformatting the wrong hard drive. 

Luckily for me, I was generally better at computer repair than the one incident otherwise this would be a very long editorial. 

Does this story have a point?  No, but I’ll shoehorn one in anyway: What I’ve found is that as geeks we learn through trial and error.  The best of us become best at it by making mistakes and by learning from them.  Sure, we may have accidentally fried a few CPUs, or, in some cases, siblings, but the point is that the surest way to learn something is to do it, learning from mistakes.

One of the problems with rote memorization and studying for certification is that it is through our mistakes that we can discover strange new things – the phrase that heralds the greatest advances in science and engineering are not “Eureka!” but “Hmm… that’s funny….”

And with a very long rant, a poster on the Overclockers Australia forums talked about how things in IT are going from “bad to worse.” Reading the whole thing is time consuming, and there’s some salty language, but it’s worth it.  Here’s some highlights:

Why is it then, that over the last three years I've seen fewer and fewer people who call themselves sysadmins understand these things? Why is it that I've been surrounded by "IT professionals" from junior sysadmins to CTOs who don't have a g*****n clue about one tenth of the above? Why is it that in three years I've met ONE person in professional IT who I would consider worthy of sitting down and having a conversation with?

Why is it that professional IT services today consist of service reps who tell you the things you are doing are untested, dangerous, unsupported, different, not usual, or a host of other words meaning they are scared s***less and unwilling to learn something new? Why is it that I spend my time building things people tell me for 6 months during build and test "will never work", only to have them go into production and work ten times faster for one tenth the cost of the old system? Why is it that IT professionals today choose brand labels over intelligence, and post-justify it by hiding behind "board confidence" when providing a solid, working, profitable system is the best thing to boost confidence from the board?...

Many moons ago, I used to have a mentor. A man who quite frankly I considered genius level. I don't throw around words like "genius" frequently. In my life I've met three people who would rightly qualify as geniuses. Only one I've had the pleasure to work with, and more importantly learn from. In the small amount of time I worked with the man my rate of learning tripled. He had the right amount of sage advice coupled with the sense to let you make your own mistakes from time to time.

This, I think is one of the biggest elephants in the room with regards to network engineering in general and network performance specifically.  There’s a reason we cherish “the old guys” in IT – these are the guys who were working on networks when it wasn’t a big deal – or at least, it wasn’t as big a deal as it is today – if they were broken.  In an environment where fault is expected and accounted for, you had a little bit more freedom to experiment.  Sometimes this leads to trouble.  Other times it leads to insight.  You repeat your insight and remember not to repeat your failures.

But the thing about enterprise networking is that the only people who own infrastructures large enough to support large networks are the companies who can absolutely not afford those large networks to fail or under-perform for any reason.  Going with the “safe” option instead of the one that offers the potential for learning is the only option that makes sense to the business.  But it is stunting IT.

I often wonder why computer security gets more coverage than computer networking.  After all, computer security doesn’t do anything, it just protects the stuff that does something.  Computer networking allows you to do amazing things.  Some of it, no doubt, is due to the “hacker mystique” – a fear of an active agent of destruction is more powerfully on the mind than planning how to eek out a 1% performance increase.

But there’s also the fact that security, rather than networking, is one of those fields where you are encouraged to try to get in there and break things.  This process of breaking things (and putting them back together) leads to more learning and more innovation… it’s a hell of a lot more fun and you learn more. 

Maybe what we need is for someone – I don’t know… academia?  The government?  Cisco?  - to develop a simulation of a working multinational Fortune 500 WAN, and just let students go hog wild on it, destroying and recreating it many times over, each time learning. 

Recently, we did an unscientific (and really, I cannot stress that word enough) but real-world test of performance using the Ono plugin for BitTorrent client Vuze/Azureus. Our results were inconclusive.

David Choffnes, the author of the Ono plugin, responded to the test in our comments section of that article:

Regarding your results, it is difficult to run controlled experiments because even when you download the same torrent, you're doing it at different times with necessarily different swarms. My research group's evaluation is not limited by this, and we showed that performance improves *on average*.

Also note that if Ono doesn't find any nearby peers, it can't help your performance. You can see if Ono found nearby peers (and is using them) in the "Ono" plugin view … Also, the plugin's effectiveness is limited by the fact that "only" 180,000 users have installed Ono. The more people use it, the more likely you'll find nearby peers.

One last point -- even when Ono doesn't dramatically improve performance, it encourages better "Internet citizen" behavior. Why transfer data from halfway across the world when you can get the same data and (potentially better) performance from peers nearby? Ono makes it easier to do the latter, which should eventually help everyone using the Internet.

Ono is part of Aqualab, a Northwestern University computer science project researching large-scale distributed computing. Choffnes, a doctoral candidate, will present his findings at SIGCOMM next month, and his paper on the subject can be found here – which is great if you like trigonometric functions in your technical literature.

There’s also a telling paragraph which may explain why we got the results we did for our tests (other than just the random variability of different BitTorrent swarms), instead of a massive throughput boost.

In our analysis, we compare statistics from peers located by Ono (referred to as Ono-recommended peers) to those from all peers selected at random by the BitTorrent protocol, which also includes those located by Ono.

In Network Performance Daily's analysis, we compared statistics from peers located by Ono combined with peers selected at random from the BitTorrent protocol, against only peers selected at random from the BitTorrent protocol.

To determine the cosine similarity value for a peer, Ono must be able to compare its ratio maps with those of other peers. The latter information can be obtained in a number of ways: through direct exchange between peers, from distributed storage and from trackers. Ono currently supports the first two options. With direct exchange, when two peers running the Ono plugin perform their connection handshake, the peers swap ratio maps directly… Though Ono enjoys a large user base, it is still a small fraction of the total BitTorrent population. Thus Ono also attempts to perform DNS lookups on behalf of other peers that it encounters, to determine their ratio maps. This enables Ono to perform biased peer selection over a much larger set of peers, including those not running the Azureus client. From both direct exchange of ratio maps and DNS lookups, our Ono clients locate over 180, 000 peers per day using our CDN-based approach.

When Ono determines that a peer has similar redirection behavior, it attempts to bias traffic toward that peer by ensuring there is always a connection to it, which minimizes the time that the peer is choked. Due to limitations of the Azureus plugin API, we are currently unable to bias other aspects of peer connections, e.g., the bandwidth allocated to each connection.

In addition to Ono, Aqualab also does other projects that are designed for improving Internet performance in a number of other areas. Choffnes’s advisor, Dr. Fabian Bustamante, has been working on "sustainable scalability in distributed systems,” called the 3R project. Many P2P and internet VoIP systems are built in a way that routing is controlled at the application layer, and that in order to identify better paths and faster throughput, the application probes the network environment repeatedly, as the application has no quick way to determine whether a particular peer or node is performing well except by trying to connect to it. The 3R project seeks to decrease probing by re-using the view of the network gathered by long-running, ubiquitous services.

While enterprise networking and Internet networking are two different beasts, performance advances in one usually lead to advances in the other, and with cloud computing promising to make enterprise networking a hybrid of LAN, WAN, and Internet connectivity, these advances remain important.

Part three of a series.
Part One: Interview with Texas State Rep. Joe Driver
Part Two: Interview with Matt Miller, Institute for Justice
Part Three: Interview with Capt. RenEarl Bowie, Texas Private Security Bureau

by Brian Boyko
Editor, Network Performance Daily

Recently, a posting on Slashdot linked to a story from PC Magazine called “Texas PC Repair Now Requires PI License.” Obviously, this story has gathered tons of attention, and if strictly true, would have a major impact on IT departments across the state, if not the nation.

Earlier, we posted a summary of the controversial law, HB 2833. We’ve also published interview with State Rep. Joe Driver, who authored HB 2833, and yesterday, we published an interview with Matt Miller of the Texas branch of the Institute for Justice, which is currently challenging the law in court.

Today we present an interview with Capt. RenEarl Bowie, of the Texas Private Security Bureau, regarding the Bureau’s interpretation of the law and policies towards PI licensing for computer and network techs:

Editor Brian Boyko, at NPD: First of all, could you tell me a little bit about who you are and what your position is?

Capt. RenEarl Bowie: Well, my name is RenEarl Bowie, I'm with the Texas Department of Public Safety, Private Security Bureau, and I'm the Captain of that bureau.

NPD: This department basically determines who needs, and issues, Private Investigator licenses, is that correct?

Bowie: Well the responsibility of the Bureau is to regulate the private security industry. And, encompassed in that industry are individuals who are considered private investigators.

NPD: The big controversy seems to be that a lot of people in the computer industry are doing investigations which they don't consider part of the private security industry, and it seems like there's been a lot of press recently - very recently - about whether or not the day-to-day operations of a typical PC repairman or network tech would constitute private investigation under the law and therefore require a license.

Bowie: Right, and I think, Brian, based on what you're saying, is that is what the intent and the spirit of the law is, under the Texas Occupations Code, 1702.104 [which] gives us a definition of what an investigation company is, and - you know, one thing you have to look at is when you read that particular statute, the interpretation is [that] the review of computer data for the purpose of investigating potential criminal or civil matters is a regulated activity under that code.

NPD: So could you give me a couple of examples as to what would be and what would not be covered under this law?

Bowie: A basic example would be an individual like a computer repairman who is providing computer repair or support services for a customer; normally that is not a regulated activity. But when an individual is performing work involving the review of computer data for the purpose of investigating criminal or civil matters, then they could fall under the 1701.104, which is considered an investigation company.

NPD: So, maybe I could give you a couple scenarios and you could help - maybe you could explain whether or not it would be covered. For example, let's say there was a network engineer who is trying to find the root cause of a slowdown on the network, and in the course of investigating that, they discover that the root cause is some sort of criminal activity, such as a virus infection, or someone engaging in massive intellectual property violation, in other words "piracy," something like that. Would they then require a private investigation license? Would they have to stop their investigation at that point?

Bowie: Based on the scenario you gave it sounds like they're performing a repair or support service, and they're not - the intent was not to go in and do an investigation, they are just collecting information that they found, and that doesn't, based on that scenario, doesn't rise to that level of an investigation.

NPD: What about a PC repairman who is being asked to check for viruses on a person's computer?

Bowie: That does not rise to that level either.

NPD: What if a parent brought in a computer that they owned, but which is primarily used by a son or daughter, and they wanted to find out, say, the browsing history?

Bowie: That's just considered normal computer repair or support service.

NPD: What wouldn't be considered normal computer repair - can you give me a very specific example where that line is crossed?

Bowie: No, it's - when you read into 1702.104, there is some interpretation there that you have to consider. I can't give you a specific example, I could probably use some type of scenario in the sense of, for example, if an individual is contracted to come in and say, for example, investigate your computer at your company - you have employees there, and you believe identity theft has occurred, that there's been some issues and you want this individual to come in, inspect the computers, you want them to come in, perform an investigation relating to the identity, the habits, the efficiency, movement, affiliations or locations or transactions and acts, or the character of a person, or the location and disposition of lost or stolen property, or some type of damage to the system, then I think you're moving more towards the spirit of the law, and falling into an investigations company.

NPD: Okay, so once you get to that point - this is something that's considered now to be routine is, if a person is suspected of - well, you could say a number of different things. Not just illegal activity but also perhaps, unauthorized use of the network - recreational network use - would that speak to the character of a person if they're browsing YouTube at work, and an investigation is made to determine if someone is browsing YouTube at work?

Bowie: I think what you have to do is take those on a case-by-case basis, and do a thorough investigation into the matter to determine whether a violation of the code has occurred. You just have to keep in mind that every scenario and case is different, and you have to take it on a case-by-case basis, and use the utmost discretion.

NPD: What happens if for, whatever reason, someone is ignorant of the law and they violate the law accidentally - that they perform an investigation, and in their particular case, even though they didn't intend to violate the law, they did? What happens then?

Bowie: Well, then again, it goes back to on a case-by-case basis, it involves good investigative work on behalf of the investigator looking into the matter, and then you have to evaluate what occurred, and what the individual knew, and what happened - and present the case to the court or to the prosecutor, if it even rises to that level.

NPD: Why do you think that this has been so controversial?

Bowie: As for as why it's been so controversial? I believe that there are entities or individuals that just want clarification and to get some understanding in regards to the statute, and it just recently became known to the media in regards to the individuals who raised the question, and of course the law was passed last year, but it has just been brought to media attention here just recently.

NPD: Is there any way you can think of to clarify the law and the interpretation of the law even further, so that instead of having to rely on the case-by-case basis scenario, to really hammer that down, "yes this case would be considered a private investigation, and this case would not."

Bowie: One thing individuals can do is they can definitely log on to our Web site, and when they get to the [Texas] Private Security Bureau Web site, there's a spot on the Web site called Private Security Bureau Opinion Summaries, and you click on that, and it has some definitions and even examples and some clarifications of 1702, and individuals can click on that and it'll definitely provide them with a lot of good information.

NPD: Well, thank you for speaking with us.