At the UBS 37^th annual Global Media and Communications Conference, Ralph de la Vega at AT&T mentioned that AT&T’s been having cellular data network problems because of heavy users – touting out the usual “80/20 rule” bandwidth hog rule (in this case, that 3% of users use 40% of the data) the company’s representative, Ralph de la Vega, suggested  to its investors that AT&T would likely introduce a pricing scheme that would penalize heavy data users, according to the LA Times.

We’ve written before about how the “X% of users use Y% of data” is kind of a boondoggle, but that doesn’t mean that congestion problems aren’t real for AT&T.

Which makes me wonder: If your data network couldn’t handle data streaming services for downloading web browsing, video, and applications, why, in fact, did AT&T team with Apple to sell a phone whose selling points were web browsing, video, and applications?

This is compounded with the idea that a lot of the problems have occurred in New York City, where AT&T rolled out the 850 MHz spectrum. The 850MHz spectrum travels further into office buildings and apartment complexes than the previous spectrum – and as a result data usage in the area went up 30%. 

Again – to me this seems like a case of infrastructure not being able to support the sales claims.  So the end result is that AT&T is trying to do the most difficult thing of all – trying to solve the problem by changing end-user behavior.  To their credit, AT&T is, at this moment at least, intending to use incentives to get people to use less bandwidth rather than punishments for heavy users. 

Still I feel a little frustrated.  Business Internet has been around for nearly, what, 15 years now?  How long will it take before people learn you don’t roll out new applications or services without baselining your performance and making sure that you have the infrastructure capable of supporting it. 

I have to admit I’m surprised – and amused – and a little frightened – (and maybe just a tiny little bit turned on) by news that Microsoft’s Bing search engine reached a 9.9% market share in October 2009 compared to Google’s 65.4%. That’s a huge slice of the search pie for a product that had only been around five months.

Additionally, ads on Bing’s searches are 75% more likely to be clicked than Google’s. I think I have an explanation for both. When was the last time you saw an ad on TV for Google? In fact, when have you ever seen an ad on TV for Google the search engine; rather than some side-project like Chrome, Chrome OS, Android or something like that?

Bing’s success is probably coming from their ad campaign – and by definition, if you draw people in with an ad campaign, you’re going to get more people who are influenced by advertising. (There are also those who need to use IE for Web applications, where Bing is the default search – and simply never change it.)

Anyway, whatever the cause, Search is a viable competitive playing field again. To be fair, Google has repeatedly kept improving their search algorithms, but nothing inspires improvement like real-world competition.

It may also be interesting from another perspective, in that, while searches don’t take up a whole bunch of bandwidth themselves, all the little add-ons and perks that search engines tack-on – from Yahoo’s Flickr to Google’s YouTube to Microsoft’s Virtual Earth 3D, can clog up the Inter-Tubes pretty quickly.

As you’ve no doubt heard, we’ve been acquired by CA, which, a few years ago, acquired Wily Technologies, which also has products in the field of “application performance management.”  Earlier today, there was a meeting among us to discuss similarities and differences between CA|NetQoS’s approach and CA|Wily’s.

And one of the things I noticed is that, ultimately, both our stuff and their stuff has the same goal, and in many ways, the same design philosophy.  We both knew that passive monitoring tools that can scale with the network and infrastructure were the way to go, and we were both trying to get down to the source of application slowdowns for quick resolution.  So from a high-level messaging perspective it can be hard for the uninitiated to tell Wily and NetQoS apart. 

But there was a difference in our philosophies, and this really did become apparent in the way the programs were designed and used.  NetQoS based their software primarily around using the information available from what the network gives you, namely packet headers and NetFlow.  And while we can drill down to the packet level if need be, our primary function is to baseline performance and then inform you when things start to deviate from the baseline. 

Wily, so far as I was able to ascertain from the discussion (which, I’ll admit, went so far over my head that I’m pretty sure it’s eligible for a DARPA prize for unmanned orbital flight,) drills down into applications much further than NetQoS.  For example, Wily’s stuff can drill down not only to the application process which is causing problems, but can steer you to the individual line of code which is causing those problems.  (This is important when you have, say, a Java application, which is just running JAR files on otherwise identical Java virtual machines – the process could be running fine, but there might be a hiccup in Java interpretation.

What’s interesting to me is that the two philosophies tend to have come from the different backgrounds of the two companies.  NetQoS’s stuff is really designed to be used by the network engineer, giving information which allows you to drill down to find and diagnose the problem as quickly as possible.

Wily’s stuff seems more geared towards the (networked) application developer, the person who needs to know where the problem is in the code to fix it. 

I’m sure that to many people, getting two application performance applications may seem like having both Matlock and Columbo on the case.  And in a way, that is.  Columbo goes around solving pretty much all the crimes in Los Angeles to make sure they have the right suspect.  Matlock comes in when they know there’s a problem – namely, the wrong person’s been accused – and then gets down to the nitty gritty in order to fix the problem. 

I like ‘em both.  (I still think Columbo’s cooler, though.)

There are only 17 shopping days till Christmas, only two shopping days till Hannukah, and eight shopping days till Samuel L. Jackson’s birthday.  Sure, it’s not a religious observance, but Jackson’s tired of monkey-fighting people forgetting his Monday-to-Friday birthday.

So – what to get a geek?  Here are some ideas – and some idea of the impact on network performance.

1) Mini HD Camcorder

Cisco’s Flip cams dominate the industry to the point where any small-sensor high definition portable cheap camera have been termed “flipcams” – sort of like “hoovering” the rug in the U.K., or blowing your nose with a Kleenex. 

Just be aware that at if you put those flip videos streaming on the network, you’ll end up with 9MBPS worth of bandwidth being sucked up. 

2) An internet TV streaming device

Whether NetFlix, Vudu, Amazon, AppleTV, or a Blu-Ray player with YouTube built in, it seems to be time of the Internet enabled set-top box.  Personally, I’ve just got a computer hooked up to my TV, and that seems to do the trick. 

Again, however, streaming video over the network could result in slowdowns, especially with high-definition content.

3) Online Games

Whether it’s World of Warcraft or Team Fortress 2, online games are often a nice buy for gamers.  They don’t take up a lot of bandwidth, per se, but they do require low-latency connections, which may be a problem if you’re also running some sort of Voice app or video communication program at the same time. 

4) A Cellphone

Droid, Blackberry, or iPhone, the smartphones of today all have some sort of Internet connectivity through WiFi – sometimes automatically.  So, as people come into the office after Christmas break, all of a sudden there’s a rush of new devices on the WiFi enabled Internet. 

And before you ask, yes, it was a bit of a slow news day. 

Anandtech has an interesting article out about how it might be time to move forward on standardizing the x86 instruction set – both Intel and AMD have had proprietary instructions included in their CPUs. On the desktop, these proprietary instructions aren’t annoying – in the worst case scenario, a program would take advantage of a “shortcut” on one processor that wasn’t available on another – but what has changed for IT users is that increasingly, x86 chips are being emulated in virtualization, which increases hardware requirements and slows systems down – cutting the benefits which have traditionally driven virtualization.

“Much worse is that this unstandarized x86 extention mess has made it a lot harder for datacenters to make the step towards a really dynamic environment where you can load balance VMs and thus move applications from one server to another on the fly. It is impossible to move (vmotion, live migrate) a VM from Intel to AMD servers, from newer to (some) older ones, and you need to fiddle with CPU masks in some situations just to make it work (and read complex tech documents).”

One of the more interesting riddles about the IT industry and computer development in general is the idea of standardization.

Innovation often occurs due to non-standard protocols and unique innovations. Some open source projects are forked and become a jumbled mess of programmers working at cross-purposes; but others are forked and create new innovations that merge their way back into the main stream of program development, and become a new standard.

And even though sometimes it seems like every vendor has a new way of doing the same old thing, the competition between technologies gives motivation and ability for improvement.

On the other hand, it can also get in the way of doing stuff you really want to do! Windows has become a “de facto” standard for operating systems – and even among Linux, Ubuntu is leading as a clear desktop standard. The entire internet was built on TCP/IP – and it is indeed ONE internet, not a collection of “data services,” like AOL, CompuServe, and Prodigy; nor is it a large collection of BBSes.

Standardization leads to efficiency – but competition can lead to innovation. Multiple solutions spring up to a problem, and the best (or surviving) one becomes a standard which is then made more efficient, which opens up new solutions to new problems, and the cycle repeats.

Kinda like the Hegelian dialectical method, if you don’t think about it too hard.

Now, if all the computer technology in the world was developed by one company – standards first, innovation second – computer technology would stagnate. Yet, still, before “standards,” interoperability and integration provides a middle ground that allows for better technology to come onto the field without overly burdening the end-user with tons of configuration problems – which is why it’s such a high priority when choosing among enterprise networking vendors.

We’ve mentioned a lot about data caps, and why they’re not effective methods for controlling congestion, though they’re often sold as such to unwitting consumers. And we’ve done the analysis that shows that the effective speed of a capped plan can be slower than uncapped dialup – at least, when you average it all out.

But Benoit Felton of the Yankee Group went further. Sure, he does the math too, but points out that the users that are labeled as “bandwidth hogs” are hogs because they’re the top 5% of users – not whether those users actually cause performance problems.

The fact is that what most telcos call hogs are simply people who overall and on average download more than others… TCP/IP is by definition an egalitarian protocol. Implemented well, it should result in an equal distribution of available bandwidth in the operator's network between end-users; so the concept of a bandwidth hog is by definition an impossibility.

Now I'm pretty sure that many telcos will disagree with our assessment of this. So here's a challenge for them: in the next few days, I will specify on this blog a standard dataset that would enable me to do an in-depth data analysis into network usage by individual users. Any telco willing to actually understand what's happening there and to answer the question on the existence of hogs once and for all can extract that data and send it over to me, I will analyse it for free, on my spare time. All I ask is that they let me publish the results of said research (even though their names need not be mentioned if they don't wish it to be). Of course, if I find myself to be wrong and if indeed I manage to identify users that systematically degrade the experience for other users, I will say so publicly. If, as I suspect, there are no such users, I will also say so publicly. The data will back either of these assertions.

A great challenge, of course, and the right approach to it: Get the evidence. Make the analysis.

As a follow-up to yesterday’s post, too often both in ISPs and in enterprise networks, social network and video usage gets blamed for problems with network congestion without enough evidence. It’s true that bandwidth consumption can cause problems with network congestion, but if your congestion problems are caused by a misconfigured router or a slow application, blocking YouTube and chastising FaceBookers are not going to solve your network performance problems.

Ann Bednarz’s latest article for Network World wasn’t that surprising in the broad sense – yes, social networks are popular at the workplace, yes, they take up a bunch of bandwidth – but I’m not sure it makes the case that it tries to.

The key to the article is an analysis from managed security services vendor Network Box, which tracked 19 billion URLs and ranked the top five Web sites visited from business addresses by volume of traffic. It recorded that 5.8% of all Web traffic went to Facebook, with Google at 4.1%. That’s by visit – by bandwidth usage, YouTube consumed 7.8% of that bandwidth, with Facebook at 4.4% and Windows Update at 3.8%.

But what gets me is that while these numbers are interesting, they’re not particularly useful. As a metric of what Web sites are popular with people at businesses, it’s okay, but it doesn’t give you any idea about the actual impact of Facebook or YouTube on network performance – in short, it doesn’t tell you whether there’s a problem with Facebook or YouTube in your company, or even in companies overall. Plus, it only looks at Web traffic. Without knowing what percentage of overall enterprise traffic is Web traffic, there’s no way to deduce the overall impact on the network. Even then, some networks might have a very little amount, percentage-wise, of Facebook traffic, and it might cause a problem, or conversely, another network might have a large percentage of their traffic going to Facebook, but their network is able to handle the demand.

As such, it’s hard to use that data in the article to make the case that “Social networks take a bite out of corporate bandwidth.” Does that mean that social networking isn’t taking a lot of bandwidth, or isn’t a concern? No – just that it’s not the right evidence to support the conclusion.

That said, gathering evidence that lets you know whether Web traffic (or any other traffic, really,) is affecting your business critical applications is still a vital part of a nutritious network breakfast.

There are two articles in NetworkWorld today that caught my attention.

The first article talks about Google Apps, and it’s inroads into enterprise environments as a replacement for Microsoft-based environments. And while it is growing – with an IDC survey saying that Google Docs is “widely used” in 20 percent of companies, there are issues with user training, legacy e-mail migration, interoperability, tech support availability, and concerns about security for companies that have to worry about data retention regulations such as SarBox.

There’s plenty of detail in that article if you’re interested in the pros-and-cons of Google Apps rollout, which I’m not going to get into here.

The other article talks about how one of the top data center challenges is now energy cost – with energy prices predicted to rise (though, let’s face it, when in our lifetimes have they ever really fallen?) companies are shifting the cost of running IT equipment from an infrastructure budget to the IT budget. We’ve talked before about how this is a compelling case for virtualization (and server consolidation.)

What is interesting to me is that these two trends combined reinforce the need and the possibility of application consolidation. While it’s not new that similar apps are bundled together, (Outlook, after all, is an e-mail app and a scheduling app,) one of the big things about Google Apps is that it’s one rollout, one setup, and I would imagine, one server. E-mail, scheduling, word-processing with network storage, and video sharing – all one app.

One of the major problems that large companies have is that between legacy systems, new business needs popping up, combined with the idea that virtualization decreases rollout time, and you have a situation where you might have thousands of applications on your network – if not hundreds of thousands. With the complexity comes waste, effort duplication, and as anyone who has played with Lego or C code will tell you, the more complex the system, the more likely something’s going to go wrong.

When one company went into gain a bid on a Navy/Marine Corps Project, which was the biggest government IT contract that had come up in years, part of the scope of the project was to reduce the number of applications on that Navy/Marine Corps network to only 5000 applications.

But when they asked the Navy/Marine Corps how many applications were on that network at the time, they had no idea. The company, along with rival bidders, had to make a guess – and 25,000 sounded like a good number. After all, there was no possible way that there could be more than 25,000 applications on this network, and so, they made that assumption and bid on that basis.

In reality, there were more than 105,000. Every single Navy base, every single ship, every single Marine installation had developed their own suite of applications independently of all the others.

If you’re able to cut down the number of applications by an order of magnitude, it also reasons that you’ll be able to cut down the number of hosts needed for those applications, and cut down the number of hard-servers needed for those hosts.

And this is where SaaS and projects like Google Apps come in. Essentially, anything that runs through the Web becomes one app – the Web app. In this way, it is almost the ultimate in application consolidation. (Well, maybe the pentultimate - there are still browser compatibility issues, extra stuff to download into the browser, etc, that increase complexity on that front. It’s simpler than having multiple apps perhaps, but another source of complexity nonetheless.) More than that, though, even among Web apps, the move is towards multiple applications becoming singular applications – one of the reasons that Google and Yahoo have been so acquisition-happy is because of the idea that these multiple Web applications – chat, voice, collaboration, photo sharing, video hosting, social networking, etc. – becoming more integrated. Today we think of Google Docs, Google Calendar, and Gmail as separate applications; tomorrow, I’m not sure we will.

From the University of Nagoya, in Negoya, Japan comes an interesting experiment that NewScientist released to video: a simulation of a traffic jam.

The University simply told 22 drivers to cruise around at a constant 30 kmph, in a circle. At first, the cars were both equidistant and moving constantly. They found, however, that little variations eventually caused the traffic to jam up and created a shockwave of slow moving vehicles that moved backwards along the circle at around 20kmph – similar to traffic jams that are observed in real life.

The experiment has often been simulated in computer models but never run full-scale. What’s interesting is that if they had robots running the experiment, the jam would not be able to occur – it takes human error to cause the fluctuations in behavior necessary to trigger the shockwave jam.

Network traffic follows different rules than motor traffic; but again, human error can lead to “shockwave jamming” as well – which is why you’d find that the most critical traffic is often the most “robot-controlled.” Human beings do not choose the bitrate of most VoIP conversations, for example. Web browsing, on the other hand, has human beings “choosing” how much bandwidth to take – whether to watch the high-def YouTube video, the standard-def YouTube video, or to not watch the YouTube video at all.

This is not to say that misconfiguration issues can’t cause problems, but to suggest that problems with network congestion might be caused by a single, human, user. Again – this is why many companies choose to sequester the “humans” away from the business critical stuff.

One of the big problems with trying to communicate with people outside of IT is that – well, people have misconceptions about what computers are and what you can do with them. From the technical support people who are supposed to “see your screen” to the idea that computers can read your thoughts… well, to many people, computers are like magic to them, so they endow computers with magical properties.

It’s kind of a modern folklore, if you will, not too dissimilar from when people believed that witches’ hexes made cows sick, or that sneezing let demons into your body and caused sickness and madness. It’s a sad but true fact of human nature that we’re more likely to believe superstition than to admit ignorance.

But it doesn’t help when popular culture presents some of these “technological superstitions” as fact.

Recently, there’s a YouTube video going around – a clip from the CBS TV show “Numb3rs,” talking about IRC. It gets the name right – “Internet Relay Chat,” but everything else…

I recently created a video which shows the clip and explains why they get it so wrong. To you and I, this may seem like “explaining the joke,” and in doing so, losing the humor. But to someone who is unfamiliar with IRC or the Internet, it shows why you shouldn’t take your technology cues from so-called “technology”-themed crime dramas.