"Aren't Routers Boring?" the developer asked.


Add a Comment Now - We Want to Hear From You

brianboyko.jpgCompiled by Brian Boyko

We (and many other experts) believe that there needs to be an industry discussion between developers and networking groups, because it seems that people - at least the people we come in contact with regularly - talk often about how applications designed and tested to run over a LAN perform horribly over a WAN.

Because this dialogue doesn't seem to be happening much yet, we've decided to try to take the initiative and help jump-start the discussion between appdev teams and networking groups.

Despite the fact that this blog deals with network performance, and as such would most likely appeal to IT managers, network engineers, and the occasional CTO/CIO, this blog's earliest success was an article by Carol Schiraldi speaking about games programming. That article made it on the front page of DZone - it's kind of like Digg for developers. While getting a developer audience was entirely unexpected, that article is what convinced us that we were starting to produce some compelling content, and caused us to set our sights very high. Over the past week, we've received over 15,000 unique visitors; which is not bad for a site that has been up less than 6 weeks!

On the other hand, some members of DZone had complained - fairly, in my opinion - that much of what we wrote didn't really affect developers. Although we got back on their front page again with an article by Russell Wilson regarding UI development, DZone has mostly remained a bit aloof to our offerings - and who can blame them. So we asked one of their editors, Lowell Heddings, how we could better serve the DZone community (and by extension, the developer community.) He had this to say:

"My main question for network engineers is... aren't routers boring? =P"

I'm pretty sure he was trying to blow us off. But he's got a point. Aren't routers boring? So I actually posed this question in a company-wide e-mail. Here are some of the best responses:

(Continued…)


stevemaercklein.jpg
Steve Maercklein, Network Analyst:

I spotted an article posted by Lowell (which was at #2 on the front page of Dzone). The title was "Is your Database Under Revision Control?" I guess "boring" is a subjective term.

But seriously, routers and router technology will probably not be of much interest to most developers.

What is interesting (or should be interesting) is the vast missing universe of collective knowledge that developers lack with regards to network technology and app development. It is my impression that many developers (not NetQoS developers, of course) are unaware of how networks function other than a real high-level "water pipe/bandwidth" analogy.

Case in point: Bill Alderson [NetQoS Senior Consultant] recently fixed a problem involving database developers that were feeding the default 64k TCP stack with trickles of data in 8k increments. When they brought us in, they were hoping for a 30% speed increase. With simple code modifications, they were able to get roughly a 1000% increase.

These developers are really smart people. They just lacked the one crucial tidbit of info they needed to fix their problem - because they didn't know networking.

Perhaps this is the sort of info we should focus on providing on the blog? I'll bet Bill has a hundred "war stories" like the one above.

manishchacko3.jpgManish Chacko, Manager, Systems Engineering:

Routers are designed to be boring. You don't want any one router to get "interesting" because it's probably having a host of issues - buffer overrun, CPU pegged at 100%, memory, etc. Like a war zone, it's a good day when nothing happens. In the networking world, it's a good day when no alerts are sent concerning any one router.

And, like when something explodes in a war zone, everybody notices when the router goes down. No one even knows a perfectly working router exists. When it goes down, everybody and their momma is calling and breathing down your neck. You don't want the router to act funky and have the alphabet soup executives (CEO, CIO, COO, CFO, etc.) calling and demanding an explanation.

My question in return is: "Wouldn't you want routers to be boring?" Routers give us the gift of boredom and I'd rather not look this gift horse in the mouth.

It is a boring router that makes applications interesting. Imagine a world with no routers. We would have to have applications run on every machine - without any client/server model.

And of course, without routers, there would be no real Web or e-mail. In other words, boring routers make for interesting applications and vice versa. Imagine a router that was flaky and the end-user could not communicate with the server; this application will be soon abandoned for another that does not need a router.

Routers do more than pay network engineers' salaries, they indirectly pay the developer and the server team as well. You just have to see the big picture. Writing code and checking millions of code is boring, but the end result is something beautiful like Gears of War or Katamari Damacy (or insert your favorite game here.) Similarly, configuring and administrating router/QOS policies is interesting, but the day to day maintenance/monitoring is not. Oh well, life goes on and hopefully one of those boring routers will deliver this email [to Brian Boyko] or things will start to get interesting all of a sudden!"

chrisselvaggi.jpgChris Selvaggi, Senior Software Developer:

A human heart pumps blood through the circulatory system. A router pumps packets through a network.

Is the human heart boring? Well, if doing something reliably over and over for years on end, through "busy hours" and "sleeping hours", responding to attacks from within and without, then, yeah, a router is boring.

My father-in-law had a heart transplant four years ago. That was rather exciting - and not just for him. Then there was all the monitoring afterward, as well avoiding viruses, infections, etc.

Let's see... is a router important to your network? Then you'd better monitor utilization, CPU, and memory, and make sure it doesn't go down because of malformed packets, DoS attacks, etc.

Anyone who's seen a heart deteriorate, anyone who knows how prevalent heart disease is; anyone who's marveled at how important such a simple contraption is can understand why it's not boring.

Why not take a look at some Cisco blogs and newsgroups - see just how active they are with questions about configuring, troubleshooting, and upgrading routers. I bet most router administrators wouldn't say routers are boring.

I'm sure IT chiefs get excited when the routers go down. No routers, no work. Not in this day and age.

jimmcquaid.jpgJim McQuaid, Senior Product Manager:

Routers are the fundamental building blocks of all networks today. They have made virtually all earlier technologies obsolete.

As they have done this, routers have gradually taken on many additional missions. They've grown complex without compromising the basic function and reliability of routing.

As a result, however, the care and feeding of routers has remained a challenging task for experts. Command line interfaces are tedious, difficult, and unforgiving, but not exactly boring.

What do routers do?

They route IP packets in one place and out the other. They talk to adjacent routers to understand where to send packets. They maintain tables, some of them huge, about how to reach various networks through adjacent routers.

They monitor traffic - they're the primary form of network instrumentation today and Netflow is probably the leading info source about networks on the planet. Ping is the most widespread form of "active test" in the network and IP SLA and its kin are another form of traffic / network monitoring, using synthetic transaction.

They monitor themselves - the health and load of the router is a concern, so the router has to report on itself for management.

They prioritize traffic - they started out life as first come, first served devices. Now routers have multiple classes of service and multiple schemes for getting different classes of traffic serviced in the right time and manner.

And they enforce policies.

The classic "developer" issue with networks is that developers have often pretended that WANs are the same as LANs. That causes application performance problems repeatedly seen over the last decade or two: chatty, interactive designs that don't work when distance and latency intervene outside of the development lab. Yes, developers may find routers boring, but not network engineers, necessarily. Developers sometimes try to influence network behaviors at the NIC card and usually fail or create unintended side effects.

Developers, Network Engineers - please send in your stories about development related networking problems (or network related development problems) in our comments section.

Brian Boyko is the editor of Network Performance Daily and New Media Communications Specialist at NetQoS

Technorati Tags:




TrackBack

TrackBack URL for this entry:
http://www.netqos.com/MT/mt-tb.cgi/88

Comments

Great article, guys! I'm happy that I've inspired an entire article =)

I've written a short followup on the dzone link back to this page:

http://www.dzone.com/links/arent_routers_boring_the_developer_asked.html

Lowell had a bit of trouble formatting the following post, so I'm reposting it for him - many good points in it. Also, it alerted us to a problem with our comments settings! Now our comments section works with converted line breaks!

========

I'm happy to see that I've sparked discussion with my joke =)

I have to agree with a lot of the points in the article, especially the point that developers should understand the network. I've found that the best developers are those that fully understand the inner workings of a computer, a network, and the TCP/IP stack. If you've ever wondered why a lot of older developers stress that "real programmers should learn C"... it's the same argument.

With the ever-expanding growth of high level languages like C#, more and more of the underlying code is hidden away from developers. Combine that with the boom of web services, and developers are writing applications that do every single call via a web service, trading data over the network in uncompressed, bloated XML format even when it's not necessary to do so.

The explosion of O/RM layers has sparked an even bigger problem... pulling extremely large sets of data from the database when it's not needed. I'm not blaming the O/RM layers for this, they are great frameworks that are getting better all the time... if you've ever used ActiveRecord you will understand the beauty of it. Still, put these tools in the hands of a programmer that doesn't fully understand the underlying technology, and you end up with an application that does separate lookups for every single sub-item, killing network and application performance.

As an illustration, there was a page in our ecommerce application at work that pulled a list of accessories for a given product. For every single accessory item, there was a separate lookup to another table. The page ended up doing 110 queries against the database on every single page load.... I rewrote the page to use a single query against the database, and then had to argue with the object oriented people about why we should use my version (1 query) against their version (110 queries). If they had fully understood the network and database layers, they wouldn't have been arguing.

Great article... I think it's the first time I've been quoted anywhere!

Post a comment

Verification (needed to reduce spam):

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)