I remember my first computer. I remember deleting files to save space on it – files like “config.sys” and “autoexec.bat.”
I remember my second computer. I remember forcing a RAM chip in too hard and breaking off a piece of the slot.
I remember my third computer. I remember not grounding it properly.
Around that time I started working in a mom-and-pop computer repair shop. I remember accidentally reformatting the wrong hard drive.
Luckily for me, I was generally better at computer repair than the one incident otherwise this would be a very long editorial.
Does this story have a point? No, but I’ll shoehorn one in anyway: What I’ve found is that as geeks we learn through trial and error. The best of us become best at it by making mistakes and by learning from them. Sure, we may have accidentally fried a few CPUs, or, in some cases, siblings, but the point is that the surest way to learn something is to do it, learning from mistakes.
One of the problems with rote memorization and studying for certification is that it is through our mistakes that we can discover strange new things – the phrase that heralds the greatest advances in science and engineering are not “Eureka!” but “Hmm… that’s funny….”
And with a very long rant, a poster on the Overclockers Australia forums talked about how things in IT are going from “bad to worse.” Reading the whole thing is time consuming, and there’s some salty language, but it’s worth it. Here’s some highlights:
Why is it then, that over the last three years I've seen fewer and fewer people who call themselves sysadmins understand these things? Why is it that I've been surrounded by "IT professionals" from junior sysadmins to CTOs who don't have a g*****n clue about one tenth of the above? Why is it that in three years I've met ONE person in professional IT who I would consider worthy of sitting down and having a conversation with?
Why is it that professional IT services today consist of service reps who tell you the things you are doing are untested, dangerous, unsupported, different, not usual, or a host of other words meaning they are scared s***less and unwilling to learn something new? Why is it that I spend my time building things people tell me for 6 months during build and test "will never work", only to have them go into production and work ten times faster for one tenth the cost of the old system? Why is it that IT professionals today choose brand labels over intelligence, and post-justify it by hiding behind "board confidence" when providing a solid, working, profitable system is the best thing to boost confidence from the board?...
Many moons ago, I used to have a mentor. A man who quite frankly I considered genius level. I don't throw around words like "genius" frequently. In my life I've met three people who would rightly qualify as geniuses. Only one I've had the pleasure to work with, and more importantly learn from. In the small amount of time I worked with the man my rate of learning tripled. He had the right amount of sage advice coupled with the sense to let you make your own mistakes from time to time.
This, I think is one of the biggest elephants in the room with regards to network engineering in general and network performance specifically. There’s a reason we cherish “the old guys” in IT – these are the guys who were working on networks when it wasn’t a big deal – or at least, it wasn’t as big a deal as it is today – if they were broken. In an environment where fault is expected and accounted for, you had a little bit more freedom to experiment. Sometimes this leads to trouble. Other times it leads to insight. You repeat your insight and remember not to repeat your failures.
But the thing about enterprise networking is that the only people who own infrastructures large enough to support large networks are the companies who can absolutely not afford those large networks to fail or under-perform for any reason. Going with the “safe” option instead of the one that offers the potential for learning is the only option that makes sense to the business. But it is stunting IT.
I often wonder why computer security gets more coverage than computer networking. After all, computer security doesn’t do anything, it just protects the stuff that does something. Computer networking allows you to do amazing things. Some of it, no doubt, is due to the “hacker mystique” – a fear of an active agent of destruction is more powerfully on the mind than planning how to eek out a 1% performance increase.
But there’s also the fact that security, rather than networking, is one of those fields where you are encouraged to try to get in there and break things. This process of breaking things (and putting them back together) leads to more learning and more innovation… it’s a hell of a lot more fun and you learn more.
Maybe what we need is for someone – I don’t know… academia? The government? Cisco? - to develop a simulation of a working multinational Fortune 500 WAN, and just let students go hog wild on it, destroying and recreating it many times over, each time learning.
Recently, we did an unscientific (and really, I cannot stress that word enough) but real-world test of performance using the Ono plugin for BitTorrent client Vuze/Azureus. Our results were inconclusive.
David Choffnes, the author of the Ono plugin, responded to the test in our comments section of that article:
Regarding your results, it is difficult to run controlled experiments because even when you download the same torrent, you're doing it at different times with necessarily different swarms. My research group's evaluation is not limited by this, and we showed that performance improves *on average*.
Also note that if Ono doesn't find any nearby peers, it can't help your performance. You can see if Ono found nearby peers (and is using them) in the "Ono" plugin view … Also, the plugin's effectiveness is limited by the fact that "only" 180,000 users have installed Ono. The more people use it, the more likely you'll find nearby peers.
One last point -- even when Ono doesn't dramatically improve performance, it encourages better "Internet citizen" behavior. Why transfer data from halfway across the world when you can get the same data and (potentially better) performance from peers nearby? Ono makes it easier to do the latter, which should eventually help everyone using the Internet.
Ono is part of Aqualab, a Northwestern University computer science project researching large-scale distributed computing. Choffnes, a doctoral candidate, will present his findings at SIGCOMM next month, and his paper on the subject can be found here – which is great if you like trigonometric functions in your technical literature.
There’s also a telling paragraph which may explain why we got the results we did for our tests (other than just the random variability of different BitTorrent swarms), instead of a massive throughput boost.
In our analysis, we compare statistics from peers located by Ono (referred to as Ono-recommended peers) to those from all peers selected at random by the BitTorrent protocol, which also includes those located by Ono.
In Network Performance Daily's analysis, we compared statistics from peers located by Ono combined with peers selected at random from the BitTorrent protocol, against only peers selected at random from the BitTorrent protocol.
To determine the cosine similarity value for a peer, Ono must be able to compare its ratio maps with those of other peers. The latter information can be obtained in a number of ways: through direct exchange between peers, from distributed storage and from trackers. Ono currently supports the first two options. With direct exchange, when two peers running the Ono plugin perform their connection handshake, the peers swap ratio maps directly… Though Ono enjoys a large user base, it is still a small fraction of the total BitTorrent population. Thus Ono also attempts to perform DNS lookups on behalf of other peers that it encounters, to determine their ratio maps. This enables Ono to perform biased peer selection over a much larger set of peers, including those not running the Azureus client. From both direct exchange of ratio maps and DNS lookups, our Ono clients locate over 180, 000 peers per day using our CDN-based approach.
When Ono determines that a peer has similar redirection behavior, it attempts to bias traffic toward that peer by ensuring there is always a connection to it, which minimizes the time that the peer is choked. Due to limitations of the Azureus plugin API, we are currently unable to bias other aspects of peer connections, e.g., the bandwidth allocated to each connection.
In addition to Ono, Aqualab also does other projects that are designed for improving Internet performance in a number of other areas. Choffnes’s advisor, Dr. Fabian Bustamante, has been working on "sustainable scalability in distributed systems,” called the 3R project. Many P2P and internet VoIP systems are built in a way that routing is controlled at the application layer, and that in order to identify better paths and faster throughput, the application probes the network environment repeatedly, as the application has no quick way to determine whether a particular peer or node is performing well except by trying to connect to it. The 3R project seeks to decrease probing by re-using the view of the network gathered by long-running, ubiquitous services.
While enterprise networking and Internet networking are two different beasts, performance advances in one usually lead to advances in the other, and with cloud computing promising to make enterprise networking a hybrid of LAN, WAN, and Internet connectivity, these advances remain important.
Recently, a posting on Slashdot linked to a story from PC Magazine called “Texas PC Repair Now Requires PI License.” Obviously, this story has gathered tons of attention, and if strictly true, would have a major impact on IT departments across the state, if not the nation.
Today we present an interview with Capt. RenEarl Bowie, of the Texas Private Security Bureau, regarding the Bureau’s interpretation of the law and policies towards PI licensing for computer and network techs:
Editor Brian Boyko, at NPD: First of all, could you tell me a little bit about who you are and what your position is?
Capt. RenEarl Bowie: Well, my name is RenEarl Bowie, I'm with the Texas Department of Public Safety, Private Security Bureau, and I'm the Captain of that bureau.
NPD: This department basically determines who needs, and issues, Private Investigator licenses, is that correct?
Bowie: Well the responsibility of the Bureau is to regulate the private security industry. And, encompassed in that industry are individuals who are considered private investigators.
NPD: The big controversy seems to be that a lot of people in the computer industry are doing investigations which they don't consider part of the private security industry, and it seems like there's been a lot of press recently - very recently - about whether or not the day-to-day operations of a typical PC repairman or network tech would constitute private investigation under the law and therefore require a license.
Bowie: Right, and I think, Brian, based on what you're saying, is that is what the intent and the spirit of the law is, under the Texas Occupations Code, 1702.104 [which] gives us a definition of what an investigation company is, and - you know, one thing you have to look at is when you read that particular statute, the interpretation is [that] the review of computer data for the purpose of investigating potential criminal or civil matters is a regulated activity under that code.
NPD: So could you give me a couple of examples as to what would be and what would not be covered under this law?
Bowie: A basic example would be an individual like a computer repairman who is providing computer repair or support services for a customer; normally that is not a regulated activity. But when an individual is performing work involving the review of computer data for the purpose of investigating criminal or civil matters, then they could fall under the 1701.104, which is considered an investigation company.
NPD: So, maybe I could give you a couple scenarios and you could help - maybe you could explain whether or not it would be covered. For example, let's say there was a network engineer who is trying to find the root cause of a slowdown on the network, and in the course of investigating that, they discover that the root cause is some sort of criminal activity, such as a virus infection, or someone engaging in massive intellectual property violation, in other words "piracy," something like that. Would they then require a private investigation license? Would they have to stop their investigation at that point?
Bowie: Based on the scenario you gave it sounds like they're performing a repair or support service, and they're not - the intent was not to go in and do an investigation, they are just collecting information that they found, and that doesn't, based on that scenario, doesn't rise to that level of an investigation.
NPD: What about a PC repairman who is being asked to check for viruses on a person's computer?
Bowie: That does not rise to that level either.
NPD: What if a parent brought in a computer that they owned, but which is primarily used by a son or daughter, and they wanted to find out, say, the browsing history?
Bowie: That's just considered normal computer repair or support service.
NPD: What wouldn't be considered normal computer repair - can you give me a very specific example where that line is crossed?
Bowie: No, it's - when you read into 1702.104, there is some interpretation there that you have to consider. I can't give you a specific example, I could probably use some type of scenario in the sense of, for example, if an individual is contracted to come in and say, for example, investigate your computer at your company - you have employees there, and you believe identity theft has occurred, that there's been some issues and you want this individual to come in, inspect the computers, you want them to come in, perform an investigation relating to the identity, the habits, the efficiency, movement, affiliations or locations or transactions and acts, or the character of a person, or the location and disposition of lost or stolen property, or some type of damage to the system, then I think you're moving more towards the spirit of the law, and falling into an investigations company.
NPD: Okay, so once you get to that point - this is something that's considered now to be routine is, if a person is suspected of - well, you could say a number of different things. Not just illegal activity but also perhaps, unauthorized use of the network - recreational network use - would that speak to the character of a person if they're browsing YouTube at work, and an investigation is made to determine if someone is browsing YouTube at work?
Bowie: I think what you have to do is take those on a case-by-case basis, and do a thorough investigation into the matter to determine whether a violation of the code has occurred. You just have to keep in mind that every scenario and case is different, and you have to take it on a case-by-case basis, and use the utmost discretion.
NPD: What happens if for, whatever reason, someone is ignorant of the law and they violate the law accidentally - that they perform an investigation, and in their particular case, even though they didn't intend to violate the law, they did? What happens then?
Bowie: Well, then again, it goes back to on a case-by-case basis, it involves good investigative work on behalf of the investigator looking into the matter, and then you have to evaluate what occurred, and what the individual knew, and what happened - and present the case to the court or to the prosecutor, if it even rises to that level.
NPD: Why do you think that this has been so controversial?
Bowie: As for as why it's been so controversial? I believe that there are entities or individuals that just want clarification and to get some understanding in regards to the statute, and it just recently became known to the media in regards to the individuals who raised the question, and of course the law was passed last year, but it has just been brought to media attention here just recently.
NPD: Is there any way you can think of to clarify the law and the interpretation of the law even further, so that instead of having to rely on the case-by-case basis scenario, to really hammer that down, "yes this case would be considered a private investigation, and this case would not."
Bowie: One thing individuals can do is they can definitely log on to our Web site, and when they get to the [Texas] Private Security Bureau Web site, there's a spot on the Web site called Private Security Bureau Opinion Summaries, and you click on that, and it has some definitions and even examples and some clarifications of 1702, and individuals can click on that and it'll definitely provide them with a lot of good information.
Recently, a posting on Slashdot linked to a story from PC Magazine called “Texas PC Repair Now Requires PI License.” Obviously, this story has gathered tons of attention, and if strictly true, would have a major impact on IT departments across the state, if not the nation.
Matt Miller, Executive Director of the Texas State Chapter, Institute for Justice:
Editor Brian Boyko, at NPD: So, could you tell me a little bit about your organization?
Executive Director Matt Miller: Sure. We are a public interest law firm; we're based in Arlington, Virginia. We have offices now in Minnesota, Washington (state), Arizona, and now in Texas, and we file public interest litigation on behalf of individuals whose Constitutional liberties are taken away from them by government.
NPD: How many cases have you filed?
Miller: The Institute for Justice, in total?
NPD: Yeah.
Miller: Probably close to a hundred. "IJ" has been in business since 1992, and we work in four areas. We work in property rights - you may have heard of our "Kelo vs. New London" decision that came out of the U.S. Supreme Court, we work in free-speech in the areas of commercial speech and campaign finance reform, we work in economic liberty - which is what the case that we'll be discussing today is about - which challenges licensing restrictions. And then we're also the lawyers for the school choice movement.
NPD: So, could you tell me a little bit about who you are and what your position with the organization is?
Miller: Sure. I'm the executive director of the Texas State Chapter, so I run the office here in Austin. I have a staff attorney that works with me, and then we have an office manager and some law clerks from the University of Texas Law School.
NPD: So, could you tell me a little bit about this bill that has been passed into law - House Bill 2833?
Miller: Well, last Thursday [June 26, 2008], we filed suit against the [Texas] State Private Security Board on behalf of the owners of some computer repair shops here in Texas and their customers. Last year, the state of Texas passed a law that basically said that to perform a lot of types of data analysis; you have to have a private investigator's license. And, if you perform that analysis without a license, or if you are a customer and you seek to have that analysis performed by somebody without a license, it is punishable by up to one year in jail and up to $14,000 in fines.
NPD: Could you tell me a little bit about the language of the bill, where exactly it says that in the bill?
Miller: Well, what was changed in the bill - they amended the Texas Occupations Code, Chapter 1702, Section 104 of the Texas Occupations Code. And they added one little line - and it was done in subsection B, and they that for the purpose of subsection A, "obtaining or furnishing information" includes "information obtained or furnished through the review and analysis of any investigation into the content of computer-based data not available to the public."
This case got on our radar screen because the Private Security Board has issued a series of interpretations saying flat-out that this law applies to computer repair shops and a lot of people who analyze computer data in certain ways.
NPD: Sorry, what board was that again?
Miller: The Texas Private Security Board. They're basically charged with licensing private investigators, security guards, guard dog trainers - people of that type.
NPD: Alright, is that a government agency, or private function?
Miller: It is a State Agency. They are a sub-agency of the Texas Department of Public Safety.
NPD: The Lawsuit names them as the defendant?
Miller: It does. We have sued the members of the board in their individual capacity - excuse me, I'm sorry, let me correct that. In their official capacity. Which is what you're required to do when you file a lawsuit of this type against a state agency. And we are asking the Judge to declare that the law violates our clients' constitutional rights to practice their occupation free from unreasonable governmental interference.
NPD: Is the problem with the law or the interpretation of the law that the Texas Private Security Board has taken?
Miller: Well, it's with both. Laws can be interpreted in a lot of different ways, and the private security board has chosen to interpret this law very aggressively. Since the law can be interpreted in that way, there are problems with the law itself. The interpretations that the board has issues, is the reason that this case has come to our attention, because they say specifically that computer repair shops should be aware that if they offer to provide these services they've committed a crime. And that kind of caught our attention, so we started looking into it, and the law itself is problematic because it is subject to such a broad and aggressive interpretation.
NPD: Would it also affect network engineers performing network analysis on their own companies' computers?
Miller: Sure, and let's talk about that because, it is complicated and there is quite a bit of nuance. It kind of leads to how this applies to these guys. We've gotten calls from people who say, "Well, if somebody's switching out a hard drive, then that doesn't apply to them, right?" And the answer to that is, yes. It doesn't apply to them. But anyone who is analyzing data in a situation where that data points back to the actions of a third party - so, somebody who is not the computer's owner, or someone who is not the owner of the company - anytime a third party is implicated by data analysis, this law is potentially triggered.
What the board came back and did was, they said that any analysis of non-public computer data to determine the causes of events or the conduct of persons is what they're calling a regulated service. Of course, that is extremely broad. You know, for instance, if an employer went to a company and wanted to know how their employees were using the computer - that constitutes an investigation. The Board has said that when the service provider is charged with reviewing the client's computer-based data, for evidence of employee malfeasance and a report is produced that describes the computer related activities of an employee, it has conducted an investigation and has therefore provided a regulated service.
NPD: So, other than the lawsuit, is your organization taking any other actions?
Miller: We've obviously tried to bring this issue to light in the media. Because it is somewhat technical, we've had to educate the media on how this works. And they've been very responsive. But the primary vehicle we're taking here is this lawsuit and our goal is just to change the law. We're not seeking monetary damages, this is not a personal lawsuit - we're going to a judge and saying: "Judge, this is a bad law, and it stops our guys from practicing their profession - it stops a lot of people from potentially doing the things they do on a daily basis, and the law needs to be changed." So we're asking the judge to strike the law down.
NPD: Have you spoken to the author of the law? Rep. Driver?
Miller: We have not. We will do that in the due course of a part of our litigation, but we've not talked to him prior to filing this litigation.
NPD: What would happen if the judge does not find that the law is a bad law, but rather that the interpretation of the Texas Private Security Board was overly broad?
Miller: Well, in that event, then the board would be limited in the future in how they can enforce the law. And that would be a partial victory for our clients, because, if they were prohibited from enforcing the law against people who were just basically analyzing computer data in a way that was legal and that someone had asked them to analyze it, then obviously that would be a partial victory. The problem is that the law is still hanging out there, and it's going to be difficult for a judge to say that the interpretation is a problem without also saying that the law in which that interpretation is based is also a problem.
NPD: Well, is there anything else you wanted to add, anything that you think I've left out?
Miller: Well, again, I appreciate you interviewing me for this. The law is tricky, and the computer community just needs to be aware that anything they're doing that implicates third-party data or any reports they're producing for customers or for employers that says something about how a third party has used a computer is potentially regulated by this law. And they just need to be careful. We are working hard to have the law struck down in court, and we're moving as fast as possible on that, but in the meantime, people just need to keep an eye out and be aware of the issue.
Recently, a posting on Slashdot linked to a story from PC Magazine called “Texas PC Repair Now Requires PI License.” Obviously, this story has gathered tons of attention, and if strictly true, would have a major impact on IT departments across the state, if not the nation.
The law in question is Texas HB 2833, which is an updated collection of amendments to laws regarding private security services. It explains who, exactly, is required to get a private investigator’s license.
The controversial bit of the law in question seems to be this bit. The underlined part is what has been added:
SECTION 4. Section 1702.104, Occupations Code, is amended to read as follows: Sec. 1702.104. INVESTIGATIONS COMPANY.
(a) A person acts as an investigations company for the purposes of this chapter if the person:
(1) engages in the business of obtaining or furnishing, or accepts employment to obtain or furnish, information related to:
(A) crime or wrongs done or threatened against a state or the United States;
(B) the identity, habits, business, occupation, knowledge, efficiency, loyalty, movement, location, affiliations, associations, transactions, acts, reputation, or character of a person;
(C) the location, disposition, or recovery of lost or stolen property; or
(D) the cause or responsibility for a fire, libel, loss, accident, damage, or injury to a person or to property;
(b) For purposes of Subsection (a)(1), obtaining orfurnishing information includes information obtained or furnished through the review and analysis of, and the investigation into the content of, computer-based data not available to the public.
Because the law can be difficult to interpret, the Texas Private Security Bureau issued an opinion statement which clarified their position on this matter. The controversial statements there seem to be:
Computer Repair & Technical Assistance Services October 18, 2007
Computer repair or support services should be aware that if they offer to perform investigative services, such as assisting a customer with solving a computer-related crime, they must be licensed as investigators… [Text of law posted above.]
Please be aware that providing or offering to provide a regulated service without a license is a criminal offense. TEX. OCC. CODE §§1702.101, 1702.388. Employment of an unlicensed individual who is required to be licensed is also a criminal offense. TEX. OCC. CODE §1702.386.
and:
Computer Forensics August 21, 2007
First, the distinction between “computer forensics” and “data acquisition” is significant. We understand the term “computer forensics” to refer to the analysis of computer-based data, particularly hidden, temporary, deleted, protected or encrypted files, for the purpose of discovering information related (generally) to the causes of events or the conduct of persons. We would distinguish such a content-based analysis from the mere scanning, retrieval and reproduction of data associated with electronic discovery or litigation support services.
For example, when the service provider is charged with reviewing the client’s computer-based data for evidence of employee malfeasance, and a report is produced that describes the computer-related activities of an employee, it has conducted an investigation and has therefore provided a regulated service. On the other hand, if the company simply collects and processes electronic data (whether in the form of hidden, deleted, encrypted files, or otherwise), and provides it to the client in a form that can then be reviewed and analyzed for content by others (such as by an attorney or an investigator), then no regulated service has been provided.
… Consequently, we would conclude that the provider of computer forensic services must be licensed as an investigator, insofar as the service involves the analysis of the data for the purposes described above.
In order to clarify some of this and figure out what this would mean to both personal computer repair technicians and network engineers, analysts and system administrators, we contacted Texas State Representative Joe Driver, who authored the bill, Matt Miller at the Texas branch of the Institute for Justice, which has launched a suit against the Texas Private Security Bureau, and RonEarl Bowie of the Texas Private Security Bureau. We’ll have podcasts and transcripts available on this site soon.
First, Texas State Representative Joe Driver, Author of Texas HB 2833:
Editor Brian Boyko, at NPD: So, could you tell me a little bit about who you are and what you do in the Texas Legislature?
Rep. Joe Driver: Hi. My name’s Joe Driver, I am state representative from Garland, Sachse, and Rowlett area which is Northeast Dallas County. I’m the current chairman of the Law Enforcement Committee, and this is my eighth term.
NPD: How often are each of those terms, two years, four years?
Driver: Two years.
NPD: So you have 16 years of experience writing legislation. And you authored this bill, I believe it’s [Texas] HB 2833?
Driver: Yes, sir.
NPD: Now that's currently a bill, not a law, correct? Or has it been passed?
Driver: No, it's been passed. The governor signed it.
NPD: Let me just bring up the law right here - and I'm looking at it. It is "an act relating to the licensing and regulation of certain private security services." Could you tell me a little bit more about what this act was designed to do?
Driver: Basically, it was a clean-up situation for the Securities Act. We felt like we had to go in and clean some things up. Some of it was old stuff, some of it was new stuff, but basically we worked pretty hard to try to just get it so that it was easier for people to interpret and - you know, some things hadn't been changed for quite a few years, so we were going through it, trying to just basically do a real thorough clean-up, and it turned into what you'd call an omnibus bill which is basically something that encompasses a lot of different areas.
NPD: How has the law changed for people who practice investigative services?
Driver: Well, there's quite a few changes in there. I really truthfully couldn't go into all of it, I mean, it's a pretty good sized bill. Of course, the one that's - there's some area that's getting some, I don't know, "interest" out there, but I think it's interest that has been generated by a group of folks, and basically in their newsletter, they just opened a new chapter in Texas and decided to file a lawsuit. That's all in one sentence - so it sounds like they decided to file the lawsuit so they could bring some attention to their new chapter.
NPD: It does to me that the law... now, I am not a lawyer...
Driver: Me neither.
NPD: I am not a... um... pretty good reader of bills. So, what I wanted to know... The claim is that people who repair personal computers would need to get a private investigator's license in order to continue repairing computers.
Driver: Yeah, and that's what they're claiming. It's interesting that they're claiming all that, and they filed a lawsuit on the same day that they decided to open their Texas chapter. To me, I just felt it was a way they're getting a lot of free publicity, and a lot of free press, and free TV time and free radio time, because the bill to me, it says what it says. There's three words that describe somebody that repairs computers, and that's if people retrieve or provide information, and there's three words that somebody "reviews, analyzes, or investigates" that material, then, they do need to have some sort of security clearance because they're delving into people's private lives or private property on the computer.
NPD: The one thing that I noticed was that it seems very clearly that this is for personal computer investigators, like someone who does analysis to determine whether a crime has been committed or something has been stolen, or intellectual property has been violated. It doesn't seem to me that this would apply to people trying to just recover information for the person's wishes.
Driver: Right, and you're correct. You used one of the key words in my opinion, which is "analyze." "Review, analyze, and investigate" are the three key words, in my opinion, that drive the need for people to have some kind of license. Because if they're doing some of that, then they don't need to be - it doesn't need to be just anybody able to do that - they need to have somebody that has a security license. But if someone's just retrieving information and providing information for someone who is going to analyze, to use one of the words, then that's just a regular computer repair person. And those guys are great, they're good at what they do, and we never intended for them to get any kind of license other than have the ability to repair.
NPD: So, how do you think this came about - you mentioned that there was a new group - I think I may have a copy here of - are you talking about the Institute for Justice?
Driver: Yeah, yeah, that's them. Well, and I think - to me - that - I mean I've got something, I don't know if it’s a press release or just some information about them, but they actually said in here that they tell about how they're the "nation's leading litigators." They have a little cute name for them, and I thought I could think of that, but I'm not coming up with it. But, they basically said, "we fight for the rights of those violated by the government." And they're opening their new Texas chapter today (whenever this was written) by filing a lawsuit against the Texas Private Security Board. So they're kind of kicking off their opening - well, what better to draw attention to someone's opening then to get a lot of free press - they don't have to go out and advertise because - and I'm not criticizing you guys [the media] because, I'm just saying, that - to me, that's what they intended and that's part of what they did.
So. Lawyers can interpret, like you and I know, and we're not both, either one of us, luckily, they can interpret the same word three different ways if you get three different lawyers. And, I think that's what they decided to do here, and - to me, if someone reviews, analyzes, or investigates, they need a license. If they're just retrieving, providing or preparing information, that's what computer companies do, and as long as they want to do that, they're fine.
NPD: There is another possibility though - there is, - you wouldn't call them computer repairmen. There are people who work in enterprise networks, and we even have a term for it, "Network Forensics."
Driver: Like forensic scientists and all that stuff?
NPD: Not so much forensics...
Driver: That's the investigative part.
NPD: Not so much forensic scientists like a criminal forensic scientist. But for example, if a network is running slowly, not running at peak performance, there are tools that people can use to determine which computer may be slowing it down. Is it a virus - and that's all investigative work, but not investigative work related to criminal activity. It's just - so basically I'm wondering if maybe the law could have been written - not thinking about this possibility, and that maybe there might be some sort of loophole that needs to be amended. Does this just not apply to companies trying to improve their network performance?
Driver: Truthfully, you may be just a little bit out of my realm of comprehension on that, because, maybe that's something we need to look at tweaking, along those lines, to clarify that situation. We talked to lots of folks when we were writing this. Maybe we didn't talk to enough folks. But, as far as those types of things - maybe just a little bit far out of my comprehension on that. But the whole deal - like, if you have an IT person, (just cause that's all the terms I know,) IT person that somebody says, "Hey, we want you to delve into this person's computer, and find out what's going on." Well, if they delve into that person's computer, and - this is all I know about computers - and hands the information over to somebody else, then they don't need any kind of license because they're just doing their retrieving job. So, if the area you're talking about is different from that, you're probably out of what I understand and maybe something we have to look at.
But - anytime we do anything this massive, a lot of times there are areas of tweaking. But I just thought the coincidence of this particular group filing this lawsuit and bragging about filing the lawsuit on the day they opened their new chapter was just - coincidental and - because the intent of the bill was, as I've been saying, was, if you retrieve and provide information, you don't need a license.
Because I'm sure not trying to put anyone out of business. I'm a small business person, I would never do that!
NPD: What business do you run?
Driver: I'm in insurance sales.
NPD: What I'm wondering is if there is - like a specific exemption in the law that - most of these forensic investigators for network performance tend be of one of two types - the first time is that they're in-house, and that the company hires these people to do this job on the computers that the company owns. And if there's a specific exemption for investigative work on material that you own yourself. And the second, sometimes the people are hired by the company as a separate company - not direct employees, but outsourced. Is this something that might be protected under the law even if it falls under the "investigative" arm?
Driver: If it falls under the investigative arm, probably not, but I - I don't know about what you're describing to really comment more than that. I mean, I wish I did. But in this particular case, I don't. It's just a little deeper into the computer world than I know about.
NPD: You don't foresee legis-- any activity... what's the word I'm looking for...
Driver: A future bill, maybe, corrective measures, tweaking, something along those lines?
NPD: I was actually thinking of enforcement against-- you don't see this possibly being enforced against..
Driver: I don't. I don't. I really don't. I don't see - and then again, and it may be something that we may need to look at. And we may have somebody else look at it. Every time we have something like this come up we have people that want to tweak it just a little bit or change it just a little bit. And I'm not hardcore set against it. If it causing somebody problems then we ought to change it. I don't foresee it doing that but, I don't know. I mean, I really don't think it is. But if we find out that it does, that's what we're there for, to make sure it's written correctly and if it's not, we're going to change something a bit to make it right. Cause we're not after anybody, that's for sure, except the people that are doing investigative service for a living and yet, they don't want to bother with having - giving any kind of background or being qualified or licensed in any way.
NPD: That's pretty much all the questions I had.
Driver: Well, I wish I could have given you better answers. I think I kind of danced around one that - just because I don't have enough knowledge.
NPD: Don't worry about it - I'm not saying that - we're getting into some technical stuff. This isn't even a technical bill.
Driver: Not in intent, anyway.
NPD: History's full of bills that had to be amended after the fact because of something.
Driver: Well if you find out more information about it and found we really need to do something about it, call me back, and we'll get back in session, maybe we can use you for a little information, as far as how to do it right.
No, wait. Things never used to be easy. In fact, they were horribly complex and frustrating to the point where engineers pull their hair out. But now we usually expect around 99.99umpteen% uptime from our network equipment.
So frustration today often stems from the new tasks that enterprise IT engineers are expected to handle beyond the routers and switches. Application delivery controllers, WAN Optimization controllers, and more latency sensitive applications such as VoIP and Teleconferencing simply mean that the IT teams are being tasked with problems that require them to think in new ways about what it means to be in IT.
If you’ve been to any networking convention or conference, you’ve probably heard “in IT you either develop applications or deliver applications” more times than you’ve seen the Brady Bunch episode in which Marcia gets hit in the face with a football. That’d doesn’t make it any less true.
Ann Bednarz, writing for Network World, suggests that companies take research firm Gartner’s advice and look to hire “application delivery architects and engineers.” The idea is that there should be at least one person in the IT department whose full time job is worrying about application delivery and tuning on a WAN – someone who can converse with application developers and security teams and end users.
At NetQoS, we’re trying to help companies get the information they need to either designate and train an existing member of the IT staff for these new responsibilities, or at least know what to look for when hiring for an Application Delivery Engineer position.
For example, some things we’re doing right now include our NetAnalyst training based on real-world examples on resolving complex network application issues, and integrating our multiple products together in the NetQoS Performance Center.
But there are some more subtle ways in which we’re hoping to get this point across. We argue that the most important metric for network performance management is application response time. And while there’s many things that can affect application response time, the most basic is that your best possible application response time is limited by the latency of the connection (especially in financial applications,) multiplied by the number of connections that the application has to make. Network engineers often focus on only one aspect of that formula, latency – while application developers only focus on the other aspect – the connections. (That’s if they bother to think about the impact of the app on the network at all. And if they do, their test environment sorely lacks any similarity to the real world WAN.)
So the value of developing the role of the Application Delivery Engineer, someone who can coordinate the two halves of that Application Response Time equation, becomes clear.
The one thing I hate more than anything else is seeing people get the blame for something that they didn’t do because the people in charge are ignorant about technology. No, wait. Paper cuts. I hate paper cuts more than anything else. But that whole “travesty of justice” thing – very close #2.
This (possibly true, possibly not, definitely plausible) story from The Daily WTF had me ticked off. “Cam” – apparently a pseudonym - had set up a Web cam so he could prove to his bosses that he was working from home instead of just saying he was working from home. But on that day….
During a quick lunch break, Cam got a panicked call from his boss's boss, Ron. "Cam, do you still have your webcam on?"
"Yeah, wh-"
"Turn it off. NOW," he said in all caps over the phone.
"Uh, ok." Cam flicked the switch on the webcam off. "So, why exactly is it so urgen-"
"Can'ttalknowbigproblems-" *click*…
See, it seems that there was a brief but major hiccup in a router somewhere between the bank's data center and their T3 provider, causing a dramatic slowdown in outbound network performance, which rippled out into hundreds of branches and affecting thousands of online banking customers. In the troubleshooting process, the lead network engineer caught wind that Cam had been "streaming live video" over the network, and was going to tell! He complained loudly to Ron that Cam had caused the issues and lost some revenues for the bank in the process. Adding to this theory was the fact that the issue had apparently resolved itself close to the time that Cam turned off his webcam.
One week later, Cam is sitting with his boss Joel to discuss the issue. "Cam, I'm going to need you to sign this disciplinary action report before we file it with HR," Joel said weakly.
Appealing to reason, Cam began, "Joel, you know exactly what happened. You know that all that was coming across the network was a static web page with a new image every so often. I never had more than five HTTP sessions at a time. It would take thousands, if not hundreds of thousands of simultaneous users accessing my web site at the same time to consume the bandwidth that it says I consumed on this report."
"I know," he said as his expression sank. Clearly, he'd fought for Cam and been overruled.
"Besides that," Cam continued, "I'm hosting my site at my house. My upstream connection is capped at 360 kbps. There's literally no physical way that anything I did from my house could even make a dent in our massive T3 lines, even if my upstream connection was 100% saturated!"
"I know," Joel said as his face slipped into his hands. At this point, it dawned on Cam that he was lucky that all that was happening to him was a writeup. It sounded as though upper management would prefer to see him hanged. Still, it was absolutely unfair that he'd be made to take the fall.
"Furthermore," Cam pressed, "what about our QoS policies? Surely internal users browsing external web sites have lower priority than-"
"I know," Joel said again. "Look, I've fought them on this. You know I trust you, and that I know you wouldn't ever — that you couldn't ever — do something like this. I'm saying this as a friend; you're better off just signing this. It's not just you; management is pissed at me now, too. It's not fair, but it's how it is."
In the end, there wasn't much Cam could have done.
Maybe, if the moment the router started having huge performance problems, there was some sort of alert delivered to the network engineer – one detailing the problem, how bad it was, and where it was originating from, that might have helped. Someone might want to look into making something like that. Or – or, bear with me - if there was a way to look at the traffic patterns and Netflow data to see exactly how much bandwidth the Webcam was taking, providing exculpatory evidence for Cam, that might be nice. Someone should get on that.
In the meantime, I’ve just set up my own Network Performance Daily webcam to give you an idea of what my job is like. Hope you enjoy it!
I’d make a better pun, but just about the only things that come to mind when I think of Denmark are Hamlet and the Jyllands-Posten Muhammad cartoons, and I can’t draw. So, here goes:
Tis a fault to heaven, A fault against the dead, a fault to nature, To reason most absurd
-- King Claudius, Hamlet, ACT I Scene 2
In this case, a network performance problem lead to a network fault – a misconfigured “piece of network equipment” in caused “IBM’s network” to be “overwhelmed,” and that disrupted business at many Danish companies. (The quotes are not for emphasis, irony, or skepticism – just to show that the WSJ story was very non-specific about what exactly went wrong.)
Find out the cause of this effect, Or rather say, the cause of this defect, For this effect defective comes by cause.
-- Lord Polonius, Hamlet. ACT II Scene 2.
Among the consequences: 188 people went without dairy delivery for a day, and bank patrons couldn’t use their ATM cards and had to visit a real-life teller.
For that last one, the International Edition of the Halsingin Sanomat, which I totally reserve as the name of my heavy metal band if I ever start one, was able to get more answers from Sampo Bank, one of the banks affected.
"It was a more extensive malfunction, which meant that the security codes of our cards did not reach their destinations. That is why the cards that needed verification did not work right", says Sampo's head of communications Hannu Vuola.
The end result is that Denmark’s Ministry of Foreign Affairs has said that many of the affected companies are looking for some sort of compensation from IBM for their trouble. And for a country that is the “world’s most networked country,” a massive network outage is a major problem.
It is a custom More honour'd in the breach than the observance. -- Hamlet, Hamlet, ACT I Scene 4
Joel Trammell, CEO of NetQoS will be producing the welcoming address at NetQoS Symposium 2008. We asked him a few quick questions about what he’ll be talking about when Symposium starts April 20, 2008.
NPD: This year’s welcome address is called: "Why networks fail and why the role of the network engineer is secure." Could you tell me a little bit about why you chose that topic?
Trammell: Well, I think with fault management issues, people really understand the network up and down. It's very clear to them the value that keeping the network up provides. With performance, it's somewhat less clear sometimes, to people. What problems are people actually solving? What causes these performance issues? They tend to be more nebulous in nature than fault issues.
NPD: So when you say "Why networks fail," you mean "Why do networks fail to meet goals?"
Trammell: Yes, "why networks fail to perform," is really the title here.
NPD: How do networks fail to perform?
Trammell: Well often it's built around changes in the environment. There are a lot of things going on in the environment these days that add a great deal of complexity, whether it be the introduction of new applications such as VoIP, whether it be different uses of the network with folks expecting anytime anywhere access to the network, so therefore instead of just having people sitting in a building on a wired connection, you may have a wireless connection or at their homes, or at hotels, or wherever they're coming into contact with network services.
There are new technologies being deployed in the network, WAN optimization being one that has been particularly hot in the last few quarters. So all these changes introduce many opportunities to cause the network to behave in a way which is different from what the users have come to expect.
NPD: The other half of the topic is "Why the role of the network engineer is secure."
Trammell: So, yes, there's one school of thought out there in the industry that IT and particularly networking as an area will become a pure utility in the near term and that companies will no longer invest in networking, just like they no longer invest significantly in their own power distribution systems or their own power generation systems. I don't believe that's the case. I don't believe, when you think about the network - is the wild west. There are all kinds of applications being introduced, both intentionally and unintentionally on that network. Lots of technology changes going on - it's a very wild environment. It's not an environment that's conducive to a utility type approach. And therefore people with expertise in networking, and particularly performance of applications across that infrastructure will continue to find the field very lucrative.
NPD: One thing that gets me is that when people talk about the idea of the network becoming like a utility - they often bring up, "Well, people don't have their own power plants. But, they do have their own power plants - if you're a hospital, you have a power plant. If you have a data center that needs 100% uptime, you probably have at least a backup power plant.
Trammell: And people often choose to locate their facilities near where power supply is good and cheap, so even though it's a utility, geography doesn't have equal access to power at the same costs. So even power is not as pure a utility as a lot of homeowners may think of it, as when you get to use it on an industrial and commercial scale. And I think networking adds at least another layer of complexity to that.
NPD: There are significant changes of course; there is truth that Software as a Service does change a lot of things. Obviously you can't ignore that. But you still need a network to access SaaS apps.
Trammell: SaaS assumes, as sort of a priori knowledge and capability that you have this highly performing, ubiquitous network available to deliver SaaS. If anything, it makes the network even more important, because now, the only way to access the software is through the Wide Area Network.
Verizon and NBC are working on serving up TV shows to home computers. The problem is, high definition video, (and I've done some HD video work for the Web - shameless plug), takes a whole mess of bandwidth.
Now, the obvious solution for NBC would be to move to some sort of peer-to-peer distribution system, right? I mean, it works for Linux distros.
The problem is that a normal peer-to-peer connection doesn't distinguish between the cheap local links - that is, links on the same ISP, in roughly the same geographic area - from the expensive remote links. So while P2P provides a more cost effective solution, it doesn't provide the most cost-effective solution for the ISP.
A third party, Pando, has developed a P2P system for pre-authorized, pre-approved content, and has come up with a way to force peer to peer connections to look for local nodes first. This increase the efficiency of the system, lowers the cost, and generally increases the performance of the streaming/downloading video.
This is exactly the type of thing we talk about when we say that how the application is coded can have a huge impact on the application performance over the WAN. Sometimes instead of needing more bandwidth, you need to find a way to make the apps work more efficiently.
In this case, decentralized P2P systems developed after the destruction of Napster. Though they were much less likely to get shut down by the RIAA, they were also much less efficient. This dominated development of P2P applications for years. But for offering only pre-authorized content, a centralized system - especially one that takes advantage of the structure of the physical network, makes a certain bit of sense.
NBC will be offering Verizon customers their shows via Pando's P2P service - which they're calling P4P, later this year. The name is a logical outgrowth, P2P, or "peer to peer," versus P4P, or "peer for peer." P3P was disregarded because it sounded too much like PCP. And if a kid with a lisp goes around school saying: "I downloaded the latesth Methallica album on P3P" and a teacher hears: "I downloaded the latest Metallica album on PCP," well, that's just not going to be a story that ends well, now, is it?
There's only one problem with Pando's plan: Each ISP will have to give up information about its subscribers in order to participate - that is, the Pando platform requires knowing which nodes are "local" and which nodes are "remote" in order to optimize for the local connections:
For other ISPs to reap the benefits Verizon did in the test, they too would have to share information about their networks with file-sharing companies, and that they normally keep that information close to their chests.
''That's one of the objectives we have to solve -- how are we going to consolidate this data and distribute it?'' Pasko said, adding that the result of the test gives ISPs plenty of incentive to collaborate.
(Okay, maybe there's two problems: No offense to NBC, but when your biggest hit is a veritable case study in game theory… you need some new shows.)
by Brian Boyko
