| The Prudential interview | 15 Oct 2003 |
This is a transcript of a 15 Oct 2003 conference call with a group of Prudential Securities investors. The Prudential guys tell me that this call broke all their records for client participation and feedback. The transcript has been lightly edited, mainly to add links and fix punctuation and the odd typo.
Brent Thill: We're very pleased to have with us this morning Eric Raymond in a rare interview. Eric is a self-professed wandering anthropologist and a trouble-making philosopher. In addition, he's the President of the Open Source Initiative, an organization that builds bridges between the hacker community and the business side of the equation.
Mr. Raymond has been involved with the software industry for over 20 years and was a member of the VA Linux board of directors from 1998 to 2002. He also serves on Merrill Lynch's technology board. And he wrote one of the first in-depth books on the open-source movement entitled The Cathedral and the Bazaar, subsequently described by Netscape as a major factor for them to open up their client software as open source.
Mr. Raymond has a new book coming out entitled The Art of Unix Programming.
Key things we're going to cover include the misperceptions of the hacker community, the uprise of Linux as a world-class operating system and the opportunities and risks that the open-source movement is creating for Red Hat and the threat to Microsoft.
So with that, Eric, just on the adoption curve, you had mentioned in your book, The Cathedral and the Bazaar, that open source is not a new idea, that its traditions go back to the beginning of the Internet 30 years ago but until recently only technical and market forces have converged to create for open source a niche role. Can you give us a sense of what you see as the largest drivers to adoption from your perspective?
Eric Raymond: Sure. The single deepest driver is the escalating complexity of software. You're probably all familiar with Moore's Law, the idea that the capability of machines roughly doubles every 18 months. Well, what this means is that the typical complexity of software also doubles every 18 months because of market pressure to use that capacity for new features, new capabilities. And the reason that's a problem is because that means that the average bugginess of software has been increasing both in absolute and relative terms.
And the way to understand that is to understand that most software bugs result from unpredicted interactions between different pieces of the software. Therefore, the number of bugs in a piece of software is sort of proportional to the square of its size. So that means that the amount of bugs that programmers and people who maintain software have had to deal with has been rising as the square of an exponential curve.
What has happened is we discovered that all of the more traditional methods for squeezing bugs out of software have basically run out of steam. We're in a complexity regime where they don't work anymore. And increasingly, the only thing that does work is decentralized distributed peer review, which is what open source is all about.
Brent Thill: And just in terms of the adoption curve, you had mentioned that Linux really came onto your radar screen in '93, and you had been working on Unix 10 years before that.
Eric Raymond: That's right.
Brent Thill: What's created, over the last 10 years, that inflection point that we've seen for Linux over the last 5 years in terms of at the server level? We haven't seen as big a splash at the database and application level, but the server level has really taken off. Why that layer, and when do you expect we'll see more application on the Linux side?
Eric Raymond: Well, Linux itself got off the ground because of a convergence of two things. One was cheap PCs and the second was the Internet. That meant that computing and communications were available at a price point that was below anything we'd ever seen historically. And what that meant in turn was that new economic arrangements were possible for producing software, specifically the software industry has begun to decentralize, has begun to move away from a factory model.
This is actually kind of interesting because it's the reverse of what we normally see historically. Normally when industries rationalize, they do so in a way that increases capital spending and centralization. Software actually started to move in the other direction in the late eighties and early nineties, and it was PCs and the Internet that enabled a sort of new cottage industry of people who had their brains and their fingers and didn't need expensive capital equipment.
If you project that forward, you see that more and more of the software industry is being captured by a non-factory mode of production, and servers — that happened with server software first because, in a sense, that's the easiest. That's the infrastructure that's broadest and most susceptible to a purely technical attack. It doesn't have pesky considerations like end user interfaces and so forth. So servers happened first because it was easiest.
And what's happening now is the open-source community is going through a somewhat difficult but achievable process of learning how to cope with things like user interface design.
Brent Thill: And 5 years ago there weren't really any desktop applications on Linux. It seems that that's changed. I think the last count there was something like 6000 different applications that are being built for the desktop. When does that start to impact companies such as Microsoft? It's obviously impacted them on the server, but do you see as big of a hit on the application side as we saw in the server side?
Eric Raymond: Oh, definitely. The question isn't if that will happen; the question is when. And to demonstrate that this is very likely, I won't just make arguments based on my own experience. I'll make arguments based on what Microsoft is saying. Microsoft has repeatedly stated both in public and in private internal memoranda that have leaked that it regards Linux and open source as the only serious strategic threat that it faces, and a very serious one.
Brent Thill: Okay. So if you had to put a time scale on it, when do you see companies such as Windows and some of the other desktop applications being more seriously threatened by Linux, next two to three years or beyond that time frame in terms of when you see adoption?
Eric Raymond: I don't know. I actually thought that the process would be further advanced than it is by now. I'm not sure why I got the time scale wrong. But because I know I did, I'm reluctant to make time scale predictions. What I can say is we have here a classic case of what Clayton Christiansen called disruptive technology. So what we've seen is open source got established in niches first. And what we're waiting on is the breakout into the general desktop market. It will happen because Microsoft hasn't cannibalized its business to bring it down to a price level where it would be competitive with open source. And, by the way, what is that price level? Approximately zero.
Brent Thill: So many have characterized Linux as the rise of good enough, and I think you've made the case multiple times that it's better than good enough and it's perhaps even much better than the commercially built systems from closed-source companies that are visible. What are the underpinnings of that argument in terms of why and how that process has managed to build better software in an open-source community versus an established, in your words, cathedral type establishment?
Eric Raymond: It all goes back to peer review. In response to one of your questions, I said that traditional methods of verifying software have petered out because they don't work very well at today's complexity scales. What does work is having a lot of people look at the source code. It's really that simple. Secrecy is the enemy of quality. And the open-source movement and its allies in industry are the people who have learned the converse of that statement, which is that peer review is the best friend of quality.
Brent Thill: So when you talk about the coordination of the open-source community, you know, you've mentioned that it's developed into a world class operating system. Linux has been developed by part time hacking by several thousand developers. One of the biggest misperceptions I think in the community right now is how do you manage a process like that? And when you talk to Red Hat, they can't even tell you when the 2.6 kernel is coming. So how do you manage the opposite end consumption expectations of when these features are going to come?
Eric Raymond: Well, let's dispose of an illusion first. There's a false comparison here. When people say, how do you manage open-source software, they're implicitly assuming that closed-source software can be managed. This is a belief that is actually false. Things take as long as they take. You can set deadlines in closed-source software, but our record at being able to fix both deadlines and a ship date is very, very poor.
There's a folk theorem that over 70% of projects fail either during their development phase or are not accepted by customers after delivery. That's a really shocking failure rate. And what it goes back to is this inability that any development system (whether closed source or open source) has, to simultaneously set deadlines and ship dates for anything as complex as software.
So that's one background fact to keep in mind when you ask, how do we get open-source software produced on a deadline? The reality is we can't get closed-source software produced on a deadline. When you try, generally what you get is a pile of crap that falls apart the moment somebody pokes it. So that's one thing to bear in mind.
When you ask, how do you manage open-source software, I'll continue that thread by saying that you manage it rationally the same way you manage closed-source software, which is to accept that you can set one of two things. You can either prioritize features and set a ship date and say "we'll take the features that are done by this ship date", or you can say "this is a set of features that I need, wake me when it's done". And open-source and closed-source software rationally are both managed by the same rule, which is: pick one of those.
Brent Thill: I think one of the other big misperceptions on open source is that how does the open-source community get paid, and how can they coexist with commercial software vendors such as Red Hat to create a common goal? I think you had mentioned that roughly 40% of the Linux development team is actually employed by corporations. Can you further extrapolate on that theme?
Eric Raymond: Sure. I'll give another example that I think is even more interesting. There's a group called the Apache Software Foundation. Apache Software Foundation is a user consortium that pays developers to work on the Apache web server, which is the open-source software that runs over 60% of the world's websites. The interesting thing about the Apache Software Foundation is that in one sense it barely exists. All of the people who are core developers for the Apache Software Foundation are sitting in corporate cubicles at one or another large web server user.
These web server users have banded together to support the Apache Software Foundation because they've realized that it is through cooperatively funding an open-source effort they can get the best, most flexible, most capable software. So what you have is you have a core group of developers that is actually being funded to work on Apache part time or full time, typically upwards of half time, by the project's corporate sponsors. And this is a very typical and increasing way for open-source software to be funded, not out of its sale value by some organization that is a market intermediary between users and developers but directly by the users themselves.
Brent Thill: And as it relates to Red Hat, there's been an ongoing debate that Red Hat is going against the grain of the open-source community. I think that misperception is not valid. Can you talk about where your group, the open-source community, and the interface between Red Hat, where that relationship is at and how important that is to your movement?
Eric Raymond: Sure. In general, the open-source community is pretty happy with Red Hat and always has been. There's been a certain amount of grumbling, but there's always going to be a certain amount of grumbling in any community of people as fractious as open-source developers tend to be. But Red Hat has a secure and recognized place as a leader of the community. We're happy to have them as a corporate ally. We're happy to have corporate allies of other kinds as well. The community has developed a fairly close relationship with IBM, which is pretty entertaining if you've been around long enough to remember when IBM was the Great Satan the way Microsoft is now.
We write our software and we put it on the Internet because we want people to use it. And if the most effective way for people to get to use it is for it to be packaged and, in effect, retailed by commercial allies, then we're quite happy with that development. We don't mind when other people make money from our work. The only thing that bothers us is when people claim an exclusive right to make money from our work, when they want to deal us out of the process and take control of our product. That, we won't stand for.
Brent Thill: And as it relates to that topic, where does The SCO Group (SCOX) sit in terms of where you think they're headed with their framework at this company?
Eric Raymond: Gee, I should not start uttering a torrent of profanities, should I? They're thieves and carpetbaggers of the worst sort, and they're headed for a major fall.
Brent Thill: So the argument that they, in the case they've had against IBM and others in the industry, how valid are their claims?
Eric Raymond: Not at all. I'm, among other things, a historian of technology. I'm a historian of Unix. I know a great deal about the factual basis behind SCO's assertions. I could talk for an hour about this, but I've written papers about it, which you can find on my website and I'll just say pretty much everything they're claiming is nonsense. And furthermore, they know it's nonsense. This is not inadvertent. It's lying.
Brent Thill: So the perception from many that they're using this lawsuit as a perverse marketing scheme to bolster their revenue would be an accurate description.
Eric Raymond: Well, let's look at their recent 10-Qs and note that 25% of their income and the only source of income that is rising for them rather than dropping is payments from Microsoft. I think that pretty much tells its own story.
Brent Thill: On the open-source movement, many have said the bigger threat will first come to Unix, which has shown up in some of the market share numbers that third party analysts have put out and, you know, secondly will eventually reach Microsoft. What are your thoughts in terms of how that is progressing? I know you've been involved with the Unix community for quite some time.
Eric Raymond: Tell it to the citizens of Munich or to the citizens of South Africa or of Korea or any of a number of other places where we have recently seen huge waves of Linux desktop adoption. And Microsoft may like to talk about how Unix is going to get damaged first, but the threat and the damage to Microsoft's desktop market share is already here. It's already happening.
Brent Thill: I guess one of the bigger threats that faces Microsoft as well as an open-source movement is the issue of security. And if you could comment from an architectural perspective where open source and Linux fits in terms of security as it relates to commercially built systems and where you see the pros and cons to both.
Eric Raymond: Sure. Well, first I'll make an assertion about the facts, and then I'll explain those facts. The factual assertion is that Linux systems in general are much, much, much more secure than either Windows or closed-source software in general. You do not see in the Linux world any phenomenon analogous to things like the SoBig virus or any of the other viruses or worms of the week that cause so many problems. This is not an accident, and it is not simply a result of the fact that there are lots of Windows systems out there. If exposure to the world were the main driver of software’s incidents of security vulnerabilities, then Apache would have huge numbers of cracks, bugs, and holes in it because, as I pointed out earlier, it runs somewhere around 60% of the world's websites. But, in fact, if you look at the statistics, you discover that Microsoft IIS, which is far less used than Apache, is cracked a great deal more often. Therefore, it is not frequency of deployment that mainly drives the bug statistics. It's the software's intrinsic vulnerability to security problems.
And closed-source software has a very high intrinsic vulnerability to security problems. Why is that? Well, now I'll get into the causal explanation. More than 100 years ago there was a Dutch military cryptographer named Kerckhoff. Before computers, before electronics even, he studied military cryptographic systems back in the day when that meant signal towers and mirrors and stuff like that. And what he noticed was that in order for any crypto system, any code system to be really secure, it had to depend only on the secrecy of its keys, not the secrecy of its methods. That is relying for your security system on the implementation method, on the code, as it were, being secret is a short route to disaster.
The Germans found this out in World War II. They had a top-level military and diplomatic cipher called Enigma, and they relied for the security of Enigma on the fact that nobody knew how the machine that did the Enigma encryption was constructed. Well, the allies were able to read their codes because they were able to use some patterns in the traffic, repetitive headers in military radio messages, to slowly deduce the construction of the machine. And once they were able to do that, they were able to crack the codes and read them pretty much at will.
The same lesson actually applies to software. If you rely on the secrecy of your code as opposed to the secrecy of your passwords, what happens is the normal operation of code leaks information about its internals, even if the bad guys aren't smart enough to get their hands on a copy of the source code, which they often are. Because the normal operation of code leaks information about its internals, eventually you can reconstruct enough about what's going on inside to crack it.
What's great about open source is that it forces people implementing security systems not to take the bogus path — the doomed path — of relying on the secrecy of the code. And that's why open source is more secure than closed source because, in effect, it forces people to do the right thing, which is to rely only on the security of the keys.
Brent Thill: So understanding the underpinnings of the whole movement, many have speculated that Microsoft will eventually have to port [Office to Linux]. Is there a way that Microsoft could coexist with the open source movement?
Eric Raymond: It's an interesting question. It's hard to see how because they're used to a level of pricing and a level of profits that can really only be sustained if you have monopoly lockdown. So in that sense I don't really see how they could survive.
Brent Thill: Okay. And from an investment perspective, if you looked across the landscape of companies that are out there in Oracle, Dell, Red Hat, a number of technology companies have jumped on this movement, how do you position who are the best positioned names, what collection of companies do you think have identified this from a public perspective that are supporting this movement?
Eric Raymond: I would say the best investment possibilities are companies who can use open-source technology to boost the value of our product bundle without being pure play software producers themselves. The prospect of capturing high returns from pure-play software companies I think is increasingly a mirage. I don't think you're going to see that happen anymore. And the reason again is fundamental.
What is it — let's look back at basic economics and think about what is it that enables us to capture high returns from founding and floating a company? Well, basically it's capital investment. You need to look for business opportunities where it's possible to scale up returns by pumping in lots of capital and then collecting those scaled up returns. That's not going to happen much anymore in software because we've passed out of a regime where you can scale up returns by pumping in capital.
The reason we used to be in that regime was because computers were expensive. So by pumping in capital, you could afford to equip the business with capital goods that would greatly raise productivity. Also, software distribution was expensive. You had to move physical media around, and that's also something [else] that you could enable by pumping in a lot of capital.
Well, both of those bottlenecks are gone now. Computers are really cheap, and distribution over the Internet is nearly free, and that means there isn't really an opportunity to multiply returns by pumping in capital, and the ultimate consequence of that is you shouldn’t bet the farm on a pure play software company. There are still ways for companies like that to make money but not at a volume that is going to sustain a lot of stock market investment.
So I'd say look for companies that are not pure play software but for which software is a critical enabler, something that greatly raises the value of their products, and look for the companies that are most successfully harnessing the energy of the open-source community and working with it to raise product value.
Participant #1: I'd just like a little bit of clarification if I could. It's very easy to demonize Microsoft, but let's take Oracle as an example. Aren't they a similar company to Microsoft, closed proprietary software? So why is Microsoft a demon and Oracle a friend?
Eric Raymond: I wouldn't say that Oracle is a friend exactly. There's sort of a — they're working with us in some respects, but they haven't joined their destiny to ours in any sense. I guess friendly neutral would be a better way to put it. If you want to know specifically why Microsoft is frequently regarded as an enemy, it's because they have a long history and a lot of present day actions that are aggressively aimed at locking down the market, at shutting out independent developers and making independent development impossible.
Participant #1 : And Oracle doesn't do that in the database space?
Eric Raymond: No, no, they don't. And the best evidence that they don't is that we have flourishing open-source database projects, MySQL and PostSQL, which are actually gaining a pretty good amount of traction.
Participant #1: You wouldn't say that Oracle's actions against People Soft and everything else don't show them trying to close out competition in the application market?
Eric Raymond: Fights between different closed-source vendors basically don't concern us very much.
Participant #1: I was thinking more about the outcome of the fight.
Eric Raymond: What outcome are you anticipating?
Participant #1: There's less competition in the application space.
Eric Raymond: Okay. You may think my point of view is kind of peripheral on this, but to me the difference between three closed-source vendors and one closed-source vendor isn't very significant. Since I can't review any of the code, I don't want it.
Participant #1 : Thank you.
Participant #2: Thanks. I had two questions. First, I've chuckled at your comments about SCO because I tend to agree with you. I'm wondering, to the extent that a company like SCO tries to hold customers hostage, what are you seeing in terms of customers getting together and trying to unify to stand up to, you know, this open bribery or extortion?
And secondly, as open source gets more broadly adopted within the corporate space, I'm wondering what kind of magnitude of developers, how the numbers changing going forward in terms of the number of people contributing to open source and whether or not you're seeing a lot more of the corporate side help contribute IPs such that their work finds its way back into open source going forward.
Eric Raymond: Okay. In answer to the first question, I don't have a lot of visibility on what SCO's customers are doing. I mean, in the abstract I sort of know who they are. They're big franchise chains like McDonald's, but I don't have a lot of visibility into what they're doing, and I don't know if or in what way they're responding to SCO's behavior.
Participant #2: It's actually not SCO's customers to the extent that they're going after the open-source community and trying to extract — you know, they talk about the number being basically — if they get paid a billion dollars in this by the collective industry in response to their lawsuit and then it goes away. So I'm wondering it would be natural to assume that customers, you know, big enterprises would get together and somehow put up a unified front to fight SCO's efforts. And I'm wondering what you're seeing in that regard. Is that happening?
Eric Raymond: So far that's not happening. I think it's not happening because big vendors with a stake in open source, IBM, Red Hat, and so forth, I think there's a perception among customers that they are effectively carrying the ball on this.
Participant #2: Oh, so it's the vendors that are fighting the battle on behalf of the customers?
Eric Raymond: That's right. Now in response to your second question, I understood your question to be: as there's more corporate participation in open source, how do I see the developer population shifting?
Participant #2: Yeah. Because my understanding is that most of the open-source contribution has been from people maybe such as yourself who aren't corporate developers that I'm wondering whether or not the next phase of open-source contribution will come from the work that's being added to open-source by the corporations that are more aggressively adopting it, particularly on the server side.
Eric Raymond: Well, sure. Yeah. That is a trend. I pointed out the Apache Software Foundation earlier. There are user consortia of that kind being founded all over the place, and there's lots of in-house development at corporations that is being put in the open-source pool now as well. Again, for sound economic reasons, which is basically that if you do that, you basically get to co-opt the rest of the world to help you. You actually get advantage out of the fact that most of the smart people in the world work for somebody else. So, yeah, that is an increasing trend.
Participant #2: Okay. Maybe one other question. You're a developer at heart. A lot of people that work at Microsoft are similar to you and maybe don't share some of the, you know, corporate views of the leadership team there. I'm wondering whether or not there's a risk of defection of the people that maybe don't share some of those philosophical perspectives up in Redmond as they see sort of the open-source movement being a free version of what they've been evolving over time.
Eric Raymond: To a certain extent that's already happened. I routinely get memoranda leaked to me from inside Microsoft by disaffected techies. For any of you who are interested, there's a series of documents on the www.opensource.org website called the Halloween Documents. Most of these are Microsoft internal memoranda that were leaked to me through various interesting routes that clearly began with Microsoft employees.
Brent Thill: To Wendell's point, Eric, do you think that is starting to accelerate from over the last year in terms of these defections?
Eric Raymond: I don't know. Other than the memoranda themselves I have very little information about what people are thinking inside Microsoft, so I can't say I know anything about that.
Participant #3: I want to go back to Red Hat for a minute if I can. When Red Hat packages or takes pieces of open-source software and puts them in a product and then puts it out there I guess for free but charges services for it, and then they develop supporting products around their operating system and charge money for that, how do you figure out when they're crossing the line between an exclusive claim on profit for the open-source community and when they're getting a justified profit from their own efforts?
Eric Raymond: They would be crossing the line if they tried to close up the source code or to put it under a license that forbade other people from modifying it and selling it. It's a very bright line, and they've never crossed it, and I don't believe they ever will.
Participant #3: They were doing something where you could get it retail. And now they're taking it off their Internet site, and they're just going to market it through the partnerships. Does that ring a bell with you?
Eric Raymond: Okay. There has been a
shift in their mode of distribution. It's actually a very interesting
development. They've terminated the consumer line. Instead they've
turned what was the development of their consumer line over to a
project called Fedora, which is now a joint effort between some
university somewhere and the Red Hat engineering staff. Those people
will be producing a distribution which will essentially be distributed
over the Internet both as ISO images that you can download and
incremental upgrades that you can pull onto your machine at any time
and install. The reason is, apparently they weren't making a lot of
money on the consumer distribution. They want it to still be there as
a technology incubator for their enterprise addition, which they sell
to large corporations, and apparently that's where they make their
money. But the really exciting thing about this is the incremental
upgrade capabilities that they're putting in place. There is a utility
called apt-get, which I have on my machine now. I'm actually running
the Fedora prototype. And what you can do is whenever you feel like
it you can say "apt-get dist upgrade", and it will go to
their repository, look at all the binary installable packages that
have been put in that repository since the last time you did your
update, download them onto your machine, and install them for you.
Instant update, instant free updates. This is big.
Participant #3: Okay. Then if they make money servicing that or developing supporting software products and services, that's okay with you?
Eric Raymond: That's right. Oh, that's absolutely fine.
Participant #4: Hi. I'm still kind of confused, frankly, about the economic model for the industry, the open source industry, i.e., the bevy of developers that are out there, how are they going to earn their living? And when you talk about the benefits of free updates, etc., for a company that is out there trying to have a business that involves supporting Linux, if Linux is such a secure and a reliable and robust platform, as it seems to be, in the end of the day there doesn't seem to be really any, quote, unquote, market. And I know that the idea of profiteering is somehow disdained by the open-source community. However, there is the basic economic reality out there that, you know, people need to make a living and —
Eric Raymond: Right. They need to pay rent.
Participant #4: — yeah.
Eric Raymond: Okay. Let me start with your last assertion first. The open-source community does not disdain profits. There is a very small ideological wing that is hostile to profits and markets. But that was never, even at its most visible, that was never more than 15% of the community. And now I'd say it's down to about 5%.
The reason a lot of people have the idea that the open-source community is hostile to profits is because that's a very noisy 5%. So, yeah, let's dispose of that first. We're not hostile to profits. We like markets. We're happy with profit making companies and profit making situations. The only place where we draw the line is at secrecy and exclusive control. That's a different proposition from whether you make profits or not.
Participant #4: On that same point, let's say Red Hat, they have this enterprise edition server that they develop, and they have engineers that they pay to develop that product. Now you're saying that they wouldn't have a problem — let's say they put a lot of effort and energy and resources into developing this product and then you take the CD, you know, the image of the server, whatever it is, and you put on the Internet and then anyone can download it for free. Is that something that they support? And then how can they support that? How in the long term can they justify putting resources to work that have no payback?
Eric Raymond: Oh, they do have a payback. What Red Hat is actually selling is the role of being a trusted provider. Not only do you get the bits, but also you know where they came from, and you know that they came from people with a trackable reputation who are responsible for the content.
Participant #4: But you don't have to pay for it theoretically, right? I mean, you're saying if Red Hat started selling that stuff and not allowing it to be distributed for free, then you would have a problem with it in an open source — am I correct in that?
Eric Raymond: As I said, if they tried to put their source code under a license where you couldn't duplicate and modify it, yes, then we'd have a problem with that.
Participant #4: There seems to be a logical problem — they're trusted as a resource because they're accountable.
Eric Raymond: Right.
Participant #4: But if they're giving it away for free, they can't be accountable to everybody that's just using it for free.
Eric Raymond: Well, no. If you buy a copy with Red Hat's brand on it and with Red Hat's seal on it, that's the only circumstances under which they'll support it. You can download the binary image, but then you're out in the cold. You can't get support from them.
Participant #4: Right. But back to my first point about if the code is so robust, which apparently it is, why would you need support?
Eric Raymond: I'm really glad you asked that question, because the answer is: the world changes. The world always changes. The code that is perfectly adapted to your situation today is going to be not so perfectly adapted to it tomorrow. As your requirements change, as your hardware changes, as the kind of situational stress you place it under changes, there is always room for improvement. There is always room for a service industry. No matter how good the quality of your code gets, you cannot perfectly anticipate the future. And that's why there's always a requirement for support and maintenance.
Participant #4: So could you just help me understand the completely, as someone ignorant to the service industry, what the nature of those services would be precisely? In your mind, is it tweaking the code and, therefore, specifically to a corporation, is that what it is or —
Eric Raymond: That's part of it. Another part of it is customization to an individual customer's needs. And a third part of it is timely delivery of updates from a trusted source so that, for example, you know you don't have Trojans in the bit stream you're getting.
Participant #4: So the initial service would come with updates, but those updates would also be free for anyone else who could find them, right, if they took the effort to find them themselves?
Eric Raymond: That's right. But they —
Participant #4: So it's the convenience of having it — it's sort of like in a way the same logic for the legal downloading of music. You're going to pay for it if the price is right if it means you're going to get a good copy.
Eric Raymond: — that's right. That's an interesting parallel. That hadn't occurred to me before.
Brent Thill: Eric, it's Brent. One of the big questions, if I can just follow up with a couple, and if anyone else has a question, feel free to jump in. But, you know, Red Hat has a couple of hundred developers. A lot of investors have questioned their viability in terms of, is a couple hundred enough to create the next generation platform? And, you know, what's the barriers to entry for another company such as SuSE or someone else that's trying to replicate this? Maybe Oracle is trying to replicate this internally to create their own version. What are the barriers to entry on Red Hat's business model, understanding they have a first mover advantage?
Eric Raymond: Sure. First, let me correct a misperception. Red Hat doesn't only have a couple hundred developers. They have somewhere around three-quarters of a million. They have the entire open-source community feeding value to their product. In fact, you could make the claim that Red Hat, in that indirect way, in the sense that it's capable of leveraging the entire community, probably has more developers working for it than the entire rest of the software industry put together. So that's a different way of thinking about it that perhaps is a little more illuminating about the sources of their success.
As for what the entry barriers are to somebody duplicating what they've done, well, has somebody duplicated what they've done? There are a lot of other Linux distributors out there, many of whom actually predate Red Hat in the market, and yet none of them is as pre-eminent in that market as Red Hat is. I think the correct answer is it's all in execution, and it's all in how you maintain a positive relationship with your developer community and your customers. And to beat Red Hat at what it's doing, some other company would have to be better at that game. Is that a useful answer?
Brent Thill: Absolutely. So to not think of Red Hat as 200 internal engineers, to think of it as three-quarters of a million —
Eric Raymond: Well, —
Brent Thill: — with the community.
Eric Raymond: — yes. Yeah. Think of them as the visible tip, the visible spearhead of the entire movement, which in a sense they are. They're not the only spearhead, but they're certainly one.
Brent Thill: How do you position the relative strength of some of the distributors in open-source community?
Eric Raymond: Red Hat clearly is the big dog. That's as true inside the community as it is outside it. SuSE is pretty strong technically, but they seem to be a market force mainly in Europe and outside the U.S.
The only other distribution that is really on developers' radar a lot is Debian, which a lot of you probably haven't heard of. It's a nonprofit distribution founded pretty specifically to keep the commercial vendors honest. They pioneered the kind of incremental upgrade that Red Hat is now doing through the Fedora project. They have a very large base of installed packages. They've long been a favorite of hard-core Linux developers. It's going to be interesting to see what kind of future they have now that incremental updates, which was their major selling point, is something that's being done by the Fedora project.
So I'd say from inside the community the distributions to keep an eye on are Red Hat, SuSE, Debian, and maybe Mandrake. I guess [Mandrake is] the most successful of the Red-Hat-based distributions.
Brent Thill: Okay. And just from a vertical perspective, financial services have led the charge in terms of adoption, it seems, in mass quantity. At least if you look at Red Hat's base, that's really where the first customers have adopted. What are the barriers for other industries such as retail or automotive, airlines? Are there barriers for other verticals that are greater for them to adopt, or what are the dynamics from your perspective?
Eric Raymond: It's interesting that you mention retail, because that's actually a relatively high-penetration area for us. There's a lot of stuff going on in Linux point of sale systems and related areas, you know, supply chain, inventory control. Some of those other verticals we haven't got a lot of traction in yet. I don't really know why. I don't have a clear understanding of what distinguishes one vertical from the other.
In general, in my model of the changeover, verticals are probably the last to go after horizontal infrastructure and middleware. But I can't really distinguish between different verticals very well.
Brent Thill: Okay. And you touched briefly on some of the emerging companies such as MySQL, and I was wondering if you could touch on from their perspective of the database layer how big a threat they are to SQL Server and Oracle.
Eric Raymond: Eventually SQL Server and Oracle are going to get rolled over. They're going to get run over. When that will happen, I don't really know. I suspect that that's predicted by the general open-source adoption curve rather than being something that's going to lead that curve.
Brent Thill: And would you see a natural progression to move from the server to the app server to database applications in that order?
Eric Raymond: Yeah. That's exactly what's going to happen. This is something I actually predicted back in '97, '98, and we've been seeing it happen, which is that open-source software started at the bottom of the stack with operating systems, network layers, and so forth, and has gradually been climbing the stack up towards applications and out towards the user.
Brent Thill: If you put your position in the position of Steve Ballmer, the CEO of Microsoft, how would you react to this movement knowing what you know?
Eric Raymond: By trying to make us illegal, which is exactly the thrust of their long-term strategy. They back things like the DMCA, Palladium, and so-called “trusted computing”. They have a two-phase strategy. Phase one is technological lockdown of the hardware. That is arranging things so that you need a cryptographic key issued by Microsoft to even get your operating system to load and boot on a motherboard. And the second phase of that strategy will be to use a DMCA and other legal means so that if you circumvent those protections, you go to jail.
Brent Thill: So from that perspective, that's their only defense?
Eric Raymond: That's right.
Brent Thill: And that defense move doesn't seem to be something they could sustain for the long haul?
Eric Raymond: It's an interesting question. The legal system holds on to bad laws and discredited mechanisms a lot longer than the market does. So if they were able to get that kind of system in place, it could long outlast the technical viability of their software. It’s something to keep an eye on. It's something I worry about.
Another possibility is an attack through the patent system.
Brent Thill: Last question for me, Eric, and we'll see if there's any other questions out there, please feel free to jump in, is what would be the biggest misperception right now that you hear on the open-source movement? What is the one or two topics that you think are most misunderstood that you could add clarity to?
Eric Raymond: Okay. First, there's something in our preliminary conversation you brought up, you said I should talk about this, and it's a relatively small point. It's kind of a historical one, but I'll start with that.
The word “hacker”: some of you have probably been misled by lazy journalists into thinking that hackers are people who break into computer systems. This is a vulgar error. “Hacker” properly refers to somebody who is an enthusiast at programming, an artist about it. It started to be abused by journalists around 1985, and this is actually a point that matters because there are two different cultures out there, one of which is the one I come from. That is the hackers. We are the Unix developers. We are the people who built the World Wide Web and most of the Internet infrastructure. We build things.
There's another culture out there, the people we call “crackers”, who are the people who actually run around producing viruses and breaking into systems and doing things like SoBig, and they break things. That's the fundamental difference. We build things, they break them.
Understanding that distinction is important because to the extent that people confuse those two cultures, they set up conditions under which we could see some very nasty laws passed that would eventually have a bad impact on peoples’ ability to communicate and eventually even their political freedom. So that's a good reason to be a little careful about that confusion.
The next time you see somebody claiming that a hacker is a person who breaks into computer systems, please correct them. It may be your own freedom that you're saving. So there's one misconception.
Another important misconception is that we spend all our time hating Microsoft. We don't. I mean, I talked about Microsoft in this call because it was clearly something that people wanted to hear about, but we don't in fact spend a lot of time hating Microsoft. You know, you hear that line from some of our younger and more testosterone-filled developers. They're adolescents. They're out there looking for fights to be in. That's fine. It's normal. That's part of the process of development.
But for most of us, most of the time, it's about doing good software. It's about doing good art. It's about building something that works and lasts. And if Microsoft happens to be in the way of that, that's a problem. And if Microsoft is doing things technically or illegally that are attempts to shut us down and monopolize the market, that's a problem, and we deal with that problem. But there really isn't a lot of percentage, there really isn't a lot of benefit, in spending extra energy hating them. So that's two important misconceptions.
And the third one I guess I'll come back and reinforce is the idea which persists in a lot of circles that we're somehow anticommercial or opposed to people making profits. I hope I've dispelled that impression during this phone call. But in case I haven't, I guess maybe I should explain why a lot of people have that idea. It comes from the fact that for a long time our most visible or loudest public advocates were people who came from that very small fringe of developers who had an anticommercial ideology, but that's no longer true.
The open-source community has found a voice for what the other 85%, 90% of the community believes. One of the voices for that viewpoint is me, and we're out here saying what has really been the majority viewpoint all along, which is we’re quite happy with the market, as long as it doesn't try to enforce secrecy, as long as it doesn't try to enforce single-point control.
Participant #5: Eric, just had a question for you following up on Brent's question regarding database and application server, you know, middleware opportunities. At the open source level there's been great corporate sponsorship for Linux adoption, particularly from Oracle and IBM. Who do you think could provide that sponsorship in order to move up in some kind of a mass fashion in those other layers?
Eric Raymond: Well, to some extent this is already happening with projects like, for example, Open Office where you have Sun and several other industry players sponsoring an attempt to produce a complete office suite which can read and write all of the Microsoft file formats. And that attempt is pretty far advanced. 1.0 is out, and I know a lot of people who are not techies who are using that software for production. I know that that is the impetus behind some of the pretty large-scale adoptions we've seen by governments all over the world.
So that kind of sponsorship is happening. It's coming partly from the Fortune 500. A lot of it is coming from political entities that do not want their internal security to be hostage to whatever Microsoft feels like doing this week.
Participant #5: How about specifically in database and app server, though? Is there a technology vendor you can envision that would be viewed as a long-term survivor? I would argue that many people don't take Sun seriously on a long-term basis. And we know IBM and Oracle aren't going to provide the sponsorship.
Eric Raymond: IBM's Eclipse project?
Participant #5: Could you tell me about that a little bit?
Eric Raymond: Sure. That's an application development and middleware project, which IBM is directly sponsoring. Another thing to look at is IBM's Websphere product, which is essentially a complete middleware suite for people who want to do e-commerce.
Participant #6: Couple of questions real quick. One sort of thing I'm stuck on is you mention Mandrake, and I believe they're trying to distribute a desk top offering Linux system, is that correct?
Eric Raymond: That's right.
Participant #6: Now if I'm understanding — I guess I might be stuck in sort of an old way of thinking, but I can go out and buy their software, right? I can go purchase boxed software, a Mandrake desktop system, is that your understanding?
Eric Raymond: Yes. They have a retail box set.
Participant #6: So then what would they think if I took that and just put it on my server and then let anyone else download — in other words, I don't need support necessarily as a consumer or as a desk top user. I don't necessarily use a lot of support from Microsoft or anyone else.
Eric Raymond: Right.
Participant #6: I just buy software and use it. Do they have or does the open-source movement have a problem with let's say me taking this Mandrake CD and then distributing it to — you know, available to anyone for free so that now they have no way of capturing revenue from this product?
Eric Raymond: Are you aware that Mandrake makes ISO images of its distribution available on its own FTP site?
Participant #6: No, I'm not.
Eric Raymond: I believe they do.
Participant #6: So then how are they going to sustain themselves? How do they envision sustaining themselves if they want their product to be given away for free and there's not really a need for support necessarily?
Eric Raymond: Well, I think I addressed that before. First of all, there's always a need for support. People who want support will pay for it, and there's ways to capture revenue there.
As for the second, who makes money from end-user support anyway? That's a really difficult business. The profit margins are really low, and there's a lot of friction.
Participant #6: Right. Exactly. So how is Mandrake going to stay around if they can't sell their software and they're not going to be getting money on support?
Eric Raymond: The same way other people who make money off support make money off support. They find a few large customers who have particularly intensive needs and a lot of money, and they rely on that business.
Participant #6: Okay. Also, I was curious, you mentioned Red Hat has the support of, you know, three-quarters of a million developers, i.e., I guess the hackers out there that are interested in developing stuff for Red Hat.
Eric Raymond: That's right.
Participant #6: If you have an object or a piece of code that you develop let's say for the Red Hat Linux system, is that transferable to let's say the SuSE system or any other Linux system, or —
Eric Raymond: Good question. Thank you for opening up that question. Yeah, you're right. I need to explain this. I personally am responsible for 35 individual software projects. That's an unusually large number. Most people are responsible for, at most, one or two. Each one of my projects is a program or a set of programs that can be built and shipped as a unit, which people might want to use on their computer somewhere.
I do not ship to customers directly. People can download the software from my site if they want to, but that's not the main distribution channel. Instead what you get are distributions or organizations that exist to pull from the net the best available projects, the highest quality designs, the freshest software, package them in ways that make them easily installable, and some of those distributions then sell the product on the open market. Okay?
So the sense in which Red Hat has three-quarters of a million developers working for it is that there are people at Red Hat whose job it is to watch what the community is doing, notice when there is a package or a package update there that has value for Red Hat's customers, and then grab that and put it in their inventory for the next version that they release. Is this making more sense?
Participant #6: It is. And let's just say I started another company, Blue Hat or whatever, and I just take those same objects or those same projects from your site and easily — does it take a lot of individual customization to make it coherent within my suite, or is it very, you know, plug-n-play as it were in terms of — in other words, can SuSE take the same project as Red Hat and put it into their—
Eric Raymond: They can. The licenses that we all use guarantees that anyone can take my software and put it into distribution, whether that's for a profit or nonprofit. A distributor does a couple of different things. I'm really glad you raised this question because all of the people on this call need to understand this aspect of the market. A distributor does a couple of things. First of all, they apply judgment about what selection of software will best meet their anticipated market's needs. That's a really valuable service in itself.
The second service that they provide is that they take what we call tarballs, that is essentially pieces of source code that you sit down and run a make on and run a compiler against and produce binaries, they take these source code tarballs and they turn them into binary packages in one of several formats, of which the most important is RPM format, Red Hat package manager. Another important format is deb, Debian packages. They take the packages that are as shipped by the development groups. They put them in binary packages, they apply a certain amount of local customization, and they usually integrate them with some preferred desktop presentation. And that packaged, customized, tweaked, tuned aggregation of packages is what constitutes a distribution, and that's what they ship.
The third function that they perform is quality control. They typically do a certain amount of internal testing of the software that comes in, and sometimes they coordinate external testing. So three things the distributors do: one, selection, selection for their customers’ needs; two, packaging in a way that makes installation easier and reduces the burden of an administration; and third, they do a certain amount of quality control. This is where distributors add value.
Participant #6: Okay. Thank you very much. And I have one final question, and hopefully it's a quick one, about — just wanted to get your thoughts on this Red Flag Linux that China has developed and kind of understand a little bit more about that. It's sort of an open-ended question, but we'd love to hear your thoughts.
Eric Raymond: Sure. Well, there's two really interesting things about that development. One is that it illustrates the major driver for governments to adopt open source. Governments all over the world have a concern and, on a historical basis, it's quite a legitimate one, that closed-source software may contain back doors which are intended to provide a window into their secure operations for, let's be quite blunt and frank, the U.S. National Security Agency. This is a major driver behind, for example, the fact that the government of Norway pioneered going to open source several years ago. Even though they're an American ally, they didn't want their operations to be transparent to anybody at Fort Meade who cares to listen.
And after — there were a couple of flaps that reinforced this concern, one being the revelations about the NSA's Echelon system, which it turned out that, yes, indeed, the NSA has been doing data swap arrangements with British and Australian intelligence services to spy on not only our allies but even domestically in the U.S. for many years. This has been going on since shortly after World War II. There was an initial project called Shamrock, and that turned in Echelon and so forth and so on. So there's historical evidence that the NSA regards using back doors in communications systems and computer software as part of its remit, even against friendly governments.
And the second issue was when it was discovered that there were a couple of undocumented cryptographic keys in Microsoft Office Suite, one of which was actually labeled internally NSA. And at the point when that came out, a lot of intelligence agencies all over the world said, wait a second, we can't afford to have Microsoft Window's software, office software in our sensitive applications, that's a Trojan horse right in the heart of what we're doing. And that is a hidden but very powerful motivator behind even friendly governments going to open-source software where they can audit everything that's going on. And if that motivation applies so powerfully to governments like Norway and Germany, which are, in recent history at any rate, staunch American allies, how much more powerful is it going to be for communist China who's a bleedin' enemy and knows it?
So that's the scoop on Red Flag. Now that's half of the story. The other half of the story is that the communist Chinese government's effort to enforce Red Flag on its citizens isn't working. It's backfiring. The reports I hear out of China is that a lot of people are taking these systems with the Red Flag distribution put on them, and they're stripping that off and they're installing pirated software from Microsoft and elsewhere. And that really doesn't surprise me.
Moving to open source works very well when it serves what people perceive as their actual needs. When you try and force it on them, it backfires. I don't think that's bad. I don't think that's wrong. I think that software should serve people's needs, not somebody else's predetermined agenda, whether that predetermined agenda is that of Microsoft or the Chinese government.
Participant #7: We talked just a little bit on patents and the danger that they pose. It even looks like some that are directed against even companies like Microsoft could ricochet into other areas like the, I think it's Eolas patents. Could you comment on the intellectual property and protecting it? In open source is the way that you protect your intellectual property the GPL license?
Eric Raymond: That's one way that we do it. I guess that's the major way. Yeah. Inappropriate intellectual property protections are probably the biggest, single, long-term threat to what we're doing. The DMCA is really, really bad news, terribly bad news because it can be — and, in fact, already has been used — as a pretext for shutting down open-source development that is competitive with fairly narrow interests. So this is a long-term problem. It's one that I don't really know how to deal with other than by publicizing the problem enough to create political pressure against the exploiters, against the people who want to use these laws in a damaging and dangerous way.
Brent Thill: Eric, thank you very much for taking your time. This has been very educational, and we really appreciate your insight on the open-source movement, and I look forward to staying in touch with you going forward.
Eric Raymond: Thank you very much.