The Cathedral and the Bazaar is a series of essays about what the linux kernel development taught the software world. They cover the unique (at the time) speed of development and quality of Linux, how to apply its important lessons elsewhere (including the commercial world), why it worked, where else it worked (and where it did not).

In the title, Eric S. Raymond (ESR) contrasts two styles of software development. The “cathedral” is a tightly controlled effort by a small, closed team running to a centralised plan. The “bazaar” is an open, continuously released and extended effort by hundreds or thousands of developers.

A lot of commercial software (and a few high profile open source projects like the Emacs core) is created like a cathedral, while Linux and most of the open source world followed the bazaar approach.

A new movement in open source

Before Linus Torvalds began Linux development, the Unix world was stagnating. Critical academic research groups had lost funding and mainframe platforms lost commercial viability.

Linus founded a community outside of these ivory and commercial towers, by harnessing the energies of thousands of enthusiasts at Internet scale: “…a great babbling bazaar of differing agendas and approaches”.

ESR (and most of the open source world) initially believed that beyond a certain limit of complexity, the bazaar approach would break down. However, the Linux kernel was far beyond that point, and it flourished: “the Linux world not only didn’t fly apart in confusion but seemed to go from strength to strength at a speed barely imaginable to cathedral-builders”.

“the Linux world not only didn’t fly apart in confusion but seemed to go from strength to strength at a speed barely imaginable to cathedral-builders”

With the evidence disproving him, he put effort into understanding why he was wrong and these essays are the result.

Overcoming Brooks’ Law

Software development management is haunted by Brooks’ Law, which states that adding more people to a late software project makes it later. There is communication overhead while building software, and it scales with the number of developers working on a project. In order to effectively write software with Internet scale collaboration, there had to be some way around Brooks’ law.

The key observation is that not all software activities are equally affected by Brooks’ Law. In particular, the practice of debugging has less need for communication, and is highly parallelizable.

This rings true for me: in projects where contractors had no context and no hope of implementing fuzzily stated business requirements, they remained effective in addressing bugs. “When I click X, I should see Y” requires very little context.

This is so central to the success of the Linux kernel that ESR dubs Linus’s law: “given enough eyeballs, all bugs are shallow”. Instead of being the stuff of detective novels (“I couldn’t figure it out until I ran it on a Tuesday”), one of thousands of debuggers will have the exact configuration necessary to reproduce the problem instantly: “Somebody finds the problem… and somebody else understands it”.

ESR dubs Linus’s law: “given enough eyeballs, all bugs are shallow”

Brooks’ Law is not dismissed with this discovery. Amongst the core developers who set the direction of the project, the communication overhead remains. This group is however a very small group compared to the number of people submitting bug reports.

Understanding open source and the success of Linux

ESR’s analysis of the open source world after Linux’s success takes on cultural, psychological and economic dimensions.

Gift culture

Culturally, the open source world is a gift culture. Unlike ways of gaining power that are based on controlling scarcity, in a gift culture, status is gained based on what you can give away. The open source world is not characterised by a drive to survival, and there is no point trying to create scarcity by controlling bits.

ESR compares open source with other gift cultures: “Thus the Kwakiutl chieftain’s potlach party. Thus the multi-millionaire’s elaborate and usually public acts of philanthropy. And thus the hacker’s long hours of effort to produce high-quality open-source code”.

“Thus the Kwakiutl chieftain’s potlach party. Thus the multi-millionaire’s elaborate and usually public acts of philanthropy. And thus the hacker’s long hours of effort to produce high-quality open-source code”

Release early, release often

Psychologically, the habit of releasing software continuously - sometimes multiple kernel releases per day - gave a significant incentive to contributors: write a patch and you’ll see your name in lights almost immediately. The book popularised the phrase “release early, release often” which describes this habit.

ESR also spends a lot of time talking about the quality and speed gains from release early, release often. It’s worth printing this paragraph in full:

“In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you’ve winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect… In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.”

Release early, release often is crucial to the agile and devops communities that followed, years after. The different emphasis I’ve encountered there is about building the right thing (real use will tell you if your software is useful), which is probably a consequence of that discussion taking place in a world where money is involved: people don’t like paying for software to be built if it isn’t going to do what they need.

Use value over sale value

Economically, open source sacrifices up front payoff (money paid once-off before use, or sale value) almost entirely. ESR’s explains how this is a minimal sacrifice. In fact, relying on sale value to fund software is in conflict with some fundamental realities.

Most professional developers are paid by large corporates to keep software working and relevant in the face of changing dependencies. They are paid to make sure use value continues, and not based on what it would cost to develop the same system from scratch. He estimates that this is 75% of professional software development work.

The use value of software drops to zero if it is unmaintainable (by virtue of being closed) and there will be no maintenance (because the company is working on the next product to be sold, or goes out of business). If there are no forthcoming updates, you’re likely to search as fast as possible for a competing product that isn’t going to lose all support, and invest in that so as not to be caught flat-footed.

Unfortunately, earning all the money up front for sales is in conflict with spending money on maintenance. Optimising for sale value produces companies whose software looks good on “shelves” (software being sold in physical stores was more relevant in the 90s) but is never used. Real use allows bugs to be identified, “creates” maintenance work, and therefore reduces profit if your only revenue is sale value.

More use value, less risk, cheaper

Open source focuses instead on use value. This value can completely cover professional developer salaries. Examples of commercial value gained by contributing to open source innclude cost sharing (as in the Apache web server which was built together by many parties who would not be able to create it alone) and risk spreading (as in the case of Cisco print spooling software that was unlikely to outlast its creators’ tenure at Cisco).

Indirect sale value

In service of generating revenue (not just reducing costs or risk for existing value), ESR discusses several models (some examples are my own):

  • “Loss leader / market positioner”, like Netscape open sourcing Mozilla to ensure that their server business was not locked out of the market that served web pages by an Internet Explorer monopoly.
  • Software (e.g. drivers) produced by hardware companies - whose core business is not software - can be open sourced to save maintenance costs, but also to gain bigger markets when outside developers fork them to integrate with new platforms
  • “Give away the recipe, open a restaurant”, open sourcing software as marketing for expertise in the use of that software, as in Automattic open sourcing WordPress but selling hosting on wordpress.com
  • “Accessorising”, like O’Reilly supporting prominent open source practitioners as marketing for the books about open source that they sell.
  • “Free the future, sell the present”, in which software is sold for a fee, but guaranteed to become open source after some period. The guarantee makes the purchase less risky for the buyer, and gives the seller the benefit of some maintenance help.
  • Certification of a standard, giving away the software but retaining tests that can prove (for a fee) that a fork is up to the official standard, like Sun did with StarOffice
  • “Free the software, sell the content”, as in Auttomatic selling themes for WordPress.

Where open source can (and cannot) be applied

ESR is not telling a story of “open source or nothing”. He discusses where open source has a strong case and where it does not.

If software is unique and could not be developed by anyone else, there is a case for extracting rent based on the intellectual property. On the other hand, if the engineering knowledge that underpins it is in the public domain, there is a case for open sourcing, and applying one of the business models above.

There is a spectrum between infrastructural software to specialised application software here. Infrastructure like networking software and operating systems will be developed by someone, eventually, if there is nothing suitable currently. At that point they can sell it - and risk losing that revenue when a competitor emerges - or they can open source it, and gain collaborators from amongst all the people who have similar needs.

At the other end of the spectrum, specialised application software is likely to include knowledge that fewer people have, and therefore competition is less likely to emerge. The more universally useful the application is, however, the more risk that someone will build the knowledge out of necessity (or realise that there is handsome profit to be made).

An important note here is that this spectrum shifts left over time. People used to write operating systems and programming languages just to enable a single project. Some of those became successful open source projects because of their universal usefulness. Nowadays there are very few closed languages being sold commercially, and they are only picked up for specialised use. Operating systems and programming languages are now infrastructure.

Later, companies needed networking software, then databases, web frameworks, mobile, VR, machine learning, and many of these were open sourced too. Someone or some company builds something they need, and then realises that it can be open sourced and survive beyond the initial needs. Examples after the Cathedral and the Bazaar include Ruby on Rails, Angular, React, and TensorFlow.

Twenty years later

Although the essays were written over several years, we now have twenty years of hindsight with which to review these essays.

The conflict between sale value and use value is illuminating, even in my current work at a product company. It also reflects reality very well: Software as a Service has become big business, and even commercial giants have embraced it (think Microsoft Office 365). Paying for ongoing use is clearly something that customers are happy to do (although in some cases they had no choice due to monopoly), and appears to be much more sustainable than old pay-once models.

In addition, commercial software giants produce loads of open source software. Microsoft’s .Net ecosystem used to only be affordable if you already had millions in revenue, but now it is mostly (all?) open sourced. Google, Facebook, Slack, Shopify, Stripe, Amazon, Netflix etc. are all in on the open source game. From this perspective, it seems like open source won, and the competition died out or joined in, apart from highly specialised exceptions like Matlab or Jetbrains’ IDEs, or monopolies like Microsoft Office.

As noted, “release early, release often” seems synonymous with modern development, although I still meet people who talk about bugs as evidence that one needs to be more careful and slower with releases.

The most notable blind spot for me is Apple. The App Store is a marketplace (that sounds familiar) completely controlled by Apple, and accessible only if you accept Apple’s vendor lock-in on hardware and OS. This is a radical departure from the vision of the book. There’s even a line in one of the essays about logical consumers not accepting vendor lockin.

One point which still remains valid is that Apple software is for personal computers, and not used for infrastructure, apart from the unusual case of Amazon recently offering racks of Mac Minis in AWS.

Cloud infrastructure is similar, but maybe not as egregious a departure. If you rent a server in the cloud you can still install anything you want on it. However, there are still significant incentives to sticking with AWS if you are hosting there, and you’re not going to be able to use ECS with non-EC2 hardware underneath it. I’m not a Google Cloud or Azure user (that says something in itself), but friends who are seem to be stuck in one of those worlds and not split across many.

Conclusion

For all the things these essays might not have foreseen, they do provide an amazingly coherent framework with which to understand the software world. There is a lot more to the book than I’ve described here, for instance ESR’s fetchmail project, where he set out to reproduce the Linux development process to test his understanding, and then wrote down the results.

There is also a lot in the detail that I’ll be going back to again and again when I need to make decisions about how to run software projects. There are many fundamental ideas here that seem relevant in- and outside the open source world.