Open Source Hardware

Back in 2010 I stopped buying test servers from Dell and began building them from components using Intel i7 processors, X58-based mother boards, and modular power supplies from Ultra.  It was a good way to learn about hardware.  Besides, it was getting old to pay for Dell desktop systems with Windows, which I would then wipe off when installing Linux.  Between the educational value of understanding the systems better, selecting the exact components I wanted, and being able to fix problems quickly, it has been one of the best investments I have ever made.  And it didn't cost any more than equivalent Dell servers.

For this reason, a couple of recent articles about computer hardware caught my attention.  First, Dell is losing business as companies like Facebook build their own customized servers.  Open source database performance experts like Peter Zaitsev have been talking about full-stack optimization including hardware for years.  Google built their original servers using off-the-shelf parts.  Vertical integration of applications and hardware has since gone mainstream.  If you deploy the same application(s) on many machines, balancing characteristics like cost, performance, and power utilization is no longer a specialist activity but a necessity of business.  It's not just cutting out the Microsoft tax but many other optimizations as well.

Second, developments in hardware itself are making custom systems more attractive to a wide range of users.  A recent blog post by Bunnie Huang describes how decreases in the slope of CPU clock speed increase over time mean you can get better cost/performance by building optimized, application-specific systems now than waiting for across-the-board improvements.  Stable standards also drive down the difficulty of rolling your own.  Components on mid-range servers are sufficiently standardized it is easier to build basic systems from components than to put together a bicycle from scratch.  Try building your own wheels sometime if you don't believe this.

Easily customizable hardware has important consequences.  At a business level, Dell and other mainline hardware vendors will adapt to lower margins, but the market for generic, mid-range appliances has evaporated.  Starting around 2005 there was a wave of companies trying to sell open source databases, memcached, and datamarts on custom hardware.   Most seem to have moved away from hardware, like Schooner,  or folded entirely (like Gear6 and Kickfire).  The long-term market for such appliances, to the extent it exists, is in the cloud.

The other consequence is potentially far more significant.  The traditional walls that encapsulated hardware and software design are breaking down.  Big web properties or large ISPs like Rackspace run lean design teams that integrate hardware with open source software deployment.  This not just a matter of software engineers learning about hardware or vice-versa.  It is the tip of a much bigger iceberg.  Facebook recently started the Open Compute Project, which is a community-based effort to design server infrastructure.   In their own words:

By releasing Open Compute Project technologies as open hardware, our goal is to develop servers and data centers following the model traditionally associated with open source software projects. That’s where you come in.

Facebook and others are opening up data center design.  Gamers have been building their own systems for years.  Assuming Bunnie's logic is correct, open hardware will apply to wide range of devices from phones up to massive clusters.  Community-based, customized system designs are no longer an oddity but part of a broad movement that will change the way all of us think about building and deploying applications on any kind of physical hardware.  It will upset current companies but also create opportunities for new kinds of businesses.  The "cloud" is not the only revolution in computing.  Open source hardware has arrived.  

The Inimitable Mr. Steven Jobs

There have been countless articles praising Steve Jobs since he announced his retirement from Apple on August 25th.  Most either catalogue Steve Job's many triumphs or assess the impact of his creativity on society.  Those are entertaining topics but not especially useful.  A more practical question is why Steve Jobs is so good at creating new products and whether the rest of us can imitate him.

Steve Job's best work seems to follow a repeated pattern.  Let's call it the Apple pattern, though of course it could just as well be the Pixar pattern or Next pattern:

1. See the whole picture of some crucial human/technology interaction and recognize gaps.  
2. Design products to fill those gaps that combine artistic sensibility and innovative technology.
3. Get a large organization to implement designs in a way that makes the end result like the handiwork of a single highly-focused craftsman. 

Two things about the pattern seem particularly striking.  First, Steve Jobs is a complete package.  I have been in the tech industry for over three decades and have met people who did one or at most two of these things at the level necessary to create products that move large markets.  Almost nobody does all three.  The fact that Steve is excellent in all areas simultaneously may be a root cause behind his long run of successes.

Second, Job's ability to drive implementation teams is extraordinary.  Maybe it's just the manager in me, but I find his ability to pick the right people to run teams and to keep those teams pointed in a clear direction without product-destroying compromises quite remarkable.  This is far harder than generating ideas in the first place.  The heart of the Apple pattern as as much about understanding people as technology--not just users but the creators as well.  I have never heard Jobs make pronouncements on team management, but there is an excellent talk from Ed Catmull of Pixar that summarizes the tensions quite well.

Steve Jobs is commonly compared to great inventors like Edison, Ford, and Disney.  When thinking about imitation, another parallel seems more illuminating:  John Churchill, Duke of Marlborough and hands-down the greatest English general of all time.

A possible Jobs ancestor?

Marlborough possessed a seldom equalled ability to see war as an integrated whole across geography and branches of arms, devise unexpected strategies to exploit the weaknesses of his enemies, and execute them flawlessly in the difficult conditions of early 18th Century campaigns.  Execution extended from handling fractious allies down to the painstaking work to ensure his men had proper meals after each day's march.  In other words:  analogous problem-solving abilities to Steve Jobs, translated into the field of warfare.   The parallel extends to the lavish praise of contemporaries and later historians.  Winston Churchill famously described Marlborough as follows.  

He commanded the armies of Europe against France for ten campaigns. He fought four great battles and many important actions ... He never fought a battle that he did not win, nor besieged a fortress that he did not take ... He quitted war invincible.

Grand problem-solvers like Marlborough and Jobs are sufficiently rare they tend to be one-offs who change society but leave no obvious successors.  English military superiority on the Continent waned after Marlborough's retirement.  Something similar will likely befall Apple after Jobs, current happy talk about product pipelines and cash position notwithstanding.  It is simply not possible to imitate Jobs by committee, which is effectively what will happen once he is completely absent.  The driving force is gone.

That said, we can all imitate Steve Jobs, albeit on a smaller scale.  Many highly successful products start with a single person who conceives the idea and drives at least the first couple of iterations to completion.  Seeing the whole problem, applying innovative designs to solve it, and managing the team to get it done is a fundamental pattern that applies across a wide range of endeavors.  Here is just one of many examples.

Many years ago at Sybase I worked for a manager named Mark Deppe.  Early in the 1990s Mark learned that Wall Street firms were patching together crude publish/subscribe messaging applications to move data between financial systems in order to speed up trades.   He recognized that there was a much better way to do this using log-based data replication and built the Sybase Replication Server product.  The Rep Server went on to generate hundreds of millions of dollars in sales.  It still sells well today, over 15 years later.  Mark was a great architect but also a great builder of teams.  He paid as much or more attention to hiring and managing people as he did to technology.  He trusted the people he hired, and he gave them the freedom and support to do great work.  At the same time Mark was also incredibly attentive to detail and did all the project management for the first releases himself.  Years later he said it was too important a task to hand off to anyone else.

Mark Deppe was the best technical manager I ever worked for.  I have consciously imitated his best practices for many years.  Looking back it seems I was unconsciously imitating the Apple design pattern.  But perhaps that was not a complete coincidence.   Before joining Sybase Mark was at Apple where he worked with (guess who?) Steve Jobs.

------------------
NOTE:  After this article was published I found the flow hard to understand and edited it a week or so later to make it more readable.  The argument is the same as before.

Is Apple Good for Innovation?

Just about everyone on the planet agrees that Apple products are the soul of innovative design.  But are they good for innovators?  For me the answer is "not so much."

I have been using Apple laptops and iPhones for years.  As a software developer, I have a list of annoyances with Mac OS X starting with Apple's incomprehensible management of Java.  However, Mac OS X is far more productive than MS Windows, with its viruses, crummy OS releases, and bloatware.  iPhones are close to worthless as telephones in the area where I live in large part due to ATT's network.  But you can now switch to Verizon, so that's not such a problem either.

The real problem with Apple is that their products are closed.  Want to install a new file system?  Not here.  Want to pick a different motherboard to play around with power utilization?  Try somewhere else.  Want to know what the OS is really doing under the covers or (gasp) inspect the source code?  Dieu forfend!

Innovation in my chosen field of databases is increasingly based on breaking down the dividing lines between hardware and software to manage massive quantities of data economically and quickly.  The more I learn about hardware, the less I want fully integrated products.  I want devices I can interact with and learn from.  I want visibility into internals.  I want works-in-progress, not ready-made perfection.  In short, I want open platforms that give me the parts but do not tell me what to build with them.

A few weeks ago my iPhone dropped on the floor and shattered.  The replacement is a Droid 2 Global running Android.  The user interface is clumsy.  You have to watch out for viruses again.  But the hardware is lightning fast.  There is a free-for-all of people inventing new Android applications.  The source code for Android itself is available on code.google.com.  The open nature of Android is rapidly making it the locus of innovation for mobile devices.  I feel at home already.

It's All about the Team

Earlier this week Giuseppe Maxia blogged about joining Continuent as Director of QA.  Creating high quality systems for distributed data management is a hard but fascinating problem.  I have been hooked on it myself for many years.  Guiseppe brings the creativity as well as humor our team needs to nail this problem completely.  I'm therefore delighted to know he will be focused on it.

That said, I'm even happier for another reason.  Beyond solving any single problem, Giuseppe strengthens an already strong team.  Ed Catmull of Pixar gave a great speech a few years ago about managing creative teams and why successful companies eventually fail.  Among other things he asked the question whether it is the idea or the people who implement it that count most.  His conclusion:  great teams implement good ideas to build great products.  But even more important, great teams can turn bad ideas into good ones, then go on to build great products from those ideas too.  Pixar has proved this many times over.

I believe strongly in the power of great teams to create great products.  Giuseppe, welcome to the team. 

Building the Open Source Hackers Cooperative

It is striking how much harder it is to make money from open source than to write it in the first place. Open source development is a sophisticated and well-understood social activity. However, the economic model is often laughably primitive: "if you build it, they will come." That applies to the question of turning your open source project into a real job. More interestingly, it applies to the question of how to make open source projects as valuable as possible to the largest number of people. In this post I would like to propose an answer to both questions.

To illustrate open source sophistication, just look how easy it has become to start and manage projects. It is almost a cookie-cutter procedure. You pick one of a number of well known licenses, manage the code on SourceForge.net or Launchpad, communicate with the project through skype and mailing lists, and tell the world about it using your blog plus Twitter. Within an afternoon you have set up infrastructure to support efficient collaborative development with team members from Seattle to Singapore. The number of projects itself makes the point: SourceForge.net alone has over 230,000 projects.

Things get much harder if you want to make a decent living from a successful project. It’s not that there is a lack of models to choose. You could form a commercial venture that offers closed source/licensed versions of your open source project. However, as many of us have seen with MySQL AB and other companies, commercial efforts have a tendency to conflict badly with broader community interests or those of other companies that might be useful partners. For this reason, they don’t even necessarily maximize the value of the original commercial effort.

You could of course try to create a foundation like Apache or Eclipse. However, these generally require established software and large commercial backers. The current experience of the fledging Open Database Alliance and other efforts shows that it can be quite complex to create such organizations from scratch even with well-established code and well-known players. This is not a model that can be repeated across a wide number of different projects large and small.

Finally, you could sell consulting and support for your project. The problem with the consulting model is that it is not scalable—in order to make a decent living you have to work on customer problems. The pro-bono work like extending the project or tending to a community tends to fall by the wayside unless you can get a customer to pay for it.

The fact that it is hard to make money off open source is a symptom of a larger problem: we are losing the wider social benefits that for many people are the real point of open source software. This is an economic problem: how do we allow hackers to make a reasonable living on open source projects while maximizing the long-term value of the software to the widest possible number of users? It turns out there’s a reasonable economic model that can do this: cooperatives, which are defined as follows in Wikipedia.

a legal entity owned and democratically controlled [equally] by its members

Cooperatives have existed for centuries in many different forms and have successfully solved problems ranging from providing student housing to delivering consumer goods like sporting equipment on a grand scale. We need a new form of cooperative that I propose to call the Open Source Hackers Cooperative, or Hackers Co-op for short. The Hackers Co-op has three different but co-equal types of members:

• Hackers, who are the core committers to the project and stewards of the code
• Sponsors, who supply funding and labor
• Community members, who use the software, test it, and contribute patches

As we will see, some members are more equal than others, which is why I added brackets in the cooperative definition.

Here are the organizational principles for an Open Source Hackers Cooperative:

1. The Co-op is a non-profit. It’s not for sale and there is no exit strategy. Like all co-ops, it exists to maximize benefits for its members.
   
2. Hackers work directly for the Co-op. Their time is divided between implementing features that interest them, integrating patches as well as fixing bugs reported from the community, and implementing features for sponsors.
3. Sponsors provide long-term funds and/or labor to the Co-op. Sponsors build businesses on the open source software and kick back a percentage of the business value in return for support and new features. They can also contribute labor for specific co-op tasks. Sponsors need not be for-profit businesses. They could in some cases even be governments or NGOs. The point is that they fund the software based on its value, for example through grants.
   
4. Community members provide leverage to the development model. They use the software, provide basic support through forums, and contribute patches for bugs and small features. These activities leverage the hackers who can use community patches and feedback to evolve the software. This is a very efficient development model.
5. The Co-op is a democracy. Co-op members vote on allocation of hacker resources. The vote is structured to keep a single group of members from hijacking the entire Co-op by dividing hacker time into a pie with allocations for the interests of each member type. Sponsors vote for their section of the pie using a vote weighted by their relative funding contributions--sponsors are not equal to encourage competition to commit more funds. Hackers effectively vote their portion by doing whatever they want with the time they get for personal projects. Community members vote through surveys or some other reasonable mechanism.
6. The Co-op has elected officers. There is a chief economist who is in charge of the business model and plans finances, contracts with sponsors, arranges employment contracts, etc. There is also a chief technologist who ensures project infrastructure and moderates technical discussions. Co-op officers are elected or at least approved by the members at large.
   
7. The Co-op pays dividends. Some open source projects are quite valuable, so a well-run co-op could easily become very profitable. The excess profits are distributed to hackers in the form of retirement and other benefits, to sponsors in the form of cash rebates, and to community members in the form of conferences, hiring of new hackers to work on features, improved infrastructure, etc.

You can try to run through a number of scenarios for the hacker co-operative to see how well the model holds together. All you really need is software with enough intrinsic value that it can sustain an active, technically aware community and where sponsors are motivated to build businesses on it but do not need to own the engineering. It is helpful to have community members be programmers, but you can also design the software to allow even non-technical users to contribute effectively.

Such conditions hold for a variety of system software like database management systems, app servers, and communications packages. With a little thought they could apply broadly to user applications like accounting systems or voting software, which so far are relatively rare in open source. The Co-op is model is quite stable, because it aligns interests in such a way that everyone does better if they stick together.

1. Hackers earn a stable, comfortable living and public recognition for working on software that they enjoy. “Comfortable” in this context means salary and benefits equivalent to a typical EU country like Germany or Finland, which are the gold standard for employee compensation. This works economically because hackers are (a) very productive to begin with and (b) become more so by leveraging an active user community. Hackers are motivated to produce because the more viable the software is, the better the co-op does, and the more benefits they receive.
2. Sponsors get features that they need using a productive open source hacking development model. This is a replacement for models like trying to take the software private and farming it out to low-cost offshore locations, which experience has shown to be badly broken on a number of levels. More important, sponsors get stability in the sense that the software cannot be taken over by hostile corporate interests and is supported over the long term, which lowers the risk of building businesses on it. They are motivated to contribute more in order to vote more resources for tasks that help their businesses.
   
3. Community members get software that is continuously supported and evolving rapidly to add features they need. They get assurance that valuable patches will be integrated. They are motivated to use, test, and develop patches for the software as that further increases its value to them and leads to recognition as authorities on the project.

The Open Source Hackers Cooperative can be structured to create a number of virtuous feedback loops that will support and extend the software over a long period of time. The Hackers Co-op model could be standardized and even backed up with software as well as cookie-cutter legal documents so that it becomes very simple to set up and manage.

I don't know if there are projects already following this model. Selena Deckelman already used the term "Hacker's Cooperative" though from a somewhat different perspective. If you know of anyone who has worked through the cooperative economics fully I would appreciate hearing from you. Meanwhile, this is not a theoretical question for at least a couple of reasons.

First, I work for a for-profit company that is willing to sponsor projects in exactly the way described in this article. We are looking for (a) control, (b) stability, and (c) a development model that is cheaper than doing it ourselves. If such cooperatives existed we would be interested in them. I'm sure we are not alone, because this is how all for-profit businesses tend to think. The cooperative is a viable model because of the way it reinforces and maximizes member interests.

Second, the Hacker Co-op leads to some interesting questions about how to design software that promotes the leveraging effect of an open source community. It is possible to design even application software so the your users don't all have to be coders in order to contribute. If you want to create something really big out of your open source project (or just have a nice job) it helps to think these issues through before you write much code. Maybe it's the first thing on the list after you set up your new project.