As you’re possibly reading this because you’re bored of the Yuletide festivities and the wall-to-wall children’s TV, I’m taking advantage of your compromised boredom threshold, and tackling a subject that will take me about twice as many words as usual.
It’s Christmas, and Seaboard hasn’t shipped yet. Somewhat happily, the reason behind this isn’t our fault: a supplier made a catastrophic error of judgement that has cost us a month, and these things happen. Fortunately, it won’t kill us. In fact, it gives us time to do a bit more testing. But the road to delivery is seldom smooth, and the more you’re innovating, the more violently it undulates.
Experience improves foresight. What ROLI has ended up doing is what, upon reflection, we might have done all along. The Seaboard is currently a brilliant controller with a small ecosystem of third-party software synths that play really nicely with it, and that’s what our first customers will get. What we’ve had to postpone is our own synthesiser. Seaboard is supposed to make its own sounds: just switch it on, plug in headphones, and perform. Now we’ll be providing that later, as a free upgrade.
Our synthesiser runs (users: ‘will run’) on its own microprocessor inside Seaboard. There are another couple of chips that take care of getting the playing surface and Sound Dial to work, so making sounds is pretty much all that this microprocessor has to do.
Now, remember that we’re a startup, and what startups do best is schmoozing like mad and building a community, seeding goodwill, and making people stop in their tracks to smile and gaze in wonder at the smart execution of a brilliant idea. What we can’t do is pretend we’re Microsoft (or even Native Instruments). We’re terrible at working on several fronts at once, and even our talented brigade of red-eyed, caffeinated programmers cannot successfully produce a minimum viable product that is neither minimal nor technically viable.
This article isn’t going to be a reflection on what’s pushed us into this manoeuvre so late. That will come when things aren’t so sensitive. What it will describe is one of the larger and more interesting bumps on our road to delivery. ROLI used a lot of open source software in order to speed up the design of the synthesiser. This got us started quickly, enabling non-specialists to make huge contributions, but it turned out to be a mixed blessing in the long run. Perhaps using free software didn’t save us time or money after all.
Open source software is software that is written by somebody you didn’t pay, that is distributed for free under fairly generous conditions. There are many good reasons why people write open source software. Books have been written on this subject, so it’s beyond the scope of this article.
The two main lumps of open source software we used are Linux and JUCE. Linux is the world’s largest open source project: a complete operating system into which volunteers have poured cumulative lifetimes of work so that it can do pretty much anything. To many coders, it is a work of unsurpassable beauty. Its progenitor, Linus Torvalds, has not made a penny from selling it, but does very nicely out of lecture tours.
JUCE is a series of utilities that provide graphics, audio, and control capabilities in such a way that a program written in C++ can simultaneously run on Linux, Mac OS, or Microsoft Windows. Jules Storer put ten years of work into this and it’s remarkable. He makes money from it when people commercialise projects that contain JUCE.
Let’s consider JUCE first, as it’s simpler. Using JUCE, our synthesiser can run in a window on Mac OS, on Linux, or on Microsoft Windows, and also inside Seaboard. Four separate products are therefore one lump of code: we fix a bug or add a feature on one, and it’s fixed everywhere. Here’s the catch, though: managing a cross-platform project (as it’s called) is easier than managing four separate ones but, even with JUCE, a great deal harder than managing code for one platform. There are a lot of reasons for this:
- It remains possible to write code that is fine on one system but stops with errors on another. In practice, every time the code changes, every platform has to be retested.
- The chip we’re compiling onto inside the Seaboard is very, very different from the Intel Core chips inside Macs and PCs. It’s also our most important target. Every time we change code, the embedded build (the code that runs on Seaboard) requires special attention.
- There are no graphics on Seaboard, so we’ve had to do some work to make sure that no vestige of the graphical stuff finds its way there.
- Every time our files are changed or reorganised, each computer’s compiler instructions need to change too. JUCE provides a way of automating this, but we still need to hack it manually.
The second reason has given us the greatest headache. We’ve had to rewrite large tranches of the synthesiser to get it to perform acceptably as a standalone musical instrument, and have needed to have some external expertise to help us do this. We’ve also needed to automate builds for Seaboard. Our engineers work and audition their code on Macs. To allow them to keep working, an external computer automatically takes the code they’ve written, builds it for Seaboard, runs some elementary tests (does it crash?) and checks that these tests pass. The programmer can check that it still compiles, and download the latest build to test on a Seaboard once they’re ready. Time taken to get this build system working: about a week. Open-source software helped us to get it going, or it might have taken a month.
Jules has kindly made a no-graphics version of JUCE, and we’ve helped him in turn by putting considerable work into optimising JUCE for our little Cortex-A8 chip. This may seem like a small detail, but there were a lot of people waiting for it: Seaboard and Raspberry Pi share the Cortex-A8 as a microprocessor, so we’ve also optimised for the world’s best-selling project board. ROLI also opened the books on the special control protocol that Seaboard uses to communicate with its own synthesiser. Why MIDI is no good for this is a posting in itself.
That’s JUCE; now Linux. Linux contains millions of lines of code, comprising things such as the Unix command line and ‘make’ that are their own little environments; the bootloader that starts Linux requires learning yet another language. The kernel has its own tools to build correctly, once the compiler has been set up properly: it’s compilers within compilers within compilers.
Linux has been under development for over twenty years. That’s an unusually long journey in software terms, and consequently it takes hours and hours to gain proficiency in just using it, let alone extending it. Very few people are experts in many parts of Linux, but the advantages of using it are that it’s been widely tested, has ways of adding any feature you can imagine, and is comparatively very quick to get started. These are very big advantages. Writing an OS from scratch is hard, and the whole point of Linux is that nobody should ever have to again. The problems with Linux, though, are the inconsistency of documentation and the lack of an easy way in. A build of Linux that works on our little chip isn’t the same as the thing that runs on a PC. To get it working, even with people who knew what they were doing, took days. When we wanted to make the chip work with raw data inside the Seaboard, we had to write a driver that required knowledge of the innermost workings of Linux and the chip itself: a very arcane combination of skills that took a while to find.
So, for the first time in ages, I’m attempting to collaborate on something at which I am a novice, and have little chance of substantially improving my knowledge on my own. I’ve had to accept my ignorance and trust my team: those who built this structure know, and will always know, more than I do. Meanwhile, we’ve paid host to a horde of consultants specialising in:
- Setting up the right flavour of Linux with the right features;
- Getting the Linux kernel compiling on our embedded system;
- Getting JUCE compiling on our embedded system;
- Writing drivers to massage our raw incoming data and outgoing audio;
- Making the USB output work reliably;
- Optimising the C++ compiler, and then the synthesiser code, for the Cortex-A8.
This has all been in addition to Jules’s work for us. Most of the rest we’ve done ourselves. So, a considerable team of consultants has been hired, for a few days at a time. Many of these clever and expensive people have been needed just to help us with ostensibly ‘free’ software, and many were delayed or floored by having to grasp the complexity of our cross-platform code base.
What do we have to show for it? We have a reasonably good software synthesiser that runs on four platforms. However, it’s questionable whether this has contributed any more to the appeal of Seaboard than would writing two separate and far smaller programs: a dedicated synthesiser for the Cortex-A8 only, and a MIDI remote control in JUCE that would tell it what to do. There may yet be long-term strategic value in our cross-platform endeavour, but we cannot realise it until after we’ve got the work to an acceptable quality, and shipped it inside a Seaboard.
Linux running on Seaboard makes certain things (such as debugging and managing the on-board sample library) easier for our engineers. Currently, though, it provides no advantage to the user, and actually means that Seaboard takes longer to start up than it otherwise would.
In fact, we might have avoided the cross-platform chimera altogether by not running an operating system at all. Most stand-alone synthesisers don’t run one: the only benefits they bring are better debugging and a decent file system. But in paring down, we would have sacrificed or delayed the Mac and PC builds of our synthesiser, and made it harder for our programmers to test and iterate their work. This might have created other problems.
Lessons so far, only some of which I already knew:
- The best software is no software. The fewer things your first product does, the better it will do them.
- Remember Hofstadter’s Law: ‘It always takes longer than you think, even when you take into account Hofstadter’s Law.’ Engineers are optimists: they must be or they couldn’t turn up to work. There are always distractions, extra tools to build, bottlenecks and impasses in a project that cannot be fully anticipated. If a project is fundamentally innovative, multiply engineers’ resourcing estimates by about 4; otherwise, it depends on the engineer, but the average is about 2.
- Basing your technology on free software is not a panacaea, but a technically subtle decision. It can end up costing more, or causing people to overpromise. The ramifications of a decision either way are potentially ruinous. Research licensing costs. Get the most experienced practitioner you can afford to inform the final decision based on expected implementation time and technical difficulty. Ignore all commercial pressures and trust their advice, or there’s no point in paying for it.
- Use Agile, because it is designed to get the simplest software out as quickly as possible.
- Employ a product manager to prevent stakeholders, be they testers, product specialists, or the CEO, from adding features once work has started and a milestone has been agreed.