supperware

Blog

  • Standing on the shoulders of Linus

    As you’re possibly reading this because you’re bored of the Yuletide festivities and the wall-to-wall children’s TV, I’m taking advantage of your compromised boredom threshold, and tackling a subject that will take me about twice as many words as usual.

    It’s Christmas, and Seaboard hasn’t shipped yet. Somewhat happily, the reason behind this isn’t our fault: a supplier made a catastrophic error of judgement that has cost us a month, and these things happen. Fortunately, it won’t kill us. In fact, it gives us time to do a bit more testing. But the road to delivery is seldom smooth, and the more you’re innovating, the more violently it undulates.

    Experience improves foresight. What ROLI has ended up doing is what, upon reflection, we might have done all along. The Seaboard is currently a brilliant controller with a small ecosystem of third-party software synths that play really nicely with it, and that’s what our first customers will get. What we’ve had to postpone is our own synthesiser. Seaboard is supposed to make its own sounds: just switch it on, plug in headphones, and perform. Now we’ll be providing that later, as a free upgrade.

    Our synthesiser runs (users: ‘will run’) on its own microprocessor inside Seaboard. There are another couple of chips that take care of getting the playing surface and Sound Dial to work, so making sounds is pretty much all that this microprocessor has to do.

    Now, remember that we’re a startup, and what startups do best is schmoozing like mad and building a community, seeding goodwill, and making people stop in their tracks to smile and gaze in wonder at the smart execution of a brilliant idea. What we can’t do is pretend we’re Microsoft (or even Native Instruments). We’re terrible at working on several fronts at once, and even our talented brigade of red-eyed, caffeinated programmers cannot successfully produce a minimum viable product that is neither minimal nor technically viable.

    This article isn’t going to be a reflection on what’s pushed us into this manoeuvre so late. That will come when things aren’t so sensitive. What it will describe is one of the larger and more interesting bumps on our road to delivery. ROLI used a lot of open source software in order to speed up the design of the synthesiser. This got us started quickly, enabling non-specialists to make huge contributions, but it turned out to be a mixed blessing in the long run. Perhaps using free software didn’t save us time or money after all.

    Open source software is software that is written by somebody you didn’t pay, that is distributed for free under fairly generous conditions. There are many good reasons why people write open source software. Books have been written on this subject, so it’s beyond the scope of this article.

    The two main lumps of open source software we used are Linux and JUCE. Linux is the world’s largest open source project: a complete operating system into which volunteers have poured cumulative lifetimes of work so that it can do pretty much anything. To many coders, it is a work of unsurpassable beauty. Its progenitor, Linus Torvalds, has not made a penny from selling it, but does very nicely out of lecture tours.

    JUCE is a series of utilities that provide graphics, audio, and control capabilities in such a way that a program written in C++ can simultaneously run on Linux, Mac OS, or Microsoft Windows. Jules Storer put ten years of work into this and it’s remarkable. He makes money from it when people commercialise projects that contain JUCE.

    Let’s consider JUCE first, as it’s simpler. Using JUCE, our synthesiser can run in a window on Mac OS, on Linux, or on Microsoft Windows, and also inside Seaboard. Four separate products are therefore one lump of code: we fix a bug or add a feature on one, and it’s fixed everywhere. Here’s the catch, though: managing a cross-platform project (as it’s called) is easier than managing four separate ones but, even with JUCE, a great deal harder than managing code for one platform. There are a lot of reasons for this:

    1. It remains possible to write code that is fine on one system but stops with errors on another. In practice, every time the code changes, every platform has to be retested.
    2. The chip we’re compiling onto inside the Seaboard is very, very different from the Intel Core chips inside Macs and PCs. It’s also our most important target. Every time we change code, the embedded build (the code that runs on Seaboard) requires special attention.
    3. There are no graphics on Seaboard, so we’ve had to do some work to make sure that no vestige of the graphical stuff finds its way there.
    4. Every time our files are changed or reorganised, each computer’s compiler instructions need to change too. JUCE provides a way of automating this, but we still need to hack it manually.

    The second reason has given us the greatest headache. We’ve had to rewrite large tranches of the synthesiser to get it to perform acceptably as a standalone musical instrument, and have needed to have some external expertise to help us do this. We’ve also needed to automate builds for Seaboard. Our engineers work and audition their code on Macs. To allow them to keep working, an external computer automatically takes the code they’ve written, builds it for Seaboard, runs some elementary tests (does it crash?) and checks that these tests pass. The programmer can check that it still compiles, and download the latest build to test on a Seaboard once they’re ready. Time taken to get this build system working: about a week. Open-source software helped us to get it going, or it might have taken a month.

    Jules has kindly made a no-graphics version of JUCE, and we’ve helped him in turn by putting considerable work into optimising JUCE for our little Cortex-A8 chip. This may seem like a small detail, but there were a lot of people waiting for it: Seaboard and Raspberry Pi share the Cortex-A8 as a microprocessor, so we’ve also optimised for the world’s best-selling project board. ROLI also opened the books on the special control protocol that Seaboard uses to communicate with its own synthesiser. Why MIDI is no good for this is a posting in itself.

    That’s JUCE; now Linux. Linux contains millions of lines of code, comprising things such as the Unix command line and ‘make’ that are their own little environments; the bootloader that starts Linux requires learning yet another language. The kernel has its own tools to build correctly, once the compiler has been set up properly: it’s compilers within compilers within compilers.

    Linux has been under development for over twenty years. That’s an unusually long journey in software terms, and consequently it takes hours and hours to gain proficiency in just using it, let alone extending it. Very few people are experts in many parts of Linux, but the advantages of using it are that it’s been widely tested, has ways of adding any feature you can imagine, and is comparatively very quick to get started. These are very big advantages. Writing an OS from scratch is hard, and the whole point of Linux is that nobody should ever have to again. The problems with Linux, though, are the inconsistency of documentation and the lack of an easy way in. A build of Linux that works on our little chip isn’t the same as the thing that runs on a PC. To get it working, even with people who knew what they were doing, took days. When we wanted to make the chip work with raw data inside the Seaboard, we had to write a driver that required knowledge of the innermost workings of Linux and the chip itself: a very arcane combination of skills that took a while to find.

    So, for the first time in ages, I’m attempting to collaborate on something at which I am a novice, and have little chance of substantially improving my knowledge on my own. I’ve had to accept my ignorance and trust my team: those who built this structure know, and will always know, more than I do. Meanwhile, we’ve paid host to a horde of consultants specialising in:

    • Setting up the right flavour of Linux with the right features;
    • Getting the Linux kernel compiling on our embedded system;
    • Getting JUCE compiling on our embedded system;
    • Writing drivers to massage our raw incoming data and outgoing audio;
    • Making the USB output work reliably;
    • Optimising the C++ compiler, and then the synthesiser code, for the Cortex-A8.

    This has all been in addition to Jules’s work for us. Most of the rest we’ve done ourselves. So, a considerable team of consultants has been hired, for a few days at a time. Many of these clever and expensive people have been needed just to help us with ostensibly ‘free’ software, and many were delayed or floored by having to grasp the complexity of our cross-platform code base.

    What do we have to show for it? We have a reasonably good software synthesiser that runs on four platforms. However, it’s questionable whether this has contributed any more to the appeal of Seaboard than would writing two separate and far smaller programs: a dedicated synthesiser for the Cortex-A8 only, and a MIDI remote control in JUCE that would tell it what to do. There may yet be long-term strategic value in our cross-platform endeavour, but we cannot realise it until after we’ve got the work to an acceptable quality, and shipped it inside a Seaboard.

    Linux running on Seaboard makes certain things (such as debugging and managing the on-board sample library) easier for our engineers. Currently, though, it provides no advantage to the user, and actually means that Seaboard takes longer to start up than it otherwise would.

    In fact, we might have avoided the cross-platform chimera altogether by not running an operating system at all. Most stand-alone synthesisers don’t run one: the only benefits they bring are better debugging and a decent file system. But in paring down, we would have sacrificed or delayed the Mac and PC builds of our synthesiser, and made it harder for our programmers to test and iterate their work. This might have created other problems.

    Lessons so far, only some of which I already knew:

    1. The best software is no software. The fewer things your first product does, the better it will do them.
    2. Remember Hofstadter’s Law: ‘It always takes longer than you think, even when you take into account Hofstadter’s Law.’ Engineers are optimists: they must be or they couldn’t turn up to work. There are always distractions, extra tools to build, bottlenecks and impasses in a project that cannot be fully anticipated. If a project is fundamentally innovative, multiply engineers’ resourcing estimates by about 4; otherwise, it depends on the engineer, but the average is about 2.
    3. Basing your technology on free software is not a panacaea, but a technically subtle decision. It can end up costing more, or causing people to overpromise. The ramifications of a decision either way are potentially ruinous. Research licensing costs. Get the most experienced practitioner you can afford to inform the final decision based on expected implementation time and technical difficulty. Ignore all commercial pressures and trust their advice, or there’s no point in paying for it.
    4. Use Agile, because it is designed to get the simplest software out as quickly as possible.
    5. Employ a product manager to prevent stakeholders, be they testers, product specialists, or the CEO, from adding features once work has started and a milestone has been agreed.

  • A talk at the University of Surrey

    Every couple of years, the University of Surrey honours me by inviting me back to talk to students for an hour about what I do for a living. I’m made to feel so welcome that I barely know where to put myself, and I will be presenting my fourth such talk on Tuesday afternoon.

    As I get older, and my career changes, my ability to connect where I am today to where I started slackens, as does any connection between the undergraduate course I was taught and what today’s students must learn. Many of this year’s undergraduates were born the same year that I first strolled onto campus.

    Although it doesn’t feel like it, I’m conscious of the fact that I’m now of a different generation, and my talk plan is becoming more reflective and somewhat generalised.

    In case they help, here are echoes from previous talks I’ve given, that I wrote for Focusrite’s blog.

    Finally, because he’s the only acoustics lecturer I’ve ever heard of who was also a self-made billionaire, I feel obliged to post the late Dr. Amar Bose here. In the last six minutes (from 54:10), he sums up everything I would want to tell a University student about starting their career.

  • Reality hits (part 2)

    One good, if fairly glib, way of summarising the purpose of a manager is that he or she is there to prevent the next crisis. This involves all the ingenuity and wisdom that human experience can bestow. Management is anticipation, organisation, and diplomacy; everything else follows.

    When a crisis does occur, though, overcoming it is surprisingly easy (assuming that it’s possible at all). There’s none of the effort of anticipation, because the next big problem is already happening around you. Everything you need to know is available, and the number of exits is strictly limited. As if that weren’t enough, most of the circuitry of the human brain is wired for reactive thought: people are at their resourceful, energetic best in times of difficulty.

    So, most fools can govern in a crisis (unless it’s a political crisis); the true hallmark of skill is preventing one from happening in the first place.

    It’s now two or three weeks since the events of the last article unfolded, placing me squarely in the ‘most fools’ category. To recap, we were left with about three working days before a major tour of the US, gazing at a supremely broken keyboard and facing a very irritated CEO. What we needed to do was obvious:

    • Stop adding features and refactoring, except to fix bugs.
    • Tackle problems in the order of greatest-impact-first rather than easiest-first.
    • Decide every day which bugs we were going to fix, circulate this list, and include the CEO in the circulation.
    • Honestly demonstrate steady improvement to restore the CEO’s confidence in the team.
    • Keep testing our work by buttonholing an independent product specialist every evening, to get a user’s perspective and to reveal new bugs.

    And, of course,

    • To work really crazy hours until we’d won.

    If you’re part of a software team and you haven’t been chained to a radiator for the last two decades, you’ll recognise a fascinating convergence. That list, without the addendum, can be translated into software-speak as:

    • Prioritising bugs over features.
    • Managing the expectations of important people outside the team;
    • Holding daily stand-up meetings to review progress and triage new cases;
    • Employing regular, independent, integrated testing.

    These are cornerstones of software best practice and, more importantly, of Agile project management. So it seems that Agile fits our problem!

    Agile almost certainly had its genesis in the reaction to a crisis such as ours, but it goes much further. The textbooks I have read on the subject recommend just foisting a flavour of Agile onto your team in one huge, indigestible thrust. I hate doing this to any fellow human being, so I am happy to discover that it is possible to creep towards new working practices effectively, little by little, minimising disruption while sharing complicity and control. When done properly, Agile should be liberating and democratic, and there’s no reason why the transition itself should resemble an abduction.

    Naturally, it helps to have a crisis, and our predicament provided the necessary catalyst for change. The great surprise to me, as it had been at Focusrite, was how hungry everybody is for more formality, and for better delineation of working rules.

    The founders of Agile must have invented the art in a piecemeal way too, without access to flashy collaborative software, burndown charts, or the lexicon of trademarked, capitalised, frat-boy words that Agile practitioners seem to love. The weird portmanteau words like after-party and touchdown are alien to British nerds; use of sports metaphors like sprint and scrum spark memories of weekly abuse on playing fields. While those sweaty signifiers are everywhere, it’s an oft-demonstrated fallacy that everybody must imitate the winners or die. Google, Apple, and Oracle look very different, and they didn’t get there overnight. Better to evolve, to question, to cut one’s own clothes, and to pass each milestone at a sustainable pace.

    The beginning was messy, as it had to be: the day’s top three bugs (along with the next three, in case we had a good day) were scrawled on a whiteboard and marked with asterisks, and crossed out with a flourish whenever a small victory was scored. Reports and updates were circulated two or three times a day via emails and ephemeral, free-form documents. Our testing was fairly ad-hoc, usually starting some minutes late and skipping from feature to feature with everybody talking at once, but it actually did the job and we ended up with a really decent, demonstrable product that supplanted our old prototype in a matter of days, and a happy boss who trusted us again. I characteristically resist enthusiasm, but a few minutes of playing a Seaboard now makes an electronic piano feel primitive.

    The last few weeks have left a lingering taste, and some time to tidy up and consolidate our new system. It’s no shame to admit that a talented software team has occasional organisational difficulties. Everybody does; it’s a fact of life. Mistakes make great lessons, though. Biographies that chronicle noble failures are far more informative and more honest, if lamentably rarer, than those that focus only on success. Creative teams are always caught between between having enough formality to collaborate effectively on really difficult tasks, and respecting those sparks of individual brilliance that will make a product shine. The equilibrium between the two must adapt continually — often painfully — to fit the evolving team, but that’s what Agile is supposedly all about.

    Our jobs are constantly in flux. Focusrite, six months after I left, has put in so many new practices that, were I to return, I wouldn’t know where to put myself. And yet, Agile is a peculiar beast to shoehorn into a hardware company, as it doesn’t fit well with the process of making physical objects.

    The next article will probably be about how software-centric Agile collides with hardware-centric ROLI. Particularly, we’ve been thinking about how we can make the necessary heaps of planning and due diligence work in our favour, by softening the boundary between our work and its documentation.

    Actually, that’s not an article. It’s a career.