Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Programming As An Expressive Instrument (with Sam Aaron)
2.8k Plays15 days ago
Sam Aaron is the creator of Sonic Pi, one of the most unusual software platforms you’ll encounter. It’s a live-coding playground for making music. A tool that lets you write code that defines sounds and musical phrases, and build up a hole program that plays anything from a short bleep to a whole nightclub set. And Sam’s creator has been using it live for years, weaving drum & bass nights out of thin air, all driven by the Ruby-esque he writes.
In this episode we go through Sam’s career path and design journey as we look at what it takes to make a programming language with enough expressivity and productivity to produce music at the speed of Sam’s imagination.
--
Sam’s Sonic Pi Course: https://www.patreon.com/posts/new-introductory-115404746
Sonic Pi: https://sonic-pi.net/
SuperCollider: https://supercollider.github.io/
Overtone: https://github.com/overtone/overtone
Power Gloves: https://en.wikipedia.org/wiki/Power_Glove
Web Audio API: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
Tau5: https://www.patreon.com/posts/announcing-sonic-112605951
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Recommended
Transcript
High-pressure software deadlines
00:00:00
Speaker
Have you ever written software to a deadline? If you're a working programmer, almost certainly. We need this software by the end of the quarter, by the end of the sprint, by the end of the day. The worst deadline I ever had was, you're on call, it's 3am, and this has to be working by the time people start coming into the office.
00:00:20
Speaker
That was a very bad night that I'd rather forget. But just for this week, I think we all have to concede that we don't really know what a deadline is.
Introducing Sam Aaron: Live coding DJ
00:00:29
Speaker
My guest this week is Sam Aaron, and he measures his software deadlines in seconds.
00:00:35
Speaker
Or more accurately, in bars of music, because he's a live coding performance DJ. He gets up on stage with his laptop, with a programming environment he created, and he writes some code that describes the music he's currently hearing in his head.
00:00:52
Speaker
and he ships it into production. The nightclub floor where people are dancing and he does that again and again the whole night long. I've been to a few of his gigs and he has a cracking act and you wouldn't need to know that it's all driven by a programmer who's designing code, implementing it and shipping to prod every four bars. How on earth do you learn to do that?
00:01:17
Speaker
I mean, what kind of programming language would you design to ship with that kind of velocity and that kind of fluidity? And what kind of language would you design with sufficient expressive power to make whatever music you're hearing in your head? And what takes your career from being a computer science researcher at Cambridge to being the coder that's playing a banging set in a nightclub in Berlin?
Career journey: Researcher to live coding DJ
00:01:43
Speaker
Let's find out. I'm your host, Chris Jenkins. This is Developer Voices, and today's voice is Sam Aaron.
00:02:01
Speaker
Joining me today is Sam Aaron. Sam, I am delighted to welcome you to the show. Hi, it's really lovely to be here. it's I've been trying to organise this since when I first started Developer Voices, I drew up a list of people I want on the show and you were on the original list. Super kind. It's taken you too long to get your house built and get over here. Yes, it's shockingly too long. yes Yeah, probably not your first priority, and how long is it taking me to get my house built? But, okay, we need to get into this. So, you I first encountered you um at a closure conference more than 10 years ago, yeah when you were a researcher at Cambridge.
00:02:43
Speaker
computer science researcher at Cambridge, which is very much not what your career is at the moment. so How did you get from where you were to where you are today? It's a very good question. Being a researcher at Cambridge was also part of the journey as well. I mean i took the journey of doing an undergraduate degree at university, then a PhD, then I was an industry programmer for a couple of years, three years or so in Amsterdam, and then I took a year out as a sabbatical.
00:03:10
Speaker
because I never really had this so idea of a year of out. Lots of other people had done it before they went to the university degrees. They sort of went traveling around the world and I never did that. So I thought it'd be really nice to spend a year out and work on my weird ideas of of of musical programming languages. So at the time I was working in Clojure, working on a system called Overtone with my friend Jeff Rose, who I met in Amsterdam whilst I was living there.
00:03:33
Speaker
okay And then my wife got a scholarship to do a PhD in Cambridge, which is like one of those amazing things you like you run straightforward towards, you know, and don't give up. And so we moved to Cambridge, and I thought it'd be a really nice idea to spend a year in this sort of year out.
00:03:50
Speaker
ah trying to find a way to to explore these ideas about programming music and closure.
Collaboration with Alan Blackwell
00:03:57
Speaker
Then after that year, then go and find a normal job and go back to being a programmer again. It'd be quite a fun thing to do. and Somehow, I ended up having a ah desk at the computer labs at Cambridge, just through the conversations I was having. and I met a guy in the music department who I asked, maybe I could do a Masters of Music. He was like, oh no You don't have enough musical background." It was totally fine. We said, but I do know somebody in the computer labs who actually might be interested in the work you're doing. and When I discussed it, there was this chap called Alan Blackwell. He was actually really, really excited interesting and interested. He said, look, it there's a spare desk there. You can have that desk and and and then let's chat and let's let's do some writing together and work on stuff together. and That then turned out but to me then having spending eight years actually at the computer labs.
00:04:41
Speaker
It kind of just fell into place. It was super weird. When you say, let's do some writing together, what kind of writing? Were you talking about academic writing or writing music? Yeah, papers. The work I was doing was very much in line with the work he was doing. He was very interested in psychology programming languages. I was looking at programming languages in terms of performance and as an instrument, as a means of expression.
00:05:07
Speaker
um He had done work in the so-called cognitive dimensions of programming, so the idea of exploring programming languages in terms of to different dimensions like viscosity. how How hard is it to think about something? is it Does it flow through your mind quickly, or is it treacle? He was trying to explore programming languages and in different kinds of ways that we normally think about them. and Normally, we talk about, ah is it completely accurate? Is it provable? Is it speed? Is it fast? How much memory does it consume? These are sort of classical engineering approaches to exploring perceiving programming languages. and He was taking much more of a psychological approach. and so there was There was definitely an overlap and an interest. that's That instantly makes me think of like live coding. right Viscosity
00:05:54
Speaker
a brand new idea in the programming
Musical beginnings and educational challenges
00:05:56
Speaker
world for me. But yeah, when you're live coding, you want something to flow very quickly. 100%. When I was talking about these ideas, he was like they were obvious to him, and he'd been working quite closely with other life coders before. so This was very very very natural for him, but obviously for me, these were new thoughts I was having myself independently. and so It was really wonderful to actually meet others who had been thinking about these kind of ideas for quite a while, and to learn from them, and and to collaborate. It was brilliant. but That's what the academic world is supposed to be, right?
00:06:25
Speaker
it was fuck yeah Can I just ask, how much which how much of a musical background did you have? That's a very good question. i mean It depends how you define it. um and Certainly not on any kind of like academic level.
00:06:37
Speaker
mean i did study um play I played the clarinet in a wind band, in the county wind band. I played in the the school jazz band Saxophone. um So, in the school level, i was i was very i was I was active as a musician, but when I got to university, um i I stopped because I didn't have... I wasn't as good as everybody else. It seemed like all the bands that were forming and all the orchestras that were forming at university were all the top level ah musicians. And there was nothing really for sort of rubbishy, fun, people having a good time kind of musicians that I could find. um It's a bit like with the the sports are the same. I quite liked playing football a bit, but I'm rubbish at it.
00:07:16
Speaker
so I was struggling to find people who were also rubbish, who just wanted to kick around a bit. Everyone was really keen, so I stuck to things I was keen on, which was programming, and kind of pushed aside the music. But it came back just because you thought you were doing a sabbatical.
00:07:31
Speaker
Oh, no. so i mean I've always been curious about music. I've been a very active listener. I think that's a really important part of of music itself. and Then I started exploring the idea of what it would it would it be like. I was at a conference um in Denmark, so long even well before the Closure Conference we met.
00:07:50
Speaker
And i yeah they had an after party and ah they just basically got a bunch of programmers who could who could play instruments and said, let's form a band and you can
Programming music: A historical perspective
00:07:59
Speaker
be the entertainment. And it was kind of a fun thing to do, but it was like, it could be a bit better. I was thinking like, what what would be the better approach? And obviously in my head being strange, it's like, well, surely we're at a programming language conference, surely we should be able to program the the the the music, the entertainment. and so That was my first thought about that. and Then, of course, as soon as you start exploring ideas, you'll realise that everyone's done this for hundreds of years already. so Certainly with with programming and music, mean computers and music were a combination before computers even existed. so
00:08:31
Speaker
um Yeah, so that was really interesting to then explore all the people who are doing this at the time and in the past and seeing whether but their journeys and looking at different systems. And that got me to discover Super Collider, which is still pretty much the predominant ah open source synthesis engine used in in pretty much all the live coding systems that aren't web-based. So define that for me, just and for everyone else. like what it What's it actually doing, and where' what's it not doing? Where do you come in on the boundaries? Yes, Superclider. It's ah an amazing piece of technology. and James McCartney wrote it about 30 years ago, so it's actually it's it's got a quite a heritage. and The idea was, i mean up to that point that he wrote Superclider, most of the music systems were kind of like a render-based system. It's a bit like Toy Story. where
00:09:21
Speaker
you might have a wireframe image of all the characters, and then you would send to that to a render farm, and then a week later you'd get your scene and with all the lighting all put in. Music was kind of like that. that The computers weren't fast enough to be able to render the music in real time, so you had a lot of systems that allowed you to compose, to design the synthesizers, and then press the render button, and then some period later, depending on the speed of the machine, obviously obviously the complexity of the system of the audio system you're working with, and obviously of the of the language that was was generating that, yeah you would get some audio far back that you could listen to and go, oh, that's not what I wanted, and around the cycle you'd go again. So, Superglide was pretty much, I think, the first system to focus purely on what can we do in real time. And so the design was all about that. So, I have to check you on that, because i I remember going back about 30 years ago, surely we had home consoles that were playing music live.
00:10:19
Speaker
Yes. i mean we so so I had a spectrum and you could run as a basic programming language. What's the difference here? It's a good question. so I think the main difference is that those something like the Commodore 64 had a specific chip in it, which was a synthesizer. You would be sending timing signals to that synthesizer ah to trigger it. so um and i mean that that The SID chip was so famous that they they pulled it out and they made synthesizers out of it since um and With Super Collider, you've got both that, you've got the thing which can generate the timing signals to to trigger the synthesizer, but also the synthesizer itself, the synthesis, is all done in real time ah using a little C functions. so The sine waves and the and the low-pass filters and the reverb filters and all of the different components that generate the sound, we're all running
Creating Sonic Pi for education
00:11:07
Speaker
in real time.
00:11:08
Speaker
um and so that That had put on quite a lot of constraints on what kind of complexity of audio could you create. It was much more limited than what other systems could do, which allowed rendering, um but also had this interesting property of you can do it all life.
00:11:22
Speaker
And are you saying this was... I'm now trying to figure out why you felt the need to build something on top of that. What was the interface to Super Collider? And were you not happy with it? Yeah, so the Super Collider had originally was a programming language and audio synthesis engine rolled into one. and That was version one and two. And then version three, James separated them into two separate systems, ah the language runtime and the synthesis engine.
00:11:47
Speaker
And the idea was that he developed this language, which was like it's like ah his own little language, it was like small talky, Ruby-like language. And um he wanted also to allow others to explore what it would be like to write music systems in other languages. And so by putting a hard division between the language runtime, the synthesis engine, and making an API, network API, um ah to talk to the between the two, you could then essentially plug in another programming language and and drive this thing in the same way you could drive a web service, say.
00:12:17
Speaker
um and so I, at the time, had been experimenting quite a bunch with Ruby, um so za just before I discovered Clojure, and had been writing some systems. and I think probably A, because my programming skills weren't that great, and B, because I discovered that certainly the kind of programming systems I wanted to write ah involved a high degree of concurrency, but certainly the two of those combined to make the kind of programs I was writing really not very good, not very efficient and and not work particularly well.
00:12:49
Speaker
and so When I discovered that, Supercollider was essentially a Ruby-like language. I was like, well maybe i'll I want to try something else. and At that point, I discovered Rich Hickey and Closure. The idea of of concurrency was something something that blew my mind. It was a beautiful concept. At the time of my life, I was like, this is this is perfect. This is like a really neat way, clean way to describe the concurrency problems that I was trying to to deal with and solve at the time for my my early sort of Ruby systems. Also, the that ah flexibility, the things that attracted me to Ruby was the fact that it's such a malleable a programming language that you can
00:13:28
Speaker
you can treat like clay and you can morph it into the kind of system that's close to the way you think about it. so Go back to that viscosity thing. You can make it low viscosity in terms of the programming language and your mind are at the same level of abstraction if you do the right things. and so LISPs and Clojure as a Lisp has that property too. and so that that That excited me. It had the concurrency and the malleability. I'm like, yes, this is this is good. I wonder if I can combine Clojure and Super Collider And then I discovered that one of my friends said that it's a guy that just started this project just last month, and that was Jeff Rose. And he also happened to live in Amsterdam, so we made instant friends and we hacked on the thing together. so yeah So it was more like I wanted to move away from Ruby at the time and Supercollider felt like Ruby, the programming language. And so I wanted to explore Clojure and functional lisps, particularly.
00:14:17
Speaker
And are we talking about concurrency as being important because you're trying to do like you're trying to write a program that is a drummer at the same time as it's a bass player? Precisely, yeah. I mean, that's the right that's a way to think about it, like a band. It's way more natural to think of a band of individual instrumental instrumentalists than it is to think about the single person attempting to play all instruments concurrently. and That's the joke. is The one-person band is like a comedy act. It's not a real sensible approach.
00:14:47
Speaker
Have you just given a damning review of your own act?
00:14:53
Speaker
I'm not the one person band trying to play all the instruments at the same time. I'm more like a plate spinner. I've got multiple plates spinning at the same time, and I'm wobbling them and keeping them going. The music's not generated by me, it's generated by the plate spinning. I'm just making sure they're keeping going.
00:15:10
Speaker
okay we've so We've got to get into the design of how you get from that research enclosure to being a live spinner of musical plates. What happened was with the university is that, um as always with the case, that contracts are hard to find and funding is always hard to come by. and There was a point where that was coming to a bit of a struggle. so I was like, I'm going to go back to being an industry programmer again. and That's the same year that the Raspberry Pi ah span out of the labs.
00:15:38
Speaker
And they were looking, at the foundation particularly, were looking for some software to help children learn how to code. And um yeah because my contract was ending, and I'd worked on this weird music stuff, I kind of put my hand up and said, well, I could, because it was only for three months this this project, um I could hang around for another three months and see if I could build a minified version of of Overtone to run specifically on the Raspberry Pi. And maybe we could use that to teach kids how to code. And they're like, cool. So that's, and then that's that that extended my period because that that was wildly successful. And they kept funding that on and on and on for a few more years.
00:16:14
Speaker
That's instantly going to seem tricky to me because you've got you've got Super Collider, which sounds like it uses a bunch of resources. You've got Closure running on the JVM, which I'm not sure I'd be able to get the JVM on an original Raspberry Pi.
00:16:30
Speaker
yeah There were two challenges. so The first one was getting supercollider running on the Raspberry Pi. That was not trivial. and Compiling it wasn't easy. I ended up getting some support from a random internet person who just said, i'm seeing you I wrote some blog articles about trying to do this. and He sent me some compile flags and I used them and and compiled. so It was great. I got a working version of supercollider.
00:16:51
Speaker
but yeah The GVM support at the time was extremely nascent, um and it wasn't really well supported. and Also, my Clojure code was clearly rubbish. This is the standard like theme for my coding skills. and so yeah The namespaces for Overtone took seven minutes to boot on the Raspberry Pi. It did work, but so it was like going back to the old Spectrum days of putting your cassette in and waiting seven minutes for the game to load.
00:17:16
Speaker
yeah so like and I had three months of this project, so like i I was hoping I could just get that working and then build a little small DSL enclosure. That was the plan. But once I realized that the the JVM enclosure particularly weren't going to fly, I dropped back into Ruby and ah built a really small prototype using that um just to get started and built the smallest, simplest thing I could possibly build. It took me two weeks to
Iterative development and educational impact
00:17:39
Speaker
to build the first version. so This was the system called Sonic Pi that i'm I'm still using today.
00:17:43
Speaker
yes so What could that first version do that you managed to put together in in three months? so i mean ah I kind of cheated in the sense that i the synthesizers I used were all the overtone ones. so i Because I'd been working overtone for a while, I had developed a library of cool sounds, and I was able to just pull those across. so That was great. and Then all I needed to do was build a ah language which was allowed allowed me to teach the the basics of computer science, as was defined by the ah UK computer science curriculum.
00:18:12
Speaker
and That's really just sequential programming, conditionals, data structures, algorithmic thinking. um and These are all actually pretty easy to teach when you've got a very simple programming language which which has sort of statements that go through time. and Those statements do something interesting. and so All I needed to do was to get it to make a beep. so That was a play command, and and you could choose a note to play, and then to sleep for some amount of time, and then make another beep. and With those two commands, um you can actually make all of Western music.
00:18:41
Speaker
In the same way, like with the logo, if you've got pen down, pen up, rotate, and and forward, you can draw any picture. If you've got the right simple building blocks, show me the means of of abstraction and the means of combination. That's what a structural interpretation computer programmers talk about, Sussman and Abelson.
00:19:01
Speaker
If you've got the right means of abstraction and the right means of combination, you can have a very expressive system. Just play and sleep were actually really great to get started. I could teach sequential programming. We could make baselines. We could make melodies. It was great to get started with that approach. Then chuck in some if statements, and you've got some conditionals in there. It was really quite a great way of getting started. All I needed was that, because then I worked with a fabulous teacher. She was called Carrie Ann Philbin.
00:19:29
Speaker
and she was teaching in a school in Dagenham in London. And um we but every week, I'd go and help her teach the computer science lesson. And then at the end of the lesson, we'd have a chat about how it went. And then we'd say, what should we work on what should we teach next week? And then we'd discuss it. And then I would then go away and build a new version of the system for that next week's class. For the next week's class? Yeah, weekly iterations. Oh, wow. OK. they will call them um so yeah so that was that was a great way of working. So all I needed to do was to have enough for this week's lesson. And then I would work on the next version for the next week and just keep iterating that way.
00:20:02
Speaker
So at this stage are you thinking this is just a project, this a just, let me not use the word just, at this stage are you thinking this is a project for how I can help teach kids coding or do you also have ambitions to turn this into your live instrument?
00:20:17
Speaker
Oh, not at all. It was just a tool for teaching how to code an engagement um and And at the time, I was still working very much on Overtone. And I was touring around the world at conferences, performing at conferences in all sorts of interesting venues and nightclubs and and large halls and all sorts of great stuff. so Okay, hang on. You've skipped a huge chunk, then, if you're already touring as a coding musician.
00:20:42
Speaker
That was my... Going back to the time in Denmark at a conference, I want to be able to do this. I want to be able to perform on stage. That was the goal. The side quest was always, can I build a system that allowed me to be expressive enough to be able to perform music by writing code? Is that even a thing I could do? Obviously, it was a thing you could do because there were definitely others doing it at the time. It just wasn't something I could do. Could I write a system that was natural for me to work with? Yeah, Overtone became that system.
00:21:11
Speaker
so Originally, you were using Closure for live gigs. Absolutely. I had a problem, though. that had that um i was It was really hard to switch out a developer mode and and to move into performer practice mode. So, I would sit down and say, right, I'm going to do some making music. and So, I'd open up Emacs and I'd start packing away. Two seconds later, I'm like, oh, it'd be really cool if I had this feature.
00:21:34
Speaker
So I started working on this feature and then the practice session would go and it would be an absolute disaster. So I knew I needed to change something. I didn't have the discipline to not code features. So the idea was if I could find somebody who wasn't a programmer, who was interested in working with me on this stuff, we could meet up and we could talk just about the music bits.
00:21:54
Speaker
and Then I could go away and program the features we wanted later on, but in the moments we're meeting, we would only do music. so but Luckily, my brother-in-law, he was actually quite interested in the opportunity for him to come and see my kids, so he came around every week. We went to a coffee shop and we sat down with our laptops, plugged them together.
00:22:13
Speaker
I taught him how to open up emacs and to shell into my machine with ah with ah with a closure. And then he was able to and run in my namespace on my machine. and We were able to work, ah collaborate together. And we created a band called Meta X, which is like a shortcut for emacs, a magic shortcut. Was he a programmer? No, he is now, interestingly, but he wasn't. He was a but a chemist.
00:22:36
Speaker
but a so it was It worked really well because it did exactly what I wanted. It was provided a a context which didn't involve programming as the focal conversational points. so I had to explain to him, in sort of layman's terms almost, how how the system worked and terms how to operate it. to We talked about musical ideas, predominantly, not about coding ideas. That really forced me out of my comfort zone into a space where it was exactly what I wanted.
00:23:04
Speaker
And then we, actually as Metarex toured, did a lot lot of different conferences um and and even music festivals. so Really? ah Super fun, yeah. um so What are you learning there as a language designer? like if you're If you're switching between someone who needs to write a programming language or a toolset and someone who's trying to use it on stage in front of a few hundred people,
00:23:30
Speaker
Yeah. i mean so it's It's about figuring out what ah aspects of the system to expose, to to talk about. whats and and it's there's and There's multiple facets to it. so it's It's not just about, what could I explain to a non-programmer, but also, what could a non-programmer and me, even being a program, because this is also at my comfort zone, do in a live setting?
00:23:55
Speaker
um like i don't know if you When you see a conference talk where they do some live coding, and it's not necessarily music making, it's just, here's a website I'm building, and w I've made ah some blog posts or whatever. yeah um If they haven't actually the developer knows what they're doing but hasn't actually practiced that, it can look it can be really awkward and really difficult.
00:24:12
Speaker
yeah I think every programme has had that experience when they're like typing happily away, and then someone looks at them typing, and their ability to even type disappears. It goes away, yeah. yeah so it's like There's a performative element that needs to be practised and drilled in. and people don't I don't think they recognise that enough. um and so yeah Having a language which actually is is considerate of that in addition to being considerate of the fact that that they're not necessarily an established programmer, it was really critical. so it was It was really easy with the lisps. You can take the complex things, you can make them through simple forms, you can make the shortcuts, teach the shortcuts. let Do this shortcut and it will evaluate this line, hear the lines, hear the things we can modify, and then we will sit down and we will practice and practice and practice. and You can't get away from that.
00:24:59
Speaker
um And through the practice and through finding a way together, we ended up doing some really nice
Improvisation in live performances
00:25:05
Speaker
things. We got onto radio stations and yeah performed all all over the world. It was brilliant. and are you pra When you say practicing, are you like drilling specific songs the way a band might? bory Or is this more freeform? Is it more jazz?
00:25:20
Speaker
Yeah, it's more like jazz. i think so we would send up Certainly, the ah we would try and separate the practice sessions into like free form. Let's just see what happens. Let's not worry about the quality of the sound, how it sounds. Let it just be noisy. Let's just explore the space. and Then when we found interesting bits that we liked, we pointed it out and we sort of wrote them down. We put them into another file and saved them.
00:25:41
Speaker
And then we try to figure out, let's let's actually go let's do a set now, let's do 15 minutes. um And let's see, let's just try and make it sound as good as possible. And so that's what we do. So we we'd have this exploratory phases and sort of more performance practice phases and try to make those performance practice phases longer and longer. So it was 15 minutes, 20 minutes, half an hour. yeah Can we do a 40 minute set, for example? but you Were you not at all tempted to just like save those good bits to a file and then show up to a gig and press play?
00:26:08
Speaker
but Well, so the yeah I mean, no, I wasn't. i think that um This comes like to the same question I get often, which is, like why don't you get the AI, some large language model to write the code for you to make the music? The goal has always been, at least for me, to express myself.
00:26:28
Speaker
So for my musical thoughts to come from my brain through some symbolic representation into the computer, then out of the speakers, and then for me to be able to hear that and then to be able to go, I'd like to go over there and then figure out what symbolic manipulation I need to do, right? Do that through whatever means, and then have the audio come out of the speakers and take me hopefully in that direction.
00:26:52
Speaker
and So it was really more of a yeah more of a conversation. i i didn't when whenever give it when If I give a talk, I don't ever have a script of what to say. I always improvise because it feels natural.
00:27:10
Speaker
um And so the same was always, for me, what you would do as a musician um is just improvised and it was just more more fun. And also it turned out actually to be really, really important because um different musical spaces have different musical properties and different acoustic properties. and so And also different audiences have different expectations. And so if you went in and had exactly the right thing prepared, you'd have to be really sure it was going to work in that space for that audience.
00:27:38
Speaker
Whereas if you were prepared and able and capable of improvising, you could you could listen and see if it's working, you see if it's resonating, and if it's not, you could change it and keep changing until it does resonate so and keep it there.
00:27:51
Speaker
And this was no different than giving um ah workshops and programming. I gave Clojure programming language workshops to to companies. And I'd do the same. I'd rock up with no slides. And then there were like some some companies would be actually quite annoyed and it's like the sort of the affronted at the start. And they said, look, let's just, it's okay.
00:28:08
Speaker
Trust me, we don't trust you when we need the slides. and like I just open the repel and say, right, so this is what we we started. and They could soon realise that they could actually ask questions straight away. and I could experiment and show them the answers through that conversation, through the fact i i understood that I understood what I was teaching. That was a critical part.
00:28:26
Speaker
yeah and that I was able to flexibly move within that space in a very fluid way that facilitated the conversation. and They were able to learn much more effectively than if I just rocked up with slides and just read the slides out, which is what they're kind of what they were expecting.
00:28:40
Speaker
yeah yeah and I can totally get that. i mean I do something similar with this podcast. right I don't give you a list of questions beforehand. I want to see where the conversation's going. but do you i mean Were you showing up to a gig thinking, ah I'll play um i play Sam's Tune No. 3 as an opener, and then we'll see if the crowd is ready for Sam's Tune No. 8 or something? yeah well so we had yeah so was so ah so it was me um yeah so it was There was metarexitude. It wasn't just me. Do you have a sort of proto-set list in mind when you show up? Yeah. so we would have um We would have things we practiced that ah that we knew sounded nice, and that we'd have themes that we'd be able to work within and spaces. so That's kind of more like jazz. You'd have like sort of concepts, and then you'd be able to then stick at that concept for a while and until it didn't sound so great, and you then you could move on to the next concept. so
00:29:36
Speaker
that was ah That was a critical thing. so like It wasn't just completely made up all the time. There were spaces that we knew sounded good, and we knew how to transition from some spaces into some other spaces. But there was definitely a large amount left open for for for improvisation at the time.
00:29:51
Speaker
and okay so i want to get into how so this was it with jonathan graham i was saying that doesnt I want to get into how you code for that. But first, haven't you just... You're saying concurrency is important. It sounds like you've solved the concurrency problem by getting a second musician back into the room.
00:30:07
Speaker
Yeah, but so working with a system that allowed concurrency as well, so that Collogia really provided the semantics, to so to safely not just safely in terms of like proof, but safely in terms of of of of separation of concerns. We didn't have to think about clobbering states. There was always a clear way to define what what the separation was, and the language provided that.
00:30:30
Speaker
so It was totally fine for him to be running his e-match code it on my computer, evaluate evaluating the code essentially, creating threads, spawning threads, and wait for them to to interlock and to work together. They're all part of the language design. so That that and that and that was really was critical for timing as well, to make sure that the different threads didn't just run concurrently. they ran because That's just like complete freeform jazz.
00:30:54
Speaker
You don't want that. Concurrency is not enough for music, unless you just do freeform jazz. If you want a band to be playing rhythmically in time with each other, then those threads need to not just be running at the same time, they need to be running in time. Yes, synchronised concurrency. Yeah, having ways to coordinate the threads across, that was critical. But these are all things I could implement myself in my own programming time and make those abstractions for the band so that we didn't have to think about that during the performance.
00:31:22
Speaker
Okay, I definitely want to get in and under the hood and talk about how you solve those sorts of problems. But first, i mean what's what's the live you must be one of the world's experts on live coding and what works for a live coding system because you've you regularly ship code to production in four bars time, right?
00:31:43
Speaker
Yes, mean i've I've done it for quite a long time. yes there There are many others doing some amazing things too. yeah but yeah but yeah But let's put you on a pedestal for a second, because i I think it's very impressive. What what have you learnt since those early days of what makes a live coding system fluid enough that you can improvise while 100 people are watching you?
00:32:04
Speaker
Good question. I mean i think actually the the lessons I learned when I was exploring Sonic Pi really probably formalized those. so With Overtone, um I was trying to build the most expressive, powerful programming system. Like like I was imagining, I was a wizard. I was electric coming out of my hands.
00:32:21
Speaker
using a pair of macros. I was like, what cool stuff can I build? and I wasn't ever concerned with how complex this is. because If I could understand it, that was fine. like because It was a tool for me. I'm building my own power gloves. and so I'm going to use the most fancy stuff that I can master and I can control.
00:32:38
Speaker
and It did did turn out to be a bit of an issue when people came up and asked, like how do I get this? They'd see my performance and say, how do you how do I get the same setup? And it'd be tricky to get them to to up to grip to scratch. and There was all sorts of issues. The fact that they had to learn Clojure and that learn SuperClyder and Sith design. I was using a really weird esoteric Emacs config which I was called Emacs Live that I designed.
00:33:01
Speaker
um But you know that was what I was doing, and that was where I felt that my artistic performance and and credibility was all behind. and Then when I started the Sonic Pi project as as a side thing, side quest or a three-month initially side quest, just to stick around in the labs for a bit longer, do some cool stuff with the Raspberry Pi computer, that seems like a pretty cool thing that just popped out. um and That turned out to be quite successful, and and that kept bringing in more funding, and that kept me sort of working.
00:33:28
Speaker
um on live coding systems rather than going back to an industry job. I still had very much the separation of Sonic Pie, toy, teaching computer science, overtone, powerful power gloves as ah as a wizard. and um and Then I got an opportunity to work with a whole bunch of different artists with the Arts Council. They funded this really cool project.
00:33:50
Speaker
and They funded a whole bunch of artists to make music videos with Sonic Pi running on the Raspberry Pi. and There was an amazing array of wildly different performances, recordings. They're online somewhere, somewhere find those. but ah they the They also gave me some money also to make a music video. I'm like, oh okay, that's a bit of annoying because I actually want to use Overtone.
00:34:13
Speaker
And this music video was specifically to run on Sonic Pi, but it's kind of like eat your own dog food. Okay. For all these other artists you're doing, like I actually should be able to do something. yeah And so I sat, I put it off and put it off and put it off like I do with everything until the last minute and the pressure was just intense. And, um, so I just sat down and locked myself in a room cause it was a week later from when I started the deadline. And so I just locked myself in a room with this raspberry pie and a small projector I bought that was projecting in a wall cause I didn't have a monitor at home, just on the laptop. Hmm.
00:34:42
Speaker
so So, I used a small projector as the monitor and sat on the floor and just coded this music. For the first few days, it was horrible. I really didn't like this because it was just nothing. We worked. It was just too simplistic and also toy-like because it was all designed to teach computer science in schools. It wasn't designed to be like the overtone thing. I was pleased and excited to work with.
00:35:04
Speaker
yeah but Actually, halfway through, it something clicked and it fundamentally changed my relationship with live coding systems. and I think the reason why I'm saying all this is I think it answers your question, what sort of goes towards that. because It turned out that all of the design I'd put into Sonic Pi to make it simple to teach computer science in schools also translated beautifully to making a simple ah system which was cognitively simple to work with.
00:35:34
Speaker
at least for me. and Although, as a sort of wearing my mathematician philosopher's hat, I could see that the expressiveness, the expressive potential of overtone was much vaster than the expressive potential of Sonic Pi. Using an analogy, I've got a car. If overtone's a car, it can drive over all manners of terrain and and and drive drive much further.
00:35:58
Speaker
Whereas Sonic Pi is a much simpler car that can only go on roads, say, or very simple services, and has a much limited li range. um That range is only a potential And that potential comes at a cost, and that comes at a cost of time and effort. And so, in order to use Overton, although I could imagine making a system, or a sound, or a weird, um ah distributed, ah interesting ah musical instrument with multiple computers working together, and I could imagine all this and design in my head to build it would take months, years.
00:36:36
Speaker
um Similarly, if I imagined a composition that was sophisticated, to actually realise that sophisticated composition would take hours, weeks. And with Psi Pi, although you couldn't travel as far, its expressiveness was limited, what you could do in a smaller period of time was much, much, much more vast.
00:36:57
Speaker
and's just that That was much more interesting. so that's For me, that's what the live coding system, at least for me, is like what can you express in 20 seconds, 30 seconds? I'm acknowledging that you need to compromise the expressibility of the program in terms of a timeless perspective. so like Given an infinite amount of time, what could you express? That's interesting mathematically, but it's not that interesting when you're in front of a thousand people and you're performing. yeah What's more interesting is what can I express in 20 seconds? What's the range of programs I can write?
00:37:27
Speaker
So do you then find that when you're writing music live and performing live that you make simpler musical choices but that are more reactive to what's going on in the room?
00:37:41
Speaker
yeah ah so Where you say simpler, I would probably translate that to higher abstraction. um so It has maybe more of a dramatic effect. it's now I'm not necessarily ah designing the synthesizers live. I'm manipulat manipulating them at that low level. I'm working much more macro scale. It's simpler in the sense that I'm moving larger shapes around, rather than doing lots of detail.
00:38:05
Speaker
um But you can obviously zoom in or zoom out as you need to for the performance. But I tend to be much more natural at a more zoomed out scale. So not as zoomed out as a DJ, where I'm putting track A and then track B and mixing them together. umm I'm ah much more of a micro level, but not micro that I'm playing an individual instrument and ah manipulating the timbre of just one of my clarinet, say. I'm looking at a much more macro scale.
00:38:29
Speaker
and You certainly don't show up to a gig like building your own synthesizer as a prelude to... People do that, and then it's amazing. That's an amazing skill. But the chances of actually it going wrong and horrible sounds coming out of the speakers is quite high. There's like a high-risk factor. There's so many examples ah of of of features I added that made it sense in schools that also worked inside a music environment. so The kids would always, as soon as they realised, they could change the amplitude of a sound with an integer.
00:38:59
Speaker
so they could say AMP2 for twice as loud. Of course, they're going to do AMP2 billion trillion zillion. That's what they're going to do. and Add as many zeros as possible. and so yeah Adding ah ah a sound limiter and a compressor on the end of the the audio chain made sense to save the children's ears. um and These also make sense in ah in a sort of musical environment. You don't have to think so much about the mixing. It sort of gets compressed. It might sound not great, but all these ideas like like translate from one to the other. yeah That sounds like it's at its best, but do you not find, I mean, it seems like at its worst, you might find that the code that you're allowing children to write reliably makes what you're trying to do live more toyish? That's a very, very interesting point. So I think that comes down to sounds similar to a lot of conversations I've had over the years. And it's a lot. And it's it's it's really weird to me that
00:39:54
Speaker
um A lot of people think that because a system is simple for 10-year-old children, which Sonic Pi is designed for, it's designed to teach 10-year-old children how to code. And also it's designed to teach to work with teachers who have rarely done a computer science degree. that That's the that's a reality. And so you need something that's really, really approachable. It just boots, double-click, and it works, and it's all very very nice and friendly. People somehow think that a system like that, therefore, can't do complex things.
00:40:25
Speaker
and um ah it's It's pretty wild to me. and but the The reality is that most programming languages for children that I've observed do have that property. They do have kind of a glass ceiling baked into the system. and That glass ceiling enables them to make things simpler. and It kind of feels like that's kind of like what you're alluding to. and I think that um that definitely is ah is a real thing in a lot of design systems. and I personally ah might Maybe sort of shaking some some branches in the wrong way here. I feel that's lazy. I feel that it's entirely possible to put the design effort in
00:41:05
Speaker
to not just make it simple for a 10-year-old child, but also enable it to be expressive. To not have that glass ceiling be required. You've got to have the big inflatable tubes down the bowling alley so that you can you can bat ball bounces around. like ah feel like With enough thinking and thought and care and design, you're able to do both.
00:41:27
Speaker
There are examples of this everywhere, in the same way that you could have scripts for films that are catered to both an adult and a child audience. It would be much easier to do one or the other, but to do both at the same time, if with enough thought, it's possible. and and and The benefits are there. um so Any Pixar film has has all has adults and children laughing at different moments. yeah That's a beautiful thing. and Then, coming back to instruments, because this is ah this is the really the obvious one, is that people are going to the idea that something can't be complicated and simple at the same time. What's a guitar? What's a piano?
00:42:06
Speaker
ah These are very, very simple systems. The piano is just a bunch of keys that you can press. and if no one's seen a if If you haven't seen a piano before, I can teach you how to play every note in that piano pretty quickly. Same with the guitar. I can teach you how to strum every string, and I can teach you how to place your finger on the fretboard and press, and you can change the notes. It might be a bit tricky and you might struggle a bit, but you can You can certainly start. and Then I'll show you a virtuosic pianist or a virtuosic guitarist, and you'll go, oh my goodness, what I can do with this instrument is just unfathomable. I hadn't realised the range of sounds and styles of music I could produce with this one instrument is just so amazing. and That's because, I believe,
00:42:50
Speaker
The guitar wasn't just invented. Someone didn't come on and make a guitar. It's evolved over many hundreds of years. It evolved alongside practice, alongside teaching. and The instrument sort of got better at being easy to learn and expressive at the same time. That instrument evolved. The same with the piano. Piano didn't just pop into existence. It's a whole lineage of instruments that that came into the piano.
00:43:15
Speaker
and so I think that we're very early in the days of live coding, certainly with music. and so I think that that that we're we're really yet to see the exciting things, but certainly the idea that you can build something that is both powerful and expressive and simple isn't to me a fantasy, it's a definitely a reality and definitely something to be chased for. and and That's what I've been chasing with Sonic Pi.
Designing for simplicity and power
00:43:38
Speaker
I really honestly believe that Sonic Pi is both simple for teaching, that was its goal, but also really powerful and expressive. and is I know it can do those things, but marketing it and communicating that to others is actually really, really tricky. People like the live coding systems, which look like the matrix. They like the live coding systems, which which are really hard to install. now I've got to download all these terminal things. I've got to run whatever commands and pull this language in. I've got to download this complex mathematical language, whatever. yeah and That can sound all quite fun and adventure, but it doesn't necessarily mean the language is any better.
00:44:12
Speaker
or more expressive. just as a There's a challenge or a goal to installing it somehow makes it feel better. It's a psychological aspect. The synthesizers with more knobs look more exciting, but are they more musical? Exactly. i mean then You've got this with the Eurorack modules. These are the synthesizers of all the wires that go in and out. you know and so because they look Basically, they're computers inside out. They're analogue computers, essentially. um People are gassed and amazed by the complexity, which is visible.
00:44:42
Speaker
and they like that. Somehow that's an exciting aspect. and That's not to be deterred in any negative comments, but when you've got something which actually has ah thought about all of that and made it simple for a 10-year-old child, somehow people therefore think it's simple and not. They they lose their ah ability to perceive its it's potential for expression.
00:45:03
Speaker
Is there also this thing, I've heard this said by some musicians, and and we've already talked about the fact that your ability to type goes to heck when um people are watching, right? There's also, I've heard musicians say you want, you you know, you lose IQ points the second you get on stage. And is there something about making a system that even a 10 year old can grasp quite amenable to the brain that's standing on stage wondering what they're going to do next?
00:45:29
Speaker
100%, yes. um yes you you did the The cognitive load that is a fact of being in a stressful performance context. It's very real. so yeah It's definitely losing IQ points. It's definitely losing dexterity. It's definitely losing focus. um and Practice, at least for me, is the only way out of all of that. um and Familiarity with the system. but Also, a system that is designed for the classroom where an error happens and you want it to be really clear to the user what the error was and you want it to be really clear to the teacher who might pop over the over the shoulder and go, oh yes, there's the error. And that's the that's the solution there. Assistant design for that is also going to be great when you've had a few beers and it's midnight and you've got a thousand people in front of you and you type a typo because you will make a typo. yeah so sonic pie So I started to add functionality sort of halfway through its life.
00:46:26
Speaker
Not just to do the teaching of computer science, once I realized that was actually pretty much solved, and not really the hard problem. The hard problem wasn't, can you teach computer science? The hard problem was, how do you get the kids to care? How do you get the kids to be engaged? That was the interesting problem. The music really helped that, but then if you've got a music system which is not very powerful or not very good, then that's crap. No one wants to play a crap instrument. No one wants to play the rubbish instrument. They want to play the good one.
00:46:54
Speaker
And so, yeah making Sonic Pie more expressive and more capable as a musical instrument wow was both an interesting thing for me as an artist. Once I'd once i'd done that,
00:47:05
Speaker
ah video for for that arts project. and I decided that that the simple was actually what I wanted, not the complex. and I essentially never touched Overtone again. I was very happy to work with Sonic Pi as my artistic thing. and so I was sort of increasing its artistic capabilities. There were other bunch of artists working as well. but yeah that The kids also cared about that. They were like, well how why does it sound like that? Why can't I do this? When functionality wasn't on the curriculum,
00:47:34
Speaker
and wasn't possible, and you weren't able to teach it. That was actually an obstacle, rather than a benefit. um so like We've made this curriculum nice and small in scope, and so therefore the stuff we can teach is good. Actually, the kids wanted to do the stuff outside of the scope, because that was interesting to them.
00:47:51
Speaker
yeah so How did you make that something you could do whilst not actually interfering with the basic computer science sort of concepts you were teaching? so That was always an interesting challenge as well. so Not just how to make it simple and complex, but how do you sort of pull the wool over the teacher's eyes so they don't realise they're teaching concurrency? but That's what the kids want. They want to play the drums at the same time as the bass, obviously.
00:48:13
Speaker
right yeah reverb. You want to add distortion. You know you want to do a lot lots of cool stuff that wouldn't be necessarily on the computer science curriculum. Adding all those concepts in in a way that didn't and disrupt the classroom also made them cognitively simple. And then, therefore, when I'm performing on stage and others are performing on stage with the system, it actually works much nicer. It's much easier. you could but With that Overtone, I was always so stressed when I'm doing a performance. Is it going to work? Am I going to be able to get it working? Is it going to boot? Is it going to... ah If it gets a problem, how am I going to fix it? With Sonic Pi, it was just, ah, it's just going to work.
00:48:49
Speaker
It's like, I rock up with my guitar, pull it out, as long as it's in tune, I'm good. yeah I wanted an instrument that had that sort of didn't have to re-scared every single time I performed. If 30 kids in a classroom can get it running reliably, then you can at a gig, right? Absolutely, yes. okay so I think we have to dive under the hood into how this works. sure right so if If I wanted to build a live coding music system from scratch, what big problems would I have to know about?
00:49:19
Speaker
Well, I think that um you'd have to have an idea of... i mean I'm a firm believer that you shouldn't just build a system for the fun of it ah without actually having a use case.
00:49:32
Speaker
okay so I mean, for the fun of it, with an actual goal in mind. I want a system that does this. and and Either you're the user that's going to work with it, or you're going to work with very closely with the user that's going to use it. Ideally, you're the user. That would be the the best. but Work very closely with the user. and Then ah build the system really quickly, and then get using it as quickly as possible. so Bootstrapping it into something that makes something work. Then iterate, and iterate, and iterate. In order to do that, I think you need to have an idea of what kind of music you want to make.
00:50:01
Speaker
what kind of abstraction, musical abstraction you want to think in, that maybe that's what you're already thinking. um And also, I always invite people, when especially when they talk about Sonic Pi, they say, I would like it to do this feature. I always say, can you just, let's not think about it other than just do a code sketch, just write down a piece of paper, the kind of words, code words, you'd like to be able to write in a performance situation. And then let's see if that makes any sense.
00:50:30
Speaker
It starts with code sketches. It does sketch the kinds of letters, symbols, combinations of them. Do you want to use full English words? mean like It's important to me, for example, when I'm performing, not that ah not just that I can express myself, but that the audience can have an understanding of what I'm doing. and so I chose not to have a language which was very terse and single letters, like a regular expression, which would be much faster to type and faster to symbolically manipulate.
00:50:57
Speaker
but would be totally impenetrably unreadable to the audience. but maybe that's what you want. right and and so like Having an understanding of what kind of level of abstraction of of of semantics and syntax. Who is the audience? Is it you? Is it others? Are you projecting it? are you and Are you aiming it for others to be able to easily learn it or easily work with it? Or are you just building it for yourself? Are you building it and as a system that's not going to be used in terms of written frequently, but you're just going to want to express ah sort of artistic installation and just have it running?
00:51:31
Speaker
all these things would would really change the kind of language that you would create and work with. Then, of course, you then want to think about what is the sound that you're producing. and so i think unless you've got I'll be super interested in if people have any other ideas, but the three categories that generally happen are, and well, maybe it's four. Maybe you've got just normal acoustic instruments you're working with, so you're just taking audio streams into the system.
00:51:58
Speaker
like your data coming in from a microphone. Yeah, exactly. Microphone or like electronic violin or a synthesizer or something. um or your um and Or you're triggering those directly we using some protocol like MIDI or open sound control um but programmatically, or you're generating synthesizers in real time. And if you're doing that, are you either doing it in a web context or a non-web context?
00:52:24
Speaker
ah Because ah the the two main so systems I think are ah worth spending time looking at are Web Audio for the browser, which is a phenomenally sophisticated synthesis engine that surprised the heck out of me when I started of pray playing with it. It's really, really capable. so so The HTML5 spec includes a decent synthesis engine. Unbelievably powerful synthesizer.
00:52:46
Speaker
Unbelievable. There's some lacking bits that I miss. um there's There's no way to observe when a synthesizer you create is completed. There's no sort of callback mechanism. But the timing's there. There's a whole bunch of different synth nodes you can use and you can link them together to create large graphs of synthesizers, which is very much similar to Supercollider. If you're doing it non-web-based, I would put but push people towards Supercollider.
00:53:10
Speaker
unless people have a very specific style of synthesis technique they they want to explore, much then they're building their own synthesizer. That might be what you want to do. But if you do if you just want to work with an existing one, SuperClyder or Web Audio, and then pick a language, ah or design your own language, whether it's a DSL, so you're taking a language like I did, like Ruby, and you manipulate it, or do the Lisp, or you build your own sort of parser and work that way.
00:53:35
Speaker
Yeah. Okay. um I'm was surprised that in none of that you you didn't mention like timing and concurrency and scheduling. Oh, that's a very good point. Yeah. So yeah, well of course. um I think that um with all those things, it depends on what you're trying to do. So um yeah, I think that so we can get there, they're critical. um You can but you can build a good So it depends where what's driving the timing. So if you're using supercollider or web audio, then they have a sense of time. So all you need to do is in your language is to say, at time t, I would like to play this note, trigger the synthesizer, modify the synthesizer, change its amplitude to be louder or whatever. And so all you need to do is you need to build a system that is able to manage a concept of time.
00:54:29
Speaker
So it doesn't have to be performant. It doesn't have to be executing sympathetically, temporally, in time with the things. It just needs to be able to run alongside in human time and and have an accurate notion of o of when the time is. So you were off loading you're offloading-time computing to the the service. to title So Web Audio has its own scheduler.
00:54:52
Speaker
and um ah well has i it yes does i mean i No, I'll talk about it at the time. But you there are other issues to do. But generally, with audio, you can send a message with a certain time, and and it will trigger that synthesizer. And Supercloud also has extremely well-timed scheduler built in. um So yeah, you're you couldn you're using that. I mean, if if you want to do MIDI, for example, then you have to build your own scheduler.
00:55:16
Speaker
ah MIDI doesn't support that. or So if you want to be able to or if you want to be able to work with visuals software and and do some VJ stuff and send messages ah to control VJ software, then again, you'll have to write your own scheduler. So it depends. If you want to stick simply, then just work with Supercard or Web Audio, use their scheduler. And then if you cons if your language has... I don't think you need the concurrency stuff, really, ah for a very simple, fun little language.
00:55:45
Speaker
that's extremely expressive. I just want to do all of those things I described.
Teaching concurrency with Sonic Pi
00:55:49
Speaker
I want to do audio in, audio out, I want to do MIDI in, MIDI out, I want to do network messages in, network messages out, I want to control web audio, and I want to control supercollider. And I want to do it all at the same time. And I want to be able to take MIDI events into the system and then use them to modify synthesizers and web audio.
00:56:09
Speaker
and So, I need lots of threads running. I need a ah thread-safe, deterministic, coordinated state system that can work ah across all of these different things. so if you if you If you're building a thing like that, then i think that then concurrency becomes really really critical.
00:56:26
Speaker
um But if you just do a basic thing, you don't. Like the first version of Sonic Pi, just a few threads, that's fine. now like It's not really, it's it's not hard. It's when you're doing the ah the live loops and and the timing of that, that you start to care about that because you want to be able to coordinate things like stopping any threads that are created. But only if you're making a system that lets you create lots of threads arbitrarily. If your language doesn't let you do that and it and it manages the threads for you, then yeah, you don't really need sophisticatedticated I'm going to pick you up on that term, though, because I know you've mentioned this a few times in the past to me, and I know it's a critical idea. Live loops. Yeah, so in Sonic Pi, that was the way to do concurrency.
00:57:08
Speaker
and So, loops were on the curriculum. Concurrency wasn't. um and So, a loop obviously has no concurrency semantics, typically. It's just a black hole for for a thread. Once the thread gets into that loop, it just stays there. It just spins around forever and never gets out. This has got like a break inside of it or something.
00:57:24
Speaker
and so ah like But the kids wanted to do a loop with the drums and to do a loop with the bass. That's what they all wanted to do. Within a couple of hours, they were all doing it. and It was like putting their hands up, how do I do the little drums at the same time? Bass, you can't. and they're like That was one of the examples where it was clear to me that a system which was just designed to teach curriculum wasn't going to cut the mustard because it wasn't what the kids wanted. It wasn't engaging them.
00:57:49
Speaker
You need to do something a bit more magical than that. I've got to ask that before you carry on. How come you hadn't already crossed that path in your own coding? um so in So this was with ah Sonic Pi. In Overtone, I was using... Oh, you were still on Overtone in those days. Yeah. I was using a concept that came from a really nice Australian chap called on Andrew Sorensen. And he created this idea called temporal recursion.
00:58:17
Speaker
This is a recursive loop, which is a scheduler built into it. so it's like At the end of the recursion, you call yourself, but with a new time to thread through. okay yeah and then and Then you use that timestamp to then as the timestamp to schedule the things. so In the case of Supercardi, that was the timestamp you passed to say, ah this at time t, trigger the synthesizer.
00:58:41
Speaker
the recursion would schedule itself in the future, so it would be roughly just before it needed to happen, but and have a ah fine timestamp that that is is accurate, like a virtual logical time. So, we're using temporal recursion in over time.
00:58:56
Speaker
um and Obviously, then yeah trying to try to like throw temporal recursion into a computer science lesson for 10-year-old kids was yeah definitely not something that the teachers were going to be happy with doing, even though not actually that complicated, just fancy words, um but a loop which had happened to look just like a normal loop The teachers didn't realize it was a special kind of loop. So in Sonic Pi, it's called the Live Loop, because it sounds cool. And the teacher's is like, fine, it kind of ticked off the curriculum loop. But it also happened to have the ability for concurrency to happen. So the Live Loops all operated and concurrently with each other. And also in addition, Live Loops supported live hot swapping of their functionality whilst executing.
00:59:36
Speaker
So you could modify the contents of a live loop, hit the run button, and the next time around the live loop, they would use their new behavior. So you could start them off doing a simple thing and then keep evolving it, which is a kind of like what the temporal recursion also had. Because you could say, next time you I schedule myself, here's the new function to execute. yeah And so that that made sense. like I just took the idea and made it into a more sort of more usual looking concept a loop. This is sounding a bit like a couple of concepts that I bet you couldn't get through the curriculum either, but it's sounding a bit like functional reactive programming and a bit like actors.
01:00:13
Speaker
yeah Yeah, okay. Functional Active Programming, where you've got like chains of functions. i mean Weirdly, Functional Active Programming is much closer to the music world. A lot of language music systems are actually function Functional Active Programming. They have big graphs of of of of boxes of lines, and the and the data flows through them. and Actually, that's how supercardic synthesizers are designed. It's kind of like Functional Active Programming. Okay. Um, but, uh, yeah. And, and you're the one with actors, actors, actors. Yeah. The actors are obviously, obviously always running. I mean, like an Erlang actor where it's, you send him messages, uh, live loops.
01:00:50
Speaker
allow you to send the messages. they They do read from a global state, but that global state is deterministic, so it's kind of like a message box. um and It turned out that actually the they did concurrency abstractions I had to design in order to make this stuff work and work effectively well turned out to be a sort of crappy ah version of Erlang's beams OTP stuff.
01:01:12
Speaker
So, it just didt me and when I met Joe Armstrong, who was in the co-creator of Erlang, it was interesting to have a conversation because a whole bunch of things I sort of had to invent in order to make this stuff possible were already clearly part of of OTP for for for many, many, many years.
01:01:28
Speaker
So it was quite an interesting sort of conversations to to discover that overlap and to realise that I had ah kind of invented, through necessity, stuff which had already been invented through other necessity of building a telecommunication system. So it is interesting that these concepts, they're there, like mathematically, they're just like there to be discovered, rather than yeah Yeah, this is why some of our concepts end up sort of overlapping and looking similar, because there's just almost like there's some fundamental shapes out there that we keep discovering. When you hit that point, were you at all tempted to kind of move to Erlang?
01:02:03
Speaker
Yes, um very much so. and In fact, I'm um doing that now. right tell me about yeah and so more like the that um The concurrency of 7-side Sonic Pi works, but it's got to a complexity now internally that there's too much technical debt really to easily modify it and keep it going.
01:02:21
Speaker
and without really requiring a large chunk of it to be written rewritten. because other otherwise As I was discovering the concurrency abstractions I needed, I was implementing them and and kind of like exploring them whilst I was implementing them. themselves They changed shape and they ended up in a weird thing that if I were to reimplement them, they would be much more clear and simple. But also, they baked in the domain. that so they were weren't like With Erlang's OTP, these concurrency things are abstract of the domain you're working with. It's like a function. It's a thing you can apply to any kind of programming language. but this is These are kind of functions that were baked into their main musical concept, so woven together. You need to unpick them apart ah to then be able to work with them more effectively. and and That's just a huge amount of work.
01:03:09
Speaker
and ah would leave it things to be very broken. and so If I'm going to do that much work, I may as well move to another system which actually has those abstractions that have been designed by engineers for over 30 years.
Refactoring challenges and solutions
01:03:22
Speaker
and to Tons of work put into it and really tested and really worked well rather than building my own stuff.
01:03:27
Speaker
So, yeah moving to Erlang makes a lot of sense, and that's the path I've been looking towards doing. But of course, if you've got a system which has got a lot of traction, a lot of users using it, a lot of schools are using Sonic Pi, the language is well supported, like chatgbt understands it, so there's obviously a large corpus ah of of Sonic Pi tracks out there for it to learn from. yeah Moving to another language is non-trivial at this point, but it's never lessening and planning on doing.
01:03:52
Speaker
So, I did spend all my time moving as much of the internals to Erlang as possible. So, all of the output infrastructure, the IO, and the um and messaging infrastructure, the scheduler as well, these are all implemented in Erlang.
01:04:08
Speaker
Oh, today? Yeah, today, yeah. so Sonic Pi contains Ruby for the language, C++ for the GUI, still got the synthesizer still implemented in Clojure, although that's not part of the runtime that was used in compilation, and then Erlang for the IO stuff. I'm super excited for the synthesis.
01:04:26
Speaker
that i'm I'm surprised if you can get that running on 30 kids, schools, computers reliably. On Windows, yeah and when yeah yeah. As an ins installer, yeah. yeah so works It works well, surprisingly well, actually. Really? Yeah, it's just a lot of effort to actually figure out how to package it all up, but once you've done that, it just works well, yeah. Okay. yeah And then have you not got have you not just created problems with getting different subsystems to communicate with each other?
01:04:49
Speaker
Oh, it's a nightmare. um yeah it's like ah so It's like all the problems you have with um microservices, essentially. I've baked into my application, so I've just brought those problems in. so having to Having to be able to reliably start all those processes up in the correct order, to monitor that those processes are running, if one of the processes crashes,
01:05:08
Speaker
You don't want it to zombie the system, so you have to observe that. So have I have monitoring software I've developed, and that was also a pain to get that working similarly on Mac or Sound on Windows, which then starts all the processes and monitors and maintains it and and watches it, and and then runs in the background and you have the GUI talks to it every 10 seconds to say, i used to I'm still here. and when If a GUI crashes, and it doesn't send a message, and there's like a kill switch happens, and then it just shuts down all the programs. so It's like designed to to shut down unless you keep it alive.
01:05:39
Speaker
and that that way That seems terrible for live performance. i mean Terrifying for live performance. As long as it's working, it's fine. right but yeah and mean yeah yeah i mean so If you've got a crash, right you want it to all die properly. um and so As long as the system doesn't crash, then you're good. um and so that's about does the Is the program reliable enough not to crash?
01:06:03
Speaker
And so that it doesn't. I've not ever had it crash on me for for many years now. If you make a bug in the program whilst you're running it that's within the runtime, then that's a different matter entirely. That won't crash the system. This this demon thing won't won't come into existence. That's only if there's some C++ plus plus pointer error. you know and and You can't read the memory, so get you get a crash on that kind of level.
01:06:29
Speaker
okay yeah I have to say, you you describing this monitoring and observing and restarting system, that sounds like yet another part of Erlangue've reinvented. Exactly. exactly okay yeah and But on on the process level. so It's more like a really crappy mini version of Kubernetes, but I'm not monitoring but containers and monitoring processes. But it's a similar kind of concept, right? like like You want to be able to observe all the things you've created, to to observe the order that they get started in, to make sure they're all running, and to have a safe way of making sure you can tear them down in the correct way. So it does all that kind of stuff. yeah So what will the future version look like? How much of that will be done?
01:07:11
Speaker
yeah I think that's um one of the the the biggest... ah In terms of where I'd like to go, I feel like I've got a really nice expressive language. and It's beautiful to work with. I love live coding with it. There are artists around the world now performing with it, and it's beautiful to see. um so I'm really happy with the language.
01:07:29
Speaker
um but There are two sort of key things I really want that I can't easily do with what I've got currently. yeah One is drastic innovation of the graphical user interface. so This thing is in Qt. um It's really slow for me to develop and modify and change and to keep to try new ideas out.
01:07:51
Speaker
and have ah loads of ideas I want to to try out to to see how if I can make the experience of both teaching computer science with it, but also as ah as an instrument, and and mainly there around giving more feedback, visualising what the computer is doing in a useful way.
01:08:07
Speaker
oh So like kind of like a live debugger, I guess, but but more more of an instrument. um So there's more that those kind of aspects I'd like to explore. And also integrating things like visuals. I want to be able to perform with Sonic Pi and not have to bring in external visual systems. I want Sonic Pi to be able to be the visual system as well as the audio system.
01:08:25
Speaker
You just want an HDMI cable out from your laptop to the giant LEDs wall behind you. Yeah, in one process, one program I did to install, and that's the one thing that kids can play with and the one thing I can perform with. At the moment, I've got like a complex setup where I'm using professional VJ software and a mixing software and NDI to take video. There's a lot of stuff I've done to make it work, but I'd like to be able to somehow bake that into my free open source application. And then the other thing that really is holding me back is that I want to build a platform that lets the world collaborate and create compositions together.
Evolution into a collaborative platform
01:09:01
Speaker
So Sonic Pi is a really nice language for composing and for performing.
01:09:06
Speaker
But it doesn't lend itself well to sharing your compositions because you're effectively sharing code which can contain all sorts of nefarious elements with it that are getting evals directly on on the machine. so yeah It's the same problem as don't ever run a bash script where you're curling from a ah yeah URL some random bash script and running on your machine because you've got no idea what that script is doing.
01:09:35
Speaker
Play the kick drum four times and then delete the hard drive. Precisely. yeah and you can Because you can shell out to the terminal, you can totally do that. yeah like Install this this um dodgy software on my machine. you know um Install a key a key hack thing, whatever. so This is something I've always avoided, is is creating this online sort of corpus ah of of of compositional elements, four pieces. and data rifts, as I'm calling them, or little bit of abstractions of parts of of code, of parts of pieces. Because I don't want to have the, the the what's the word, um responsibility of making sure all those things are safe. um yeah So I want a language which is by design safe.
01:10:18
Speaker
and that's not Ruby, and it's not Erlang, it's not Elixir, it's not Haskell, it's not any tradition, it's not Python, because all those languages don't really have a ah good, solid sound sandbox to say, I'm only going to allow this part of the computer, this part of the language to run, and it can't talk to the disk, and it can't talk to the network, and <unk> etc.
01:10:39
Speaker
Right, yeah. to the the The famous language which which does this is is JavaScript, obviously. That's designed to run other people's coding machines. That's what you do whenever you go to a website. so that was That's an obvious choice. But I also want to explore the idea of collaboration. I want to be able to do I've heard about the old software that allowed people to jam together ah remotely. so It would essentially send, not the audio, because people can do that. It's like having a Zoom jam. I don't mean that. I mean, sending the events, like I've pressed this keyboard trigger, or I've run run this code, which is ah triggering these notes. I want to send those events over to another machine in another part of the world and have them generate the the sound.
01:11:20
Speaker
And I also want to run little small algorithms, like little live loops, and and send those over to the side of the world. So I'm not having to send all the events, I'm just sending the code generates events and have that run on the person machine. So that's kind of like a sort sort of continuation of something that can be transported.
01:11:36
Speaker
um so I want those things. And also to do that, it seems like ah an interesting approach is to have it all running in a central cloud computer, um which then can then coordinate all these multiple computers together to create collaborative jamming sessions, a bit like having chat rooms, but i put jamming.
01:11:55
Speaker
And then having a language which can run on that cloud server, ah is ah it won't allow the users to then hack the cloud server. So ah JavaScript is one approach. I could use that with some like Node.js or something. But um I'm actually looking at using Lua because there's a really nice implementation of Lua called luwhirl.
01:12:12
Speaker
by Robert Verding, who's a co-creator of Erlang. That runs in the Erlang beams. I'll try and get everything into Erlang. If I have one process, it just makes everything much easier. Erlang lets me, or Alexa particularly,
01:12:26
Speaker
which are both languages which run on this this virtual machine called the Beam. I can do all my IOS stuff, which I already do with Sonic Pi. I can ah ah also then generate events to then be sent to the synthesizers. That could be super-clad, it could be Web Audio. So I've already built a prototype ah with the University of Sheffield ah called BLEEP, which is exactly this, which is ah an Elixir ah web app that runs on ah on a server which then uses the the browser as the GUI and also the synthesizer and it's sending all the the events of what to play and when to play with the timestamps that we talked about before across the the wire down on web sockets. So that seemed like a really and nice approach to work with.
01:13:09
Speaker
And then I can coordinate all those things. So if I have and if i have the code running on the server and ah in a sandboxed environment, I can effectively then send code to be run on that machine to then send all the events to everybody else. Or I can then read the code to be put on other people on machines locally. There's loads of opportunities where your code is essentially could be can be moved around to the most sensible logical place. And then you're firing events from that from that place. And Lua can be effectively sandboxed like this.
01:13:39
Speaker
and Entirely, I mean, it was designed to be sandbox, so it's designed to be a hosted language. It's used a a lot in um and the gaming industry, so you can easily sort of chuck it into a C++ plus plus app. and and you can then And it's really nice to have bidirectional comms from the language. You can expose functions to Lua. You can also call Lua from the host language as well, bidirectionally. And the Lua implementation is really nice because it has this interesting property where you essentially essentially create a data structure in Erlang, and that represents the Lua virtual machine as a data structure, which which is essentially an empty computer almost with memory, which you can persist, and you can load it back up again. And then you can then inject some code into it, and then you can run run that code, and then you can then inspect what values there are on it, and then you can persist it, and you can load it up again. um and so you can and Then you can clone it, and you can do all things with different ones. and um As it's executing, you can you can say to it, when you call your Lua function, actually, you're going to call this Erlang function or this Electra function. so That's the way for the Lua to call out to the system. and You get to control exactly how that happens and where, and so you can maintain the sandbox nature by by keep by making sure those functions are sound and safe for themselves.
01:14:55
Speaker
Oh, so you, as the Lua host, decide which lower-level functions it has access to, and you don't give it shell access? Entirely. In Lua, you get essentially a table. everythings Everything in the key-value pair tables even lists weirdly. This is one of the annoying things about it. Key-value pairs where the the keys are integers. That's how they're lists. But the the function tables are the same kind of thing, and you can just remove the functions which do the IO and remove the function to do disk stuff and just take them out and remove the functions that could do system calls. And so that your virtual virtual machine literally can't do that anymore. um And then you can say these are the functions which which are mapped together. When I call this Lua function, I'm actually calling this Elixir function and you implement that and you control what that does. So that maybe sends a message out to the browser to say, trigger this note. um Yeah, and and then you have that you have that complete control. That's sandbox nature. It's really nice.
01:15:52
Speaker
I can see how you could then make that safe for both musicians and kids to collaborate. Once you've got that, you can build this huge corpus of music that people can share and jam with each other. It's going to be a beautiful thing.
Latency and networked music collaboration
01:16:07
Speaker
I can see two problems that you must have thought about, so I'm going to ask you. One is latency. As soon as you're going over the network, music I know A musician can cope with about 10 milliseconds of latency, which probably doesn't even get me to France at the speed of light, does it?
01:16:25
Speaker
ah won No, probably we not. no um I mean, actually, musicians can cope with latency just fine. and That's what an organist is. They can cope with two or three seconds of latency when they press their key down, the wind builds up in the bellows, comes up to the pipes, and then then they hear the sound. So, latency is not necessary. It's not ideal, of course, but latency is something that musicians can deal with.
01:16:50
Speaker
if they're trained. Part of an organ scholar's job is not just to learn how to play the notes, it's how to play the notes ahead of time and somehow be playing in time when the sound is coming multiple seconds after you've actually triggered the notes.
01:17:05
Speaker
The hard part is actually is is um is jitter when you press something and it's a random amount of time after you've hit it. That's something you can't deal with and that makes everything just sound rubbish and drunk and terrible. That sounds like the internet. Both parts, actually. yeah so but both of those commit one so Certainly, the the jitter can be obviously always ah dealt with in the same way that a a video stream or this the the fact we can talk to each other now.
01:17:33
Speaker
um with with a but adding adding latency and having a window. and so that If the messages come out of order slightly out of time, you're still reading them in at at a sort of sense of sort of periodicity. so By adding latency, you can cope with Jitter.
01:17:47
Speaker
um to the real questions about latency. um With the first version of Sonic Pi, the machine and original Raspberry Pi was so slow, um I actually added by default half a second of latency ah from when the code got executed to when the events got heard, just to give the system some room to breathe.
01:18:10
Speaker
And that was what I needed the the early days. And I've actually kept that all the way till now. So even though now I could actually reduce that down to tens of milliseconds, it's still 500 milliseconds. And no one has ever complained.
01:18:25
Speaker
There's never been an issue. And you've played low for that a lot. Absolutely. but Because that's about the scheduled events. It's not about when I press a key, then I have to wait for that 500 milliseconds.
Real-time coding strategies
01:18:38
Speaker
So it's more about the system generating the events generates them 500 milliseconds ahead of when they need to be played.
01:18:45
Speaker
that's okay so it's only to Now we've reduced it from jitter to latency, and we've taken latency and actually reduced it down to, it's only the immediate events you care about. When I press a button, I want to be able to hear that sound.
01:18:58
Speaker
That's the actual problem, and you can't solve that. so that so if you As long as you're not doing, I'm playing the keyboard, I'm hearing the sounds, and I want everyone else I'm jamming with across the internet to be able to hear them ah directly at the same time as me, then you're okay. If you're just doing, I'm writing code, the code is making music, then you're fine.
01:19:19
Speaker
Because as a musician, you're not really playing instruments as much as you're cueing up a composition. Precisely, yeah absolutely. And you can cue it at 500 milliseconds ahead of time, and that's totally totally fine. It feels immediately enough to work with. It's no it's never really felt latent to feel like Mars were in the future.
01:19:38
Speaker
and But when I press a key, I want to make an event happen, that is an issue. And Sonic Pi has sort of a cheat mode for you can set put a thread into a real time mode, and it won't add that latency. And so it does that real time, but that that works obviously just fine.
01:19:53
Speaker
in your on your machine, when you're generating the event on your machine, and you're hearing
Future remote performances and visual management
01:19:58
Speaker
the event on your machine. If you're trying to then coordinate that event with others, that's when that's not's not going to be so simple. But it would be possible, if you're happy with a bit of latency, to for that to work, certain kinds of sound, actually, so we can actually reduce it even further. If you're doing very sort of synthy, background-y, drony sounds, the punctuality of it isn't that much of a problem.
01:20:21
Speaker
It's only really rhythmical things you care about. um So even then, it's like a smaller subset. um But then you could then imagine things like, I want to actually play it locally, create a data riff a data structure, and then send that to be scheduled and played everywhere else. Define data reforming because I love the term.
01:20:39
Speaker
Oh, it's just a fancy word. The phrase I came up with is just really a time series data. so like At this time, do this event. At this time, do this event. and so okay so like I'm imagining like recording like me playing the keyboard or twiddling some knobs or pressing some buttons over a particular window, recording all those events as a time series so data data structure. It could just be JSON or whatever. And then ah giving that a name.
01:21:02
Speaker
and then being able to then ship that across to others, and for them to then plug it into their algorithms, running the little Lua things, and then they can play the same thing I'm hearing. right right so That would work perfectly fine. um so I could create my data riff locally, ship it out to everybody else, and then say, at this time, go, and then we would all hear it exactly at the same time.
01:21:20
Speaker
Although we wouldn't hear at the same time. That's naturally not actually that important either. We'd all be hearing it staggeredly at different times. What would matter is that the relative timing of each note is is the same across all formers. So as long as we end up hearing the same composition? That's all that matters. Yeah. yeah Because you never get ah you've got got the fact that you've got the speed of sound is always an already an issue.
01:21:39
Speaker
Like if you're over to the side of a football pitch and I clap my hand or tingle a bell, like the sound comes much later than the the visuals. yeah So you can't you can't get away from the fact that sound is always going to be at different times. But what you can do is make sure that the relative timings are all the same for all listeners. So d is there a future then in which you are collaborating with another live coder to play to an audience who are all like completely separate geographic locations as well?
01:22:09
Speaker
So this is the goal, yeah. I want to be able to to say to an event, hey, I'll do a performance. All you need to do is open up a browser, connect the computer with the browser to a decent sound system and a projection, and then I will then control that browser remotely. And I can do it either myself or I will collaborate with others, and we will work with our events and send those events to that particular browser. And all they need to do is hire the backing dancers.
01:22:35
Speaker
Well, if you go if you want dancers, we just call visuals and call lights are probably fine. But yeah, I think that the ability to essentially control another browser ah through the web audio and through web sockets essentially attending web events and doing the visuals on the browser as well, yeah I think is actually a really interesting space. And you can even do this when you're co-located. I'm totally imagining, because when I'm performing, for example, I'll want to see the codes I'm writing on my screen.
01:23:01
Speaker
and I want to project that code with some cool visuals. and When the cool visuals are particularly going crazy, ah that might make the code slightly harder to read. and That's a compromise. like it's It's nice to be able to have like some cool fancy flashy visuals a bit, and then the code be used to read a bit, and then go back to visuals a bit. and Whereas if you just have the code all the time on its own, that can be a bit stayed and a bit boring. um But I do not want to see the visuals when I'm performing.
01:23:29
Speaker
like I don't want to see them. like They'll annoy me. I want to be able to see see the code. yeah So there's a difference between what I want to project and what I want to see. So having a system that lets me control the browser, essentially, and composites a view for that browser, which is different than one on my local machine, ah it would also be quite interesting, too. So this would be the same kind of thing. if I imagine this performance with me and so made me me and you, Chris, we're jamming together at homes.
01:23:53
Speaker
we would have our own unique view of what we're doing. I might have like a ah window of what of of an aspect of what you're doing, you might have a window of an aspect of what I'm doing, and then we would then create a composite window of our combined performances to send to the performance venue.
01:24:09
Speaker
make some choices about what we actually... Are we going to send our code? Are we going to send the visuals? Are we going to send the events? and so Whatever is the right approach for that particular performance. and Being able to mix and match and work with with the with a composite set of of of graphical elements, I think is a quite interesting place to be.
01:24:26
Speaker
yeah And is there going to be someone separate just working on the visual side of the performance and all that stuff? yeah Exactly. I just want to see their thing emphasized. and Maybe they want a little window of what we're doing to be able to see. so when i When I perform, I do see my visuals as a small box in the bottom right, so I can see it, but it's not like not interfering with the code. It's not making it hard for me to read the code, which is, for me, the critical thing.
01:24:48
Speaker
OK, so I have to ask you the other big collaboration question, which is, like, you're, I don't know, um you're doing something with the kick drum. I decide I want to do something with a kick drum. Is it just a collaboration? Like, is it is a race condition? Whoever got their last wins the kick drum? Or are you looking into doing CRDT stuff? Or are you just going to agree before the gig that Sam does drums?
01:25:16
Speaker
Yeah, and that's a very good question. I think that the answer is yes. It really depends on on what we want to do. I mean, is there only one kick drum? so like Typically, when I'm with Cosonic Pi, you can have as many kick drums as you want.
01:25:31
Speaker
um And so we would just have our own kick drums and we'd just be kicking them, banging them away.
Collaborative coding techniques
01:25:36
Speaker
like And that would be totally fine. But maybe we want them to be in time with each other, or maybe we only want one kick drum. So these are all decisions to make as ah as an artist about what the performance is and what the composition is.
01:25:49
Speaker
um I think that ah in terms of like the broader question of coordination, in my mind, I have CRDT-like structures for ah dealing with time series of events that I want to be able to have be generated independently, but always be able to be brought together in a deterministic manner. and Then coordination happened after I've made some attempt to bring them all together. What does it look like? and That's how I make decisions off of of of that. so Bring the data streams together in a deterministic way, and then look at the weaving of them, and then make some decisions about that. so if we're If we're both conflict fighting for the kick drum,
01:26:32
Speaker
then there will be some way to say, once we've sent our kick drum events, they get merged together. Who clobbers who? clubers who and That will be part of the decision of of the design of the system. and We might say, well, actually, you're in charge of the kick drum, but there might be moments when your mind is doing something else, and I want to be able to drive it. If you come back to the kick drum, more you should be able to take over from me, for example.
01:26:55
Speaker
It's a sort of soft lock, some particular instrument. The way I see it is, like if you've got two events which actually are actually operating over the... or ah creating the same kind of event, a similar kind of event, when you read it. So if you go into the data structure and you read an event...
01:27:11
Speaker
and There are two values in there which purport to be that event. You need some way of of having the resolution a conflict resolution. Which one are you actually going to read? and so um I think that that needs to happen, I believe, to for for for it to be interesting and and useful across performances.
01:27:28
Speaker
that needs to be deterministic, which is the CRDT start. You will have some way of saying, add a priority to it to each event. But if the priority is the same, maybe you want to timestamp. But if the timestamp is the same, maybe you want some just unique random UUID, and you alphabetically choose the one that's just first. Some kind of tiebreaker. Some tiebreaker that you can then always deterministically choose that one. and then so If we care, your child has a kick drum, give your thread a higher priority than mine.
01:27:56
Speaker
And that's part of our programming language we're writing. It's not the tiebreaker that happens at the end when everything's the same. It's something we can choose as artists.
Learning and mastering live coding
01:28:05
Speaker
Okay. Yeah, that sounds like fun. So, in that case, with a view to this whole system being rewritten this way, and the gig that we will do one day, five years in the future, when we're playing Wembley from our bedrooms, how best... I want to be at Wembley, to be honest, even in Wembley. That one you're going to actually come in for.
01:28:26
Speaker
So, this raises the two questions. If I want to learn how to do this, how do I learn how to do it from scratch? And how do you even get ah good enough to live code on stage? It's a really, really good question. I think ultimately you have to start by actually wanting to do it. um that You need that drive to be able to put you through practice. Just like when you learn traditional musical instruments, you have to want to do it to put in the practice.
01:28:54
Speaker
to be able to get the motor skills or the listening skills or the performance skills necessary to be able to do what it is you want to do. So you need to be able to want to do it and you need to have the dedication to practice. I think that's the same across all instruments and live coding is no different.
01:29:10
Speaker
I think what you learn is slightly different, and how you learn it's going to be different. but Ultimately, it's that that that sort of desire to want to do it, and the desire to practice, and the discipline to do that. That's essential. so We're starting with that. and Once you've got those things, I think really then it's about but live-cutting. First, for you to choose which system you want to work with. right so There's actually a whole bunch of them. Some really cool other live-cutting systems around. so I would play around, just like If you wanted to learn instruments, you might try trombone, or you might try a guitar, or you might try piano, and you go, yep, piano is for me. um so Similarly with live coding systems, there's tidal cycles, which is fabulous, there's Ixie language, which is a nice at language, there's Foxdarts, there's Impromptu, there's there a whole ton of really cool lines. so Go and explore the systems, find one you like, and then stick with that. and If that's Sonic Pi,
01:30:00
Speaker
um the The best way to to to learn it, really, is is twofold. One is to work through the tutorial. so Because it's designed for schools and teachers, it's designed to be super, super simple. so That's actually one of the advantages of Sonic Pi over other systems. In the design of the system, it's not just a desire to build something that's expressive, but it's also a desire to build something which is simple to learn.
01:30:22
Speaker
and so That's one of the main reasons why I learned Select Pi. and It has a tutorial built into it, so I'd read that. um I've also just created a course that you can you can buy, which starts with Absolutely Scratch, which will teach you the very basics of knowing how to write code and what the code does, how to make a sound, how to make how to play a sample, how to add effects to it, how to make a melody, how to make a drum pattern, and how to do your own compositions. That's the first course as that. but Also, just reading the tutorial will get you a long way there. so If you're a video person, the course, if you're a words person, the tutorial, of or maybe you want to use them both.
01:31:01
Speaker
and then Once you've got some basics down, the critical thing is... and Another option is, obviously, if find someone else. Someone else knows Sonic Pi or the live coding system. Go and hang out with them, make friends with them, and then then and learn from them. Once you've got it down, I think the critical thing then is how to get good at it. It's about putting aside any idea that you're going to make something that sounds good. Sounds counterintuitive. Yeah, really. so I think that yeah the critical thing is that Certainly with with Sonic Pi, what you really want to be able to do is to be able to understand the semantics of the execution of the program. These all fancy words to say, you need to understand what it's doing, what the words, how they translate to the sounds. Because once you've got an idea, once you've got like a ah very basic Sonic Pi interpreter in your brain, that can read the code and sort of imagine the sounds, you can do something magical.
01:32:00
Speaker
which is you can imagine sounds and see the code. And once you get to that level, then you can do something really fabulous. You can imagine the sounds, you can hear the sounds you're creating, you can imagine the modifications to the sounds, and then you can use your brain and understanding of how the code works to then project out what the code is to write or to change to get that to get to that place. so And that's where you want to be. So I think to get to that place, you need to be prepared for it to sound terrible for quite a long time.
01:32:29
Speaker
and That's because instead of trying to make good sounds, you're trying to understand how the system works. You're trying to understand the logic of the live loops, to understand the sequential operations, to understand how the different syntax elements work together, how the semantic elements work together. and You get a feeling for it. It's not just something you can just read and learn. It's something you have to practice and experiment with.
01:32:53
Speaker
And that's why it not sounding good is really important. You have to be able to write messy code, make messy sounds that aren't designed to sound good, but are designed for you to learn how it works. And the way I would do that is to always, before you press the run button, so you write a code, before you press a run button, try and read the code and imagine what it sounds like.
01:33:17
Speaker
and see if you can hear it in your head. Press the Run button and then often you'll be surprised because you got it totally wrong. and Then sit and figure out well why I got it wrong. and Then think about another change you can make.
01:33:30
Speaker
and then do the same again. If I make this change, surely it's going to do that to it. Press the wrong button. Oh, no, I've got that wrong again. and Then perhaps make a smaller change, and then see if you can understand what that's going to do. and To the point where you do know what it's going to do, and then you can grow south back from there larger and larger. If you're a programming person, you might have read about test-driven development.
01:33:51
Speaker
It's a similar kind of approach where you you would write smaller and smaller tests until you've got one that that you understand that works and you can then step back and make bigger bigger tests, bigger jumps and bigger leaps of functionality. and that's really This is really bigger leaps of ah cognitive understanding of how it works. so Once you've understood it ah sort how it works, then you can start to use it to your advantage. you can then and As I said before, you can imagine the sounds and you can see the code. That's where you want to get to. but To get to that first, you need to be able to see the code and hear the sounds.
01:34:21
Speaker
I'm building up that kind of feedback loop inside your own brain. Yeah. You'll be able to read the code, hear the code, and then you can see the code. And then that's when you're in a really interesting place. But this is like years worth of work, right? So that you're not going to get there straight away. So whilst you're getting to that, whilst you're putting in the practice of code reading, code understanding, code experimentation, I think it's really useful to also make it fun in and certain elements as well. So it shouldn't always be labor.
01:34:50
Speaker
and so A really great way to get bootsar bootstrapped and to make things sound fun to start off with is just to work with samples. It's to download a sample pack of a musical style or genre that you like. and There's a whole bunch of them out there. You can buy them for super cheap, like €20 or something. These are pre-recorded pieces of like someone playing a flute riff or something. Exactly. Or a drum riff, or you can have like some kind of like heavy metal thing, or you can have some operatic thing, or whatever the style of music is you like. Buy a sample pack of that. and then If you don't know anything about music and theory, actually that's probably a really good point. I remember meeting a few musicians now who are properly, formally trained, highest level university music conservatoire type people, who have also the same thing independently, which is, I wish I could unlearn music theory.
01:35:37
Speaker
Which has always surprised me, because obviously they've dedicated their lives to learning it. But actually, often some individuals can feel that learning all the musical theory constrains them. They're only able to make good sounding music in in terms of of the defined definition of good. And actually, interesting music evolution comes from breaking those patterns and and doing things that aren't defined as good, that are just interesting and and sound interesting.
01:36:03
Speaker
Yeah, there's that classic thing if you learn the rules to forget them, but forgetting them is hard, right? Yeah, really hard. so If you're starting from a place where you don't know any music theory, actually, that's a pretty cool place to be. so and All you need to then do is worry about ah making things that sound fun to you, because they'll sound fun to others. But as I said before, don't don't do that initially. do that's That's the latest step.
01:36:23
Speaker
But yeah, but if you want to get there quickly and you're not worried too much about learning lots of complex things, just making a live loop with some samples that you've you've ah you've liked, or the style you like, and mix them in and out. It's actually a really quick way of of getting to understand the motor skills of working with multiple things at the same time and bringing elements out. So you just like, say you have three live loops, one with drums, one with bass, and one with synths, or one with vocals, one with guitars, one with bass or something.
01:36:52
Speaker
And then you can then switch out the vocals to another vocals that sounds similar, but sounds different, or switch out the drums, switch out the bass. And just by doing that in and out, you can actually get to an interesting performative space really quickly, um but also making music you like.
01:37:09
Speaker
And then, so that's kind of like working like a DJ, but working at the stem level, working at the an individual level of a drum line or a bass line a vocal line and mixing those in and out. um And that's really to do. All you need to do is change the sample name, press a button button next time around the live loop, it pulls in that new sample.
01:37:28
Speaker
And this is the next course I'm going to write. She's going to cover exactly this. um But yeah, that's a great way to get to an interesting place. And then you can start to then embellish that by adding your own little synth bits, or maybe you want to then slice the drums up into individual drums and rearrange them and use ah algorithms to deterministically randomize the drum patterns. And there's loads of cool things you can do that you can get to, but just simply playing samples of a style you like together at the same time.
01:37:56
Speaker
mix them in and out, gets you to an interesting place pretty quickly.
Educational resources and video courses
01:37:59
Speaker
Yeah, that sounds like a lot of fun. I have to ask you then, um on that, like do you have a lot of plans for different video video courses for different audiences? Because like you teach kids, you teach people who might know programming, but not music. Do you teach musicians that don't know programming? Yeah, i mean that's that's a really good question. too I think one of the missing parts of Sonic Pie is it's that there are people who can get started really quickly,
01:38:24
Speaker
And the tutorial built into the Sonic Pie really gets you there really quick. And the simplicity of the system also lends itself to just getting started making your first notes. But then there's a whole void of people who just like, I don't know what to do. I don't know how to make it sound good. I don't know how to make something that's that's my style of music. But there are people somehow magically who have the persistence themselves to pop out of that and make beautiful, amazing things.
01:38:48
Speaker
and these are a growing number of people now who do some sophisticated performances. But they are quite rare. um and so It's more about trying to to make that pathway easier pathway for more people. and so I think that's really about creating learning resources, and I think course is actually a great way of doing this, um for bridging that gap between, I've just got started to, here's where I want to be.
01:39:12
Speaker
I want to be able to live code a drum and bass track. I want to ah compose an operatic piece. um And so I think there's actually different courses for different styles of music, as much as there are courses for ah very common used algorithmic musical structures that are going to be in your operatic piece and in your heavy metal piece.
01:39:31
Speaker
um and Then I think there's also and but probably be ah an interesting set of courses for specifically teaching computer science, so taking the computer science angle and using music as the as the vehicle for teaching the computer science concepts for those who just want to like learn how what a function is. what what ah ah what a syntax is and what an algorithm is. you know and Those concepts, we can we can teach directly in using the music as an idea. Although I personally find that teaching Sonic Pi is a musical instrument, and have it happily learning the computer science as a background is the better way to do it. Much more fun. Mason- Sneaking education into kids' heads. yeah absolutely yeah yeah
01:40:10
Speaker
yeah so yeah i mean if If people got ideas for what they would want to learn, but i'm going to do one so I've got this introductory one now, which is a really nice way of just getting started to make your own first composition. and Then the next I'm going to work on one, which is going to be specifically for performance. so This will be live loops. how do i How do I play a bunch of things at the same time? How do I modify it as it's working through time? How do I take my take my samples for a sample pack? How do I play them? how do i ah Make sure this sounds the same, so the right key, the right write tempo. What do those things mean?
01:40:41
Speaker
um That's what you really need to know, those two words. and Then, how do I do interesting things with my samples? I don't want to play the same one every time. it Maybe I want to add an effect, or maybe I want to reverse it, or I want to slice it up as I said before, and do various simple, really simple tricks that give you a lot of like creative possibilities. and My performances really aren't that complex. They're just lots of simple tricks I do at the same time, which in combination sound complex, but actually in isolation are really, really easy to to learn. so I can believe that. One musician I really like says, three simple things put together sounds very complex.
Live coding in corporate environments
01:41:18
Speaker
Yeah, I can believe that too. Exactly. yeah yeah i mean i'll I'll take another drum track.
01:41:23
Speaker
I'll take a bass track and then um I might take the drum track and not play it in its normal speed, but slice it up into individual drums and then randomly play one drum track, but but one drum after another and then do that in a way that repeats every sort of 18 or 16 bar that beats and then you get this pattern and then you can change the seed and change the pattern until you find something nice and tricks like that are already I can teach that really really quickly and they can you can use that same trick with a drum pattern and play the drums out of order We could also pass a bass riff, and you can play the bass hits out of order, or you can put a vocal track in there, and you can play the vocal hits out of order, and you can get some interesting places. I'm glad I wanted to see that, of course. Before Christmas, actually, yeah. Before Christmas, what a great Christmas gift that would make for friends, family, and loved ones. Oh, yes, exactly. And support Sonic Pi was dramatic. We didn't talk about it at all, but it is important to to quickly mention that that
01:42:19
Speaker
Being an independent open-source developer is actually quite a tricky financial space to be in. oh god yeah yeah and It's a shame, because I think that there are a lot of really interesting ah software systems that could be built that are in people's brains right now that are trapped, because they're currently doing some sort of boring corporate job.
01:42:38
Speaker
If somehow they could be freed from that corporate job and say, here's some finances to build the thing in your dreams, we'd have so much cool stuff in the world um that we need to find ways to to make that more possible. so yeah i think I think in general society, we we have a problem of how do we fund open source software? and Specifically, and because that's always an interesting question, how do you fund open source software which doesn't have a revenue as part of its its its goal?
01:43:06
Speaker
it doesn't Because I think it's a whole set of interesting pieces of software that could be created that aren't supporting business needs. but are supporting human creativity or expression or just joy and fun you know that don't actually sell things, but are are just there to be fun. and i think that that they're They're the hardest piece of software to fund because yeah they don't have their own revenue model to to support them or a business need that says, if we don't fund this open-source software, we can't deliver our business goals, so we're going to have to siphon some of our profits to
01:43:39
Speaker
keep this going or we'll put some of our developers on it. yeah so i think that' I think that's an interesting problem. so I'm using the courses currently and Patreon, which is a way people can just donate money if they like what I'm doing to support the work I'm doing. I feel like there's this is a real interesting problem to solve that if we can solve it, there's going to be a whole ton of exciting pieces of software in people's minds that can be made into reality. What is your sonic pie?
01:44:03
Speaker
Yeah, I think we absolutely have to do that as a society. and It doesn't have to be so much money. It doesn't have to be like $100 million dollars of venture capital funding. ah but for one For one developer, yeah, exactly. that's stuff i mean I'm able to somehow... It is ah it is a disaster, really, but but it's ah it's kind of possible to survive on my own from donations. but The course is now, I've just made they're really popping me up. But also, I do gigs and performances, so if a company wants me to do a party, I can do a Christmas party, I can do a corporate event, I can do a speech, I can do a keynote, I can do a workshop. I've seen you play the afterparty of three tech conferences.
Financial sustainability of open-source projects
01:44:42
Speaker
How was that? Absolutely fantastic. there we go You're the best tech. I don't know if you want to be limited to just tech conferences, but you are the best tech afterparty going.
01:44:52
Speaker
well That's kind. Tech parties are good because actually the audience has an inside of what I'm doing. and Because the code is written for schools, it means that people who who can program, but haven't ever seen the language I'm using, still can actually participate and get involved and and read the code that I'm projecting, which is a really nice best thing to be able to do. so Tech audiences, have ah it's a bit more of an easy win. I have done a gig for a corporate environment where some people came out up to me and asked me, yeah can you play the spy skills?
01:45:22
Speaker
like No, I can do some more drum and bass, or I can do some cyberpunk, industrial beat sounds, but I don't do pop music. So, yeah, that's a clear misunderstanding that I'm not a DJ that has some decks that they can put whatever they want on. Oh, God. Yeah. Yeah. So, education needed all round. Absolutely. Yeah.
01:45:44
Speaker
so you need to give me Before we go, you need to wrap up and give me a special undiscount code where people can sign up for your course and pay slightly more, because the average programmer can afford to support you. I've made them really cheap, actually, and I think that's because I want more people to be able to do this. and to honest that's That's the exciting thing, to see the world learn to understand that code is not just a tool for business, it's a creative force that's that's yet to be really realised.
01:46:10
Speaker
and that music is one way of of exploring that. and The more people we can get coding music, the better, I think, for society, just because it's just more people expressing themselves and more people having fun. yeah um and There's obviously lots of other things you can do to express yourself and have fun, but I think coding music is a really interesting one. um yeah so Making the course as cheap is is a way to try and increase our audience and and to get more people having fun.
01:46:34
Speaker
But yeah, if you do want to pay more, they're just join me on Patreon if you like mike what I'm doing. Or if you're a corporate entity that wants to sponsor, you want your corporate logo on some things, i I'm happy to talk to you. But but in general, um my goal is to try and make sure Sonic Pi and all the work, and this new thing I'm talking about, this Erlang thing,
01:46:51
Speaker
called Tower 5, to make sure they're always free, they're always open source, that that that you don't have to be rich or wealthy to be able to to get started and explore programming. You don't have to be super smart, you don't have to have terms of background knowledge, you can just get started with the simplest, cheapest computer.
01:47:08
Speaker
um and install something really easily and have fun. and then If you want to do fantasy, you want all that equipment I got behind me, sure, that's cost some money, although actually one of them was was a gift from Moog. but come but um But yeah, you don't need any of this stuff. right You just need a Raspberry Pi, a cheap computer and a TV, and you can get started coding your own music.
01:47:28
Speaker
and But in order to make that possible, to keep that free, I need somehow to to to pay my bills. and But luckily, that's not the same bills as a large corporate entity, it's just an individual developer. yeah yeah we need You can keep feeding ourselves. We need to feed you food, something like that. By the course. yeah by the course And if anyone from Coca-Cola is listening, Sam and an advert.
01:47:51
Speaker
Hire me for yeah if feel for your tech event. I'm pretty good, actually. I played at the Royal Albert Hall. That was amazing. With a quartet of of of four young kids, alongside it's like a whole room of Albert Hall.
01:48:09
Speaker
of kids performing. It was super fun. So lots of traditional instruments like violins and cellos and guitar bands and drumming circles, and then this quintet of live coders and the teacher and me. It was great fun. That's mad. And you've been reviewed by Rolling Stone as well, right? Oh yeah, that was true. I was at the Moogfest. That was like 2017 or something, yeah. That's quality. I transcended the present, along with Grimes. Before this turns into the Sam Aaron fan club, which I could happily join, I'll say thanks very much for joining me and taking us through it, and I wish you best of luck with the future courses. Thank you so much. It's been a pleasure talking to you. Really a lot of fun. Always. Cheers, Sam. Bye.
01:48:51
Speaker
Thank you, Sam. If you want to get hold of Sam's software, Sonic Pi, you can download it for free. I'll put a link in the show notes. There's a tutorial built into the software, but if you want to support Sam and get him to teach you step by step, there's a link to Sam's course in the show notes too. I've bought a copy and I'm going to happily go through it over the holidays. I don't expect to play this year's New Year's party, but who knows in the future.
01:49:17
Speaker
And lastly, Sam's contact details are in the show notes as well. Should you happen to be organizing a tech conference and want a really great after party? Before you go and start coding up your first album, if you've enjoyed this episode, please leave a like, maybe rate it, maybe share it with a friend or a social network, and make sure you're subscribed because we're back soon with another less musical but equally harmonious developer's voice.
01:49:45
Speaker
Until then, I've been your host, Chris Jenkins. This has been Developer Voices with Sam Aaron. Thanks for listening. okay