[Interview] Victor Lazzarini (core developer of Csound, Professor of Music at Maynooth University)

Victor Lazzarini (core developer of Csound, Professor of Music at Maynooth University)
Victor Lazzarini (core developer of Csound, Professor of Music at Maynooth University)

Victor Lazzarini is a Brazilian-Irish computer music researcher and software developer, Professor of Music at Maynooth University and Director of the Arts and Humanities Institute. He is one of the leading developers of Csound, alongside John Peter Fitch (ffitch) and Steven Yi. He is also the author of the Sound Object (SndObj) Library, an early object-oriented C++ framework for audio synthesis and processing, followed by the more modern AuLib and Aurora musical signal process libraries, amongst other work. He participated in the development of the Mobile Csound Platform, a group of projects that brings Csound’s API to iOS and Android, the development of Csound for the Web (leading to Csound WASM), BelaCsound, which brought Csound to the Bela embedded audio platform in 2018, and more recently Bare-metal Csound. His research includes adaptive frequency modulation, various forms of nonlinear distortion synthesis and spectral audio processing. He has written several books in the area, including “Spectral Music Design” (Oxford Univ. Press), and is the co-author of two volumes of  “Csound: A Sound and Music Computing System” (Springer).

Table of Contents

Your Sound Object Library began in the late 1990s as an object-oriented C++ framework for audio. How did you structure its classes so that synthesis, processing and I/O could be combined freely, and what from that design still shows in your later work?

When I began working on this in 1997, C++ was a fairly new language with only a handful of people using it for audio. It seemed right away to me that it would be suitable for creating reusable blocks of code for sound and music applications. By my reckoning, the SndObj library in 1998 was maybe one of two such libraries being developed in the FLOSS space (the other one being STK, about which I only heard years later). My use of C++ was very much a C + classes approach, and I wanted to mostly apply it as a framework rather than a toolkit (the difference being that the latter is used mainly directly by instantiation, and the latter by subclassing and extending it). This was inspired by the early application frameworks that were being developed at the time. I continued to develop it until the mid 2000s, when I mostly turned my attention to Csound. The C++ language developed significantly in the next two decades, so when I came back to play with frameworks I decided I had to start from scratch, and I did this with AuLib in 2017. By then, C++ libraries for audio were a very common thing, and this is a well trodden path now. Later in 2021 I wrote a minimalistic lib called Aurora mainly as a helper for a friend’s plugin toolkit project. This concentrates on generic programming, and is designed as a header-only toolkit. The other noteworthy C++ exploit is the Csound Plugin Opcode Framework (CPOF), which I introduced in 2018 as a very lightweight means to support the development of Csound opcodes using C++.

PySndObj brought the library into Python for interactive scripting. What did on-the-fly synthesis programming need from the library underneath, and how close to real time could it get?

I think PySndObj was a good vindication of the original SndObj design as it allowed patching of objects very straightforward to create Python-based audio processing scripts. Together, SndObj and Python could be used as a small audio programming domain-specific language. As time passed other projects came to use these ideas and now the space of audio programming in Python is saturated by various offerings. 

You extended Csound 5 with its streaming frequency-domain signal framework, the fsig system. How does it carry spectral data between opcodes, and what kinds of spectral effects became easier to write because of it?

The framework itself, and fsigs, were already there in the later versions of Csound 4, designed and implemented by Richard Dobson. What I did was to bring it into action more extensively with a great variety of new opcodes that took advantage of it. I also extended it (the design permitted extensions, that was built into the framework) to handle another form of signal, which handled spectral tracks of amplitude, frequency, and phase as opposed to amplitude-frequency bins. This allowed other forms of spectral processing in addition to the phase vocoder options.

Csound began as a MUSIC N descendant and is now used through its API from Python, Android, iOS, the web and embedded boards. What had to change in the architecture to make that possible?

The main change was the transition made in Csound 5 from a monolithic application to a reentrant programming library. That was the game changer: it allowed Csound to be embedded in other hosts written in C, C++, Java, Python, Lisp, etc. From there we could make things like Pd objects that run a complete Csound engine inside that environment, mobile applications, then web applications, and so on. With Csound 5 we began the transition from a classic MUSIC N language to the modern Sound and Music processing system (which we see today). Along the way, we introduced a new parser (twice actually in Csound 6 and 7), modernising the language, a new on-the-fly compiler, hard realtime modes (e.g. to work in contexts such as the Bela), and many other innovations. The fact that we can run it absolutely anywhere from a meagre esp32 board to a supercomputer is a vindication of this. Also, it is notable how Csound is inserted in the modern commercial audio programming workflow via VST/AU plugins, applications that run in car audio systems, and embedded in Coke bottles; the majority of its users probably have no idea they are running Csound.

With the introduction of WebAssembly, how did the possibilities and challenges for running Csound in a browser change, particularly in terms of performance and real-time audio processing?

We’ve actually started running Csound on browsers in the very early days of Web Audio. I remember asking the question around 2013 and no one had a clear answer. I then found out about Google’s Native Clients and started investigating it. These originally were OS-dependent but later they released a portable version, PNaCl. This used a just-in-time compiler that took our frontend code linked to the Csound library (also built for this environment). This allowed us low latency and good performance but was limited to Chromium-based browsers. Then we added a second version using Emscripten instead, transpiling the Csound lib into JS, with the ScriptProcessorNode from Web Audio. Performance was not great and it ran on the main browser thread, but it could be used with other browsers supporting Web Audio. Next, Web Audio introduced the Audio Worker, which had its dedicated thread and we got this working so audio was not interrupted by actions on the main thread. This was a pure-JS approach again; Web Assembly (WASM) was introduced later, and Google dropped PNaCl. We then combined both technologies, WASM and Audio Workers, to get a single replacement for the previous efforts. This then evolved through a number of iterations into Csound WASM, which is a fairly complex and capable implementation of Csound to run on browsers (it can also run on node.js, but maybe there are fewer use cases for it). It runs very well, not at native performance, but enough for us to use it widely. The Web IDE came out of that and it’s a really useful thing (I used it in class since around 2021 until I stopped teaching last year), it runs everywhere you can find a browser with WASM support (most of them have it now). The WASM technology is quite advanced and I can see it improving continuously in the future.

The Mobile Csound Platform brought Csound to iOS and Android. What were the main technical problems in real-time audio on mobile operating systems, and how did you handle latency?

Back then mobile operating systems were very limited, not supporting things like dynamic libraries, and in the case of Android having poor support for realtime audio. Since then, things have improved significantly and our mobile deployments are much more powerful and capable now. Also we have to note that with the browser platform becoming very well developed, it is sometimes easier just to write JS apps that will run everywhere, including in mobile phones.

BelaCsound brought Csound to the Bela platform, with a first release in July 2018. How does Csound run inside Bela’s hard real-time environment, and how do the analog pins and sensors connect to Csound’s control signals?

For Bela we needed to harden the internals of Csound to meet the demands of the operating system platform. So we developed the hard realtime mode that guarantees basically that no blocking operation will come in the way of performance. Alongside this we did all the usual additions to support access to the GPIOs and peripherals. We should note that this work followed our experiences with the Galileo and RPi started in 2013 (I think) so we had a good grasp of how to handle interconnectivity. This was simple anyway, given how Csound was a very open system. The realtime mode was probably the most specific part. In normal operation we are not too worried about occasional blocking operations, this works fine in desktop and similar contexts. It also guarantees a deterministic result, something that is not always possible in strict realtime operation. However, for Bela this is a must, so we implemented it to do what it needed.

Your paper on adaptive frequency modulation describes generating natural-synthetic spectra. What does an adaptive FM system do that classic fixed-ratio FM cannot?

It allows us to pick up an arbitrary input and use it as an FM carrier, basically modulate its frequency (phase actually). Also by picking up parameters from the input signal such as its fundamental frequency and amplitude, we can make the process adaptive. So we move from a strictly synthesis method to a more generic processing one.

The split-sideband method uses nonlinear distortion for synthesis and adaptive signal processing. Can you explain how it works to those of us who know classic FM and waveshaping, and where it fits among them?

The idea is that by rearranging the FM equation, we can extract four different outputs containing the lower, upper, even, and odd sidebands. Re-combining these four separate outputs we can obtain various different forms of dynamic spectral – for example we can use a joystick, like in wavetable synthesis, to navigate between the four outputs.

You also worked on spectral time scaling through partial tracking, using an alternative method of instantaneous frequency estimation. How does it compare with a standard phase vocoder?

The idea here is to go one further step in processing and start tracking partials that may be present in the analysed phase vocoder frames. This is done by looking at successive analyses and making a good guess at what partials may be there, in a streaming way as the data comes in.

Csound has a very large opcode library, built over decades by many contributors. How do you keep that code consistent, and how do you decide which opcodes belong in the core?

We have a system for deprecating opcodes that seem out of date or have been superseded by others. We do not remove them, but have them clearly indicated in the documentation and by compiler warnings. We have moved to be more strict in the curation of new opcodes, making sure they are correct and do not duplicate existing functionality (for instance). We also impose a strict backwards compatibility approach in that code that once ran in Csound still does and to the best of our ability and knowledge produce the same outputs.

Csound: A Sound and Music Computing System brings together the main developers and power users of the system. How was the book organised, and what part of the system was the hardest to document?

There were two aims with that book: bring attention to the new styles of programming and the new features of the Csound language; provide a comprehensive guide to the system. The earlier Csound Book from 2000, while still relevant, did not cover any of the new features of the system and language that came since Csound 5. With the new volume we have just released this year, we add to the earlier volume the new elements that were introduced since 2016 and also focus on the API aspects of the system.

You co-edited The Audio Programming Book and wrote its chapters on the DFT, the STFT, the phase vocoder, PortAudio and PortMIDI. How did you decide the order in which a programmer should meet these ideas?

It became clear to me that this was a logical order, that was the way I learned about these things. In other books, these topics seemed to be a lot more jumbled up.

You work as a researcher and developer while also running a large faculty as Dean. How do you keep time for code?

Running the Faculty took me away from research and development, but also gave me an insight on how to get people collaborating in a large setting. My term finished in 2019 and from then until this year I went back to the Music Department, resuming my role in the classroom and continuing my research. This year I moved out again, now to become the director of the Arts and Humanities Institute, which is a research-oriented role.

Many audio tools are now cloud-based and AI-assisted. Where do you see the line between a tool that helps a musician and a tool that creates in their place?

I think this seems to be a fact of reality and we need to find ways to engage with it so that we can leverage our creativity. In my own work, I do not use generative AI to make music, but I am incorporating LLMs to support development work mainly for code reviews and quality assurance. However, I can see how these tools can be used as helpers to the compositional and production process.

Where can people follow your latest work, releases and publications?

I think my (very sparse) blog github.com/vlazzarini is agood place, but also csound.com and the related github projects. Also you will find my music on your streaming platform of choice.