[Interview] Roger B. Dannenberg, USA (co-creator of Audacity, author of the Nyquist programming language, pioneer of computer accompaniment and the SmartMusic system, professor emeritus at Carnegie Mellon University)

Roger B. Dannenberg is an American computer scientist and musician, and Professor Emeritus of Computer Science, Art and Music at Carnegie Mellon University. In 1999 he started Audacity together with his then graduate student Dominic Mazzoni, as a tool for visualizing audio analysis algorithms; it became one of the most widely used free audio editors in the world. He is also the author of Nyquist, a programming language for sound synthesis and music composition that Audacity uses by default. And Nyquist now includes an integrated development environment so the developer community is growing. His work in computer accompaniment led to three patents and to the SmartMusic system, used by over one hundred thousand music students. He also developed the Piano Tutor, a multimedia expert system for teaching beginners to play the piano.

Audacity began as a research tool to visualize audio analysis algorithms. What exactly did you and your student need that no existing software could do?

There were a couple of things we needed that did not exist at the time. First, we wanted a system that was open so we could extend it to display various kinds of analyses such as pitch estimates or music transcriptions. Second, we wanted to be able to work with files much larger than available RAM in personal computers. Third, given the speed of computers at that time, and having used other editors that were horribly slow at simple operations like trimming silence from the beginning of a big file, we thought it would be possible to build in simple editing operations that ran much faster.

How did the work split between you and Dominic, and when did you realize it could become a product for everyone?

We both talked about the design and I think we independently came up with the basic data structure. Dominic wrote the main program and the waveform display, and I contributed the original audio file and audio I/O library. I also wrote the code to display notes and other graphics. Dominic had a strong desire to make Audacity available and led the effort to make it easy-to-use, reliable, and available, especially for the new powerPC-based Macintosh computers. Since they introduced a new processor architecture to the Mac, older audio software did not run well, leaving a big opening for Audacity.

Audacity was first called CMU Visual Audio and was released to the public in May 2000. What did the first public version look like, and what could it do?

I think there is a screenshot somewhere. As I recall, the first version looked a lot like Audacity up to 2020, and at least had select/cut/copy/paste/undo/redo operations, some analysis functions, a spectral plot, and ability to read/write audio in different formats.

Audacity is now probably the most downloaded audio editor worldwide. What was it like to see a research project expand so much and become a ‘go-to’ editor for everyone?

Yes, I’m pretty sure it is the most-used audio editor of all time. I’ve worked on, released, and maintained various software systems, and it takes real effort to keep anything running. From my perspective, the greatest thing about Audacity is that Dominic pushed it until it had critical mass and became self-sustaining. I cannot take much credit for that; I think a few things just came together in the right way at the right time.

Can you explain to the rest of us what Nyquist is, and how it changed a composer’s and sound designer’s approach to sound?

Nyquist is a programming language for sound. It is unusual in that one of the built-in data types is a sound, so you can treat Nyquist like a calculator, except that instead of a+b adding two numbers, a+b can mix two sounds of arbitrary duration. It seems like this could take huge amounts of memory to store all the variables and intermediate results, but Nyquist uses some tricks that make it memory efficient and even faster than other synthesis languages.

Nyquist also acts as a scripting language for Audacity effects. What have people built that surprised you, and have any of those plugins made within Nyquist evolved into a standalone brand? 

To be honest, I’m a little disappointed the community has not done much more with Nyquist. In retrospect, I should have worked to extend Nyquist in Audacity to allow it to read and write multiple tracks, plot analysis data, and create user interfaces. Paul Licameli implemented a tool to clean up speech audio using Nyquist, which got some traction.

What is the competitive landscape for Audacity plugins? Steinberg tries to get in with VST3, LV2, and probably LADSPA which seems a bit slow on development, which one is leading? And does any of those include a scripting language like your Nyquist?

It seems to me that developers often target multiple standards, and DAWs (especially Audacity) can support multiple standards, so I’m not sure there’s much competition. The closest thing I know of like a scripting language is FAUST, a functional language for DSP, that can be compiled directly to various plugin standards. I think there is an opportunity for a Nyquist-inspired scripting language for plugins. Nyquist is not directly suitable for real-time low-latency effects, which is what most plugins provide.

Your ‘live accompaniment’ software lets a computer follow a live performer and play along. How do you sync time, like a ‘tempo track’, how does the computer sync to the human who performs freely?

What I call “computer accompaniment” assumes that musicians are playing from a score and the computer part is also written out in some kind of score, which these days usually means a MIDI file. By following along in the score, you can estimate the position and tempo of the live player and use that to synchronize electronic parts. Some challenges are dealing with mistakes that always happen, dealing with polyphony, which poses a tricky matching problem, and synchronizing in a musical way, which requires the computer not only to anticipate when to play but also to balance rapid adjustments to stay synchronized with smooth adjustments to maintain continuity and musicality.

Your own research led to three patents and to the SmartMusic system. How did a research idea become a product used by over one hundred thousand students?

This was a fortunate blend of computer science on my part and entrepreneurship by John Paulson, who founded MakeMusic. John independently invented the concept of computer accompaniment and built a whole business plan around it, not knowing that I had already invented it as well, figured out how to do it, and patented the method. Instead of feeling scooped, John was delighted he didn’t have to raise money just to develop a technology he did not know existed, and I was delighted to find someone interested in using the patent for a great product.

Was there at any time intention to enforce those patents or was it more like proof of ‘date’, like a ‘status quo’ thing. Is any licensing actually going on or are you totally in the Elon Musk’s “open source patents” mood?

Patent protection was created to promote innovation and industry. Why would a startup like MakeMusic invest a small fortune to introduce a whole new product category and spend millions to show people what it can do if anyone else could then enter the market with no restrictions and no incentive to compensate MakeMusic for creating the market in the first place? I’d be the first to agree that patents can be misused and a lot of patents are just a way to block competition. 

Also you developed the Piano Tutor to teach beginners. What does a computer teacher do better than a human one, and what does it do worse?

The surprising thing that I saw was that Piano Tutor could be very strict and critical. If a human teacher treated a young student like that, they might leave in tears, but when a student hears it from a computer, they accept it and rise to the challenge, feeling really good about themselves when they finally pass a lesson. Also, Piano Tutor can be there always, whereas students with human teachers get no instruction between lessons, which really limits their rate of progress.

You work in computer science, art, and music at the same time. Which of those three do you think a student should learn first if they want to build audio software?

I think it’s important to master the fundamentals, including musicianship, music theory, math, and computer science, and it made sense to me to pursue everything all the time. In college, I majored in electrical engineering but I played in bands, took music theory and history, and took private trumpet lessons and practiced every day. I tell young students that are pushed to do research to focus more on fundamentals of computer science. Now that a lot of programming can be done with AI, I’m really not sure what to advise, but my feeling is that actually getting into the details and not handing them off to AI is important. 

Audacity is open source and has had many volunteer contributors over the years. How do you keep a project like that coherent for more than two decades? How difficult is it to run forums, support groups, manage bug reports, feature requests? 

We had a great team, which is largely due to Dominic’s leadership. One thing that held it together is that Dominic really was a benevolent dictator. He insisted on keeping the interface simple, established release policies that included enough testing to avoid getting swamped with bug reports, and generally had the respect and cooperation of everyone involved.

Many audio tools are now cloud-based and AI-assisted. Where do you see the line between a tool that helps a musician and a tool that plays instead of them?

I’m all for exploring all possibilities. There’s no right and wrong here. The phonograph changed music forever, destroyed jobs and also created jobs and great joy for many people. I do not think any AI applied to music is good or bad any more than the phonograph was.

Where can people follow your latest work, publications and software?

I have a website at www.cs.cmu.edu/~rbd, and my work with students is published regularly in conferences and journals. We try to put everything online. Nyquist is still on SourceForge, but a lot of other projects are findable on GitHub. I have a new book, Introduction to Computer Music, with information at compmus.com, and finally I’m collaborating with the Soundcool projects (soundcool.org). One of our projects is the opera Felicità, featuring live computer accompaniment throughout: youtu.be/pygC11IcCb0?si=3waGMUKsYx00VKH5.