An introduction to Linux audio plugin APIs
An introduction to Linux audio plugin APIs [LWN.net]


The world of music and audio production is largely dominated by
proprietary software vendors. Among them, Steinberg stands out as a company
that created some of the most-used software, including the Cubase and Nuendo digital audio
workstations. Steinberg is also known as the creator of the VST plugin API
that, largely due to its licensing policy, has irritated developers enough to
inspire multiple attempts at creating an open-source alternative. Even now,
when the VST3 SDK is available under the
GPLv3 license, the way the company exercises its control over the SDK
keeps pushing developers away toward other open-source solutions.


$ sudo subscribe today

Subscribe today and elevate your LWN privileges. You’ll have
access to all of LWN’s high-quality articles as soon as they’re
published, and help support LWN in the process.  Act now and you can start with a free trial subscription.

This is an introduction to open-source plugin
APIs for musicians and sound engineers alike. It focuses on the options in
the larger ecosystem and how their shortcomings led to the creation of new
alternatives with liberal licensing.

What audio and MIDI plugins do

Most audio editors, MIDI sequencers, and digital audio workstations
(DAWs) are extensible. Sometimes that means writing a script that accesses
the public API of a program (e.g. Lua in Ardour, ReaScript in Reaper). But the  
most common way is to write an extension, or plugin, using an API that is
usually not specific to one single program. The audio editor or the DAW
then would act as a "host" for that plugin, running it either in its own
process or as a separate process (an increasingly popular option).

Two of the most common use cases for plugins in DAWs are processing audio
(equalizers, reverbs, etc.), which means there's audio input and audio
output, and turning notes into music, which means there's MIDI input and
audio output.
Most plugin APIs allow both audio and MIDI input and output.
Additionally, some plugins exist for the purpose of controlling external
gear, such as an audio interface with on-board digital signal processing
like a built-in reverberation engine. These plugins are typically made by
hardware vendors themselves, though there are some rare exceptions.

Unlike plugins in many image editors, audio and MIDI plugins are usually
designed to work non-destructively and run in realtime mode. That
means that you don't need to render a processed audio track to
listen to it.  Instead, you can hook up a MIDI keyboard and play a virtual
instrument 
without ever recording a note, and you can change the value of plugin parameters 
over time.

A quick overview of current plugin APIs

These plugin APIs are currently in wide use:



 Virtual Studio
Technology, or VST2/VST3, by Steinberg.  It is supported on Windows,
macOS, and Linux, and has broad 
host/DAW support.

 Real Time
AudioSuite, or RTAS, by Digidesign/Avid (Windows, macOS, only supported
in Avid Pro Tools).

 Avid Audio eXtension, or AAX,
by Avid (Windows, macOS, only supported in Avid Pro Tools).

 Audio
Unit, or AU, by Apple (macOS, iOS/iPadOS).

 LV2, which is an open-source
project (Linux, Windows, macOS).


Industry-wide, plugins are mostly available
as VST2/VST3 for both Windows and macOS and as AU for
macOS. In some cases, plugins are also available as AAX and RTAS to target
Avid Pro Tools users, because Avid infamously refuses to support any APIs
other than its own.

On Linux specifically, LADSPA was the first widely used API for audio
plugins; DSSI (Disposable Soft Synth Interface) was developed on top of
it by the same team to make software synthesizers and samplers available as
plugins too. There are several hundred LADSPA plugins available, but fewer
than 50 DSSI plugins. Most audio editors and DAWs on Linux support LADSPA;
few support DSSI. Both LADSPA and DSSI have now been largely replaced
by LV2, which allows both audio and MIDI processing. There are well over a
thousand LV2 plugins out there today.

There is also a growing number of plugins — both proprietary and free —
written for the VST API and built natively on Linux, largely thanks
to the VST3 SDK being
available under the terms of both GPLv3 and a proprietary license. In fact,
some spectacular free plugins, such as the Dexed  and OB-Xd synthesizers, are only
available as VST3 on Linux.

One final API worth mentioning here is the Generalized
Music Plugin Interface, or GMPI. The original specification was
designed between 2003 and 2005 by a diverse group of developers of both
proprietary (SONAR,
SynthEdit, FL Studio, etc.) and free (Ardour,
etc.)
software. Despite their varying backgrounds, these developers all shared
the same sentiment: everyone felt fed up with Steinberg's dominance in the
industry, which we will talk about in more detail below.

The GMPI specification was turned into an unofficial SDK by Tim Hockin
(Google, but acting on his own behalf) and Jeff McClintock
(SynthEdit), with Koen Tanghe (engineer at Ghent University at the time)
working on his own version. The code by Hockin and McClintock did not
receive much peer review, despite many requests. Eventually, the
working group dissipated and McClintock used a cut-down version of the
SDK for the third iteration of his SynthEdit plugin
API. The fourth iteration will have a refactored version of the GMPI
SDK.

I've heard a few explanations for what happened. The story from
McClintock, shared in a private conversation, is the closest match to the
discussions in the public mailing list. His impression is that many
stakeholders only wanted a kind of VST2 with liberal licensing and weren't
interested in major changes and improvements, so there was no incentive for
SDK review.  Another view, expressed by Paul Davis (Ardour), is that
the MIDI Manufacturers Association, which announced the initiative in the
first place, wanted the implementation phase to be closed.
The Association reportedly also could not guarantee licensing that
everyone would be happy with. So by the time an actual official SDK would
have become available, many stakeholders would likely have lost their
interest, and stakeholders likely weren't interested in an unofficial SDK.

I'm mentioning GMPI for three reasons. First of all, it was the
first time in history that developers of proprietary and free
software teamed up to develop an open-source audio/MIDI plugin
API. Secondly, while largely unsuccessful, GMPI affected the design of both
VST3 and LV2 (here is more info on
the latter). And finally, McClintock resumed working on the full
GMPI SDK last December. The SDK is available under the terms of the ISC 
license.

Where existing plugin APIs fall short

First off, if you want to develop an audio/MIDI plugin for use on Linux,
your options are limited as compared to Windows and macOS.

AU (both v2 and v3) is not available anywhere but macOS and
iOS/iPadOS. AAX and 
RTAS are cross-platform, but only work in Avid Pro Tools, which is unavailable on
Linux.  So far, SynthEdit appears to be pretty much the only real software
that makes some use of GMPI. That leaves us VST2, VST3, LADSPA, DSSI, and
LV2. These APIs are all supported by host applications on Linux, free and
proprietary alike, although only two hosts, Qtractor and
Zrythm, support all five.

Right off the bat, VST2 is at the end of its life. If you have a license for the VST2
SDK for an existing plugin, you can make a new version of that plugin as
long as you want to. But Steinberg won't license the SDK for new
plugins, and the need to license the use of this SDK was a major reason why
VST2 never took off on Linux anyway. The limited number of native VST2
plugins that did appear for Linux were only possible thanks to a clean-room
reverse-engineered VeSTige
header file made by some LMMS developers.

LADSPA and DSSI are simple APIs, and the majority of audio hosts on
Linux support LADSPA. However, these APIs have a number of technical
limitations, they are not well maintained, and pretty much everyone
stopped making new LADSPA and DSSI plugins years ago.
Thus, we are down to VST3 and LV2.

VST3 has a specification that matches contemporary expectations and
allows the creation of capable plugins. It's also extensible, though
VST3 extensions are scattered across third-party vendors. However, over the
years, Steinberg has managed to irritate developers of both proprietary and
FOSS plugins in a major way.
The way the company is sunsetting VST2 has upset plugin developers
because it's impossible to use the old technology to create new work for
hosts that still support VST2. Beyond that, Steinberg's own hosts and plugins
will cease
supporting VST2 by January 2024, so any Cubase and Nuendo projects that
users have will fail to load entirely. Steinberg took some steps to make a
VST2-to-VST3 transition possible but, reportedly,
neither designed it well nor made developers aware of this feature in a
timely manner, resulting in mistakes that cannot be easily undone now.

And then there are the inevitable technical controversies. The VST3 SDK
is huge and has a number of hard-to-swallow idiosyncrasies like the
enforced use of UTF-16. In particular, VST3 has an entirely new way of
dealing with incoming MIDI data through a convenient abstraction layer that
makes a lot of sense but which demands rearchitecting existing plugins,
with no fallback to the old way. This complicates the task of maintaining
both VST2 and VST3 versions of a plugin.
To get a sense of how much work it takes to develop and maintain a fully
functional VST3 version of a fairly complex plugin, you can read this
comment on KVR by Urs Heckmann.

In summary, the industry is tired of living at the mercy of
Steinberg. GMPI was the first major attempt to change the balance of
forces, but, as described above, it wasn't successful.

LV2 was built in the mid-2000s using many ideas from the GMPI
specification. It's modern, extensible, and allows all sorts of wonderful
things. However, although well over 1,000 LV2 plugins are now available
(here is a
fairly recent, yet not exhaustive, list), there is a general sentiment
that LV2 is difficult to get started with.
Since its inception, I have observed a number of irritations that
developers have with this API. Some of the issues are purely technical in
nature.  For example, plugin metadata is stored separately in text files
using the Turtle syntax. For
complex plugins, those files tend to become large. To give you an idea, the
compiled metadata file for Mix
Maxtrix
has nearly 32K lines, although that's an extreme example. Other developers
cite too much flexibility being the major issue they have with the API.

Then there are organizational issues like the lack of governance body
and "too many cooks in the kitchen", which results in plugin developers
feeling like they aren't being heard. The feeling that the LV2 maintainers
are too defensive about their design decisions and don't like revisiting
them does not help either.
On an even larger scale, end users want an
ecosystem, and LV2 has never established itself as such. That is, however,
a very broad claim that, alone, is worth a separate conversation starting
with how one defines an ecosystem for a plugin API.

There are more reasons for this API's slow adoption. One is that LV2 is
not very popular with host developers outside of the free-software realm. The
lack of LV2 support in hosts substantially limits the exposure of available
plugins to end-users. Reaper is pretty much the only major DAW (on a global
scale) that supports it, but then again Reaper supports almost every plugin
API out there.

The other problem is that LV2 is currently unsupported by the major
plugin-development frameworks; these allow writing code once and then
building it for multiple plugin APIs. There's a ten-year-old fork of JUCE, one of those frameworks, by falkTX that
adds native LV2 support; the upstream project is reimplementing
this just now, with no planned completion date announced. Also, a 
third-party effort
to add LV2 support to the iPlug2
framework was never merged into the mainstream project.

All of these factors affect the adoption of LV2. Thus we see, for
example, OB-Xd developer closing the request for an
LV2 version because JUCE6 has VST3 support and doesn't have LV2
support. It's difficult enough to get plugin developers to generate a
native VST3 plugin even when they use JUCE, it's almost impossible to get them to
release an LV2 version.

I have definitely seen most of the points mentioned here refuted by
developers heavily invested with LV2. However even a naive observation
suggests that LV2 is neither popular nor known outside of the FOSS ecosystem.

In conclusion

There are two open-source audio/MIDI plugin APIs in active use today:
VST3 and LV2. Both APIs provide the means to create sophisticated
plugins but, from the standpoint of a number of developers, both are
somewhat problematic for technical reasons, as well as for how these APIs
are managed by their respective maintainers.

For VST3, developers have to deal with Steinberg's aggressive licensing
policy. This essentially means never building new VST2 plugins. For
developers of virtual instruments, it means either dropping support for
existing VST2 plugins entirely or accepting the burden of maintaining two
architecturally different code bases. On top of that, various other design
decision in the API have met resistance.

In case of LV2, developers have a steep learning curve and
little choice in case they want to develop a plugin for multiple APIs from
one code base. They would also be limited to just open-source host
applications plus one proprietary digital audio workstation (Reaper), which
significantly limits their exposure to users unless they support multiple
APIs.

This situation has persisted for years now. In 2015, it resulted in the
birth of another open-source API, CLAP. This project was
resurrected late last year and is now finding its way to both free
and proprietary software. We will talk about it in more detail in a
followup article.

[Many thanks to Paul Davis, Robin Gareus, Filipe "falktx" Coelho, William
Light, Alexandre Bique, Jeff McClintock for consulting on various technical
aspects of audio/MIDI plugin APIs.]
           Index entries for this article
           GuestArticlesProkoudine, Alexandre
            

            