How I Reverse-Engineered a Proprietary Audio Protocol

When I decided to build Linux support for the Universal Audio Apollo, I had no source code, no documentation, and no cooperation from the manufacturer. What I did have was a working macOS driver (UAD2System.kext), a set of reverse engineering tools, and a systematic methodology.

This is the story of how I went from zero knowledge of the Apollo's internal protocols to a working Linux kernel driver with full duplex audio.

The Starting Point

The Apollo is a Thunderbolt audio interface with an FPGA, four SHARC DSPs, and a complex software stack. On macOS, it's controlled by a kernel extension (kext) that handles PCIe communication, and a userspace application (UA Console) that manages mixing, routing, and plugin processing.

My goal: replicate enough of this stack to get professional-quality audio I/O working on Linux.

Phase 1: Static Analysis

The first lucky break was discovering that UAD2System.kext shipped with unstripped symbols. This meant I could see function names, struct layouts, and parameter types — a goldmine for understanding the driver's architecture.

Using a combination of Ghidra and manual analysis, I mapped out:

  • 20+ ioctl selectors with their input/output structures
  • Register maps for the PCIe BAR regions
  • DMA buffer layouts including scatter-gather table structures
  • Firmware loading sequences with block sizes and addresses

This gave me the "what" — what registers exist, what ioctls do, what structures look like. But not the "when" or "how" — the timing, sequencing, and state machine that makes it all work.

Phase 2: Dynamic Analysis with DTrace

DTrace on macOS let me instrument the running kext without modifying it. I wrote 40+ DTrace scripts to capture:

  • Register access patterns — every read and write to the PCIe BAR, with timestamps
  • ioctl sequences — the exact order of operations during device initialization, audio streaming, and teardown
  • DMA buffer operations — how the driver sets up, fills, and processes audio buffers
  • Interrupt handling — which interrupts fire and what the driver does in response

The DTrace captures were massive — hundreds of megabytes of timestamped register operations. But they told the story of how the driver actually works: the initialization sequence, the audio streaming loop, the shutdown procedure.

Phase 3: Protocol Decoding

Beyond the kernel driver, I needed to understand two TCP protocols:

StateTree (port 4710) — A CamelCase JSON-based protocol carrying the complete device state tree. On an Apollo x4, this tree has 11,244 controls covering every parameter: preamp gains, phantom power, routing matrices, monitor mixes, plugin parameters.

HelperTree (port 4720) — A binary UBJSON protocol used by UAD Console for DSP mixer session data. This 803-node tree carries the supercore device session — plugin banks, bus configurations, metering, MIDI, and track organization — using snake_case properties and array-based children, a completely different data model from StateTree.

Decoding these protocols required capturing live sessions between UA Console and the Mixer Helper daemon, then correlating user actions with protocol messages. Each knob turn, fader move, or routing change produced a specific protocol message that I could map back to the state tree.

Phase 4: Implementation

With the protocols decoded and the register sequences captured, implementation was methodical:

  1. Kernel module — PCIe probe, register access functions, DMA buffer allocation, interrupt handlers, ALSA PCM registration
  2. Firmware loading — 15 blocks, 169KB total, loaded via Linux's request_firmware() API
  3. DSP initialization — The "ACEFACE" handshake sequence that brings the DSPs online
  4. Audio streaming — Setting up DMA buffers, starting the transport, handling interrupts for buffer completion
  5. Mixer controls — 50 ALSA controls for preamp gain, phantom power, routing, monitoring

Each step was verified against the DTrace captures: if my Linux driver produced the same register access pattern as the macOS kext, I knew I was on the right track.

The Methodology

The key insight is that reverse engineering is not about cleverness — it's about methodology. The process is:

  1. Capture — Record everything the existing system does, with as much context as possible
  2. Decode — Find patterns in the captures, correlate actions with data
  3. Implement — Write code that produces the same patterns
  4. Verify — Compare your implementation's behavior against the original captures

This cycle repeats at every level: register access, protocol messages, audio streaming, device initialization. Each layer builds on the understanding gained from the layers below.

Results

The Apollo-Linux driver now supports:

  • Full duplex audio at all professional sample rates (44.1 kHz through 192 kHz)
  • 24-channel playback and 22-channel capture simultaneously
  • Linux-native DSP cold boot without macOS
  • 50 ALSA mixer controls
  • Complete dual-protocol implementation

The project produced 130+ reverse engineering tools that can be applied to similar challenges.

Lessons for Software Engineering

Reverse engineering taught me skills that apply far beyond driver development:

Read before you write. Understanding the existing system thoroughly before touching code prevents wasted effort and wrong assumptions.

Capture everything. Comprehensive logging and tracing pays for itself. You never know which detail will be the key to understanding a behavior.

Systematic beats clever. A methodical approach that covers all cases beats a brilliant insight that only works sometimes.

Verify at every step. Don't build a tower of assumptions. Verify each layer independently before building on top of it.

These principles guide everything I build now — from web applications to infrastructure automation.