Audio on Charlie Chiang's blog

Building Audio Insight: An Audio Analyzer That Actually Renders Smoothly

Sun, 16 Aug 2026 16:42:45 +0800

I like audio analyzers. It answers questions that my ears alone cannot answer quickly. Where is that resonance? Is the low end actually mono? How loud is this over the whole track? Is a limiter catching an occasional peak, or working all the time?

What I do not like is an analyzer whose interface feels slower than the display it runs on.

On my M1 Max, Excite Audio VISION 4X appeared to top out at roughly 30 FPS, with inconsistent frame timing, while consuming about one CPU core. iZotope Insight 2 looked smoother, but in my experience it was comparatively resource-heavy and expensive. These were observations from my own setup, not controlled benchmarks that apply to every machine, host, and plugin version. Still, they were enough to make me wonder: how difficult would it be to build the analyzer I wanted?

That became Audio Insight, an open-source AUv2 and VST3 analyzer for macOS. Its first goal is deliberately narrow: show useful measurements, leave the audio unchanged, keep real-time callback work bounded, and make the interface feel native on a high-refresh-rate display.

The project is still early, but it already has a Spectrum, Spectrogram, Peak/RMS meter, stereo vectorscope and correlation meter, and BS.1770 loudness measurements. Four grid-snapped splitters resize the dashboard tiles, the analysis parameters are adjustable, and a built-in metrics panel makes the renderer’s behavior visible instead of leaving performance to intuition.

This post is about how it works, but mostly about the unexpectedly interesting work required to make a meter move smoothly.

What an audio plugin actually does

People who use plugins often picture them as little applications inside a DAW. That is a useful mental model for the interface, but not for the audio path.

An AU or VST3 plugin is code loaded by a host (or, in some hosts, a separate hosting service). The host repeatedly gives the plugin a small block of samples by calling its processing function. At 48 kHz with 512-sample blocks, a new block arrives about every 10.7 milliseconds. The plugin has to finish before the hardware needs the result. Missing that deadline can produce a click or dropout.

Audio Insight is a transparent effect: it observes supported mono or stereo audio and leaves the samples unchanged. Even so, its callback has to follow the same real-time rules as a compressor or synthesizer. It cannot allocate memory, take a lock, wait for another thread, write a log, open a file, call the UI, or ask the GPU to draw something. Any of those operations can take an unpredictable amount of time.

The resulting design looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


host audio callback
 ↓
bounded, non-blocking capture
 ↓
per-instance analysis coordinator
 ↓
shared two-worker analysis pool
 ↓
immutable measurement snapshots
 ↓
display-linked Metal renderer

The callback only captures bounded chunks into preallocated storage and updates the few measurements that must inspect every sample. A per-instance coordinator coalesces work, and all instances loaded in the same plugin module share two analysis workers. There is at most one running or queued job per instance, so opening many plugin windows does not create a thread for every visualization.

The workers publish immutable snapshots. The UI reads the newest complete snapshot whenever it draws; it never waits for analysis to finish. This separation matters. Analysis targets 60 slices per second, but the latest-wins scheduler can skip stale work rather than build a backlog. Meanwhile, a ProMotion display can render at around 120 Hz. The renderer can advance display motion between discrete analysis updates without running twice as many FFTs.

All five visualizations share one Metal canvas, drawable, command buffer, and render pass. JUCE supplies the plugin shell and cross-format plumbing, while the visual layer is native Metal. Coordinates and layout use logical points, and drawable and text resources follow the current backing scale. The implementation is therefore designed to support both regular-density and Retina displays, including live backing-scale changes.

Both paths are highly optimized. The shared transform uses juce::dsp::FFT, which selects Apple’s Accelerate/vDSP implementation on macOS, and Spectrum and Spectrogram reuse each calibrated result. On the GPU side, the Spectrogram stores calibrated dB in a circular 16-bit-float (R16Float) texture: scrolling remaps texture coordinates, while shader controls recolor retained history without another FFT or a whole-texture copy. High display cadence therefore does not multiply the default 60 Hz FFT workload.

When the editor is closed, there is nothing to display, so capture, analysis, history, display-link callbacks, and Metal submissions stop. Audio still passes through normally. Reopening the editor begins fresh rather than silently spending host resources on invisible history.

Turning samples into pictures

The analyzers share infrastructure, but each one answers a different question. Here is the calculation path in a little more detail.

Spectrum: what frequencies exist now?

The Spectrum takes a short window of recent audio and uses a fast Fourier transform (FFT) to divide it into frequency bins. By default the transform size is \(N=8192\) samples, or about 171 milliseconds at a sample rate \(F_s=48\) kHz. This does not delay the audio by 171 ms; the plugin passes audio through immediately. It means the displayed estimate describes roughly that much recent history.

The FFT bin centers are separated by \(\Delta f=F_s/N\). With the defaults, that is approximately 5.86 Hz. This number is useful, but it is not the same as saying two tones 5.86 Hz apart can always be resolved: the selected window also determines how widely a tone spreads into nearby bins.

Before the FFT, samples are multiplied by a periodic five-term flat-top window \(w[n]\). Cutting an arbitrary piece from a continuous waveform creates artificial edges, which spread energy across the spectrum. A window tapers the data to control that leakage. A flat-top window trades some ability to separate nearby tones for better amplitude accuracy, which is a useful default for a measurement tool.

For channel \(c\), the transform is:

\[ \begin{aligned} X_c[k]&=\sum_{n=0}^{N-1}x_c[n]\,w[n]e^{-j2\pi kn/N},\\ f_k&=\frac{kF_s}{N}. \end{aligned} \]

Audio Insight corrects the window’s coherent gain—the amplitude scaling introduced by multiplying by \(w[n]\)—with \(W=\sum_n w[n]\). Real-valued audio has mirrored positive- and negative-frequency FFT bins, but the graph needs only the nonnegative half. In this one-sided view, the DC bin at 0 Hz and the Nyquist bin at \(F_s/2\) use \(1/W\); every bin between them represents both mirrored sides and uses \(2/W\). The calibrated stereo power and level are therefore:

\[ \begin{aligned} a_k&= \begin{cases} 1/W, & k=0\ \text{or}\ k=N/2,\\ 2/W, & \text{otherwise}, \end{cases}\\[3pt] P[k]&=\max_c\left(a_k|X_c[k]|\right)^2,\\ D[k]&=10\log_{10}P[k]. \end{aligned} \]

For mono, the maximum contains only one channel. For stereo, taking the larger channel magnitude avoids first mixing the waveforms to mono, where out-of-phase content could cancel. The calibration makes a bin-centered full-scale sine read 0 dB internally; powers at or below \(10^{-18}\) are displayed at the \(-180\) dB analysis floor.

Attack and Release then smooth each bin in linear power, not in dB. Given the elapsed time \(\Delta t\) and the selected time constant \(\tau_d\):

\[ \begin{aligned} \alpha_d&= \begin{cases} 0, & d\text{ is Off},\\ e^{-\Delta t/\tau_d}, & d\text{ is enabled}, \end{cases}\\[3pt] \bar P_t[k]&=\alpha_d\bar P_{t-1}[k]+(1-\alpha_d)P_t[k]. \end{aligned} \]

The direction \(d\) is Attack when \(P_t[k]\geq\bar P_{t-1}[k]\), otherwise Release. An Off direction follows the current FFT immediately. The default Attack is Off, allowing a short burst to appear at once, while the default 250 ms Release lets the trace fall more slowly. Peak hold, when enabled, operates on unsmoothed power instead of \(\bar P\).

Transforms target a slice rate \(R_s\) using a hop of \(H=\max(1,\operatorname{round}(F_s/R_s))\) new samples. At 48 kHz and 60 slices per second, \(H=800\), so adjacent 8,192-sample windows overlap by about 90.2%. The first result still waits for one complete window, and the latest-wins scheduler may skip stale transforms under load instead of building a backlog.

Spectrum and Spectrogram use the same continuously adjustable frequency scale. For a frequency \(f\) between \(f_0\) and \(f_1\), the scale control \(s\) blends normalized linear and logarithmic coordinates:

\[ \begin{aligned} u_{\mathrm{lin}}(f)&=\frac{f-f_0}{f_1-f_0},\\ u_{\log}(f)&=\frac{\ln(f/f_0)}{\ln(f_1/f_0)},\\ u(f,s)&=(1-s)u_{\mathrm{lin}}(f)+s\,u_{\log}(f). \end{aligned} \]

Here \(f_0=20\) Hz and \(f_1=\min(20\text{ kHz},F_s/2)\). The default is \(s=0.8\). At \(s=0\), equal distances represent equal numbers of hertz. At \(s=1\), equal ratios such as 100→200 Hz and 1→2 kHz occupy equal distances. Values in between preserve more low-frequency detail without compressing the entire treble into a tiny area. Spectrum uses \(x=u\), while Spectrogram uses \(y=1-u\) so high frequencies appear at the top. Axis labels are chosen dynamically: important anchors win first, then extra candidates fill only the space that remains.

Spectrogram: how did the spectrum change?

A Spectrum is one slice through time. A Spectrogram keeps those slices and scrolls them sideways, using color for level. Transients become vertical marks, steady tones become horizontal lines, and harmonics become stacks of related lines.

Each Spectrogram column starts from the same raw power \(P[k]\) as Spectrum, before Spectrum’s Attack/Release averaging. Let \(\mathcal K\) contain only usable bin centers from 20 Hz through \(f_1\), and let \(R_f=\min(1024,|\mathcal K|)\) be the texture’s frequency-row count. For a usable bin at \(f_k=kF_s/N\), define \(q_k=u(f_k,s)\). Its row is:

\[ r(k)=\min\left(R_f-1,\left\lfloor R_f q_k\right\rfloor\right). \]

For a row containing one or more bin centers, \(P_r\) is the greatest \(P[k]\) assigned to that row. Taking the maximum, rather than the average, helps a narrow tonal trace survive when several FFT bins land in one display row. If a row contains no bin center—common at low frequencies with a small FFT—the mapper inverse-maps the row center and linearly interpolates the two surrounding bins in power. It is honest interpolation between available samples, not a claim of extra frequency resolution.

Power at or below \(10^{-18}\) becomes \(-180\) dB; otherwise the mapper stores \(D_r=10\log_{10}P_r\). These values go into a circular Metal texture with one 16-bit floating-point level per cell (R16Float). The texture stores calibrated dB rather than finished colors. In the shader, let \(F\) be the selected floor, \(C\) the ceiling, and \(\eta\) the Color response:

\[ \begin{aligned} v&=\operatorname{clamp}\left( \frac{D_r-F}{C-F},0,1\right),\\ \gamma&=2^\eta,\\ c_{\mathrm{palette}}&=v^\gamma. \end{aligned} \]

The value \(c_{\mathrm{palette}}\) selects a point in the chosen palette. Response 0 is linear in dB; negative values reveal quieter detail, while positive values suppress low energy and emphasize stronger traces. Because this work happens in the shader, changing palette, range, or response recolors existing history without rerunning the FFT.

For a history duration \(T\) and requested slice rate \(R_s\), the texture uses \(\min(8192,\lceil TR_s\rceil)\) columns. The default ten seconds at 60 slices per second therefore needs 600 columns. A write index wraps around the texture, and the renderer changes texture coordinates instead of copying the whole image to scroll it. Missing timestamp intervals become black columns rather than stretching old information across time.

Peak and RMS: how strong is the signal?

Peak and RMS intentionally describe different things.

Sample peak examines every sample. Its live value has instantaneous attack and a 20 dB/s release:

\[ \begin{aligned} \lambda_p&=10^{-20/(20F_s)},\\ p[n]&=\max\left(|x[n]|,\lambda_p p[n-1]\right). \end{aligned} \]

In other words, a new larger sample wins immediately; otherwise the old indication decays by the amount corresponding to one sample period. A separate hold marker keeps a new maximum for two seconds, then falls at the same 20 dB/s rate. The OVER indicator latches when \(|x[n]|\geq1\), although the label deliberately does not claim that floating-point audio at 0 dBFS proves waveform clipping.

RMS estimates sustained signal power. With the 300 ms time constant \(\tau=0.300\) s, Audio Insight updates an exponential mean square for every sample:

\[ \begin{aligned} \alpha&=e^{-1/(F_s\tau)},\\ q[n]&=\alpha q[n-1]+(1-\alpha)x[n]^2,\\ \operatorname{RMS}[n]&=\sqrt{q[n]},\\ D_{\mathrm{RMS}}[n]&=20\log_{10}\operatorname{RMS}[n]. \end{aligned} \]

This is an exponential response, not a rectangular box containing exactly the latest 300 ms. It also has no AES17 \(+3.01\) dB calibration offset, so a full-scale sine reads approximately \(-3.01\) dBFS RMS. Peak reveals brief extremes; RMS behaves more like a view of sustained energy. The peak remains a sample peak, not an oversampled true-peak/dBTP measurement, so it does not predict a possibly larger value between stored samples.

These ballistics run on the bounded real-time capture path and inspect every sample. Their meaning therefore does not change if an analysis worker is briefly late.

The vectorscope turns each stereo sample pair into a point:

\[ x_{\mathrm{scope}}=\frac{R-L}{2}, \qquad y_{\mathrm{scope}}=\frac{L+R}{2}. \]

Audio shared equally by both channels has \(x=0\) and lies on the vertical center axis. Opposite-phase audio has \(y=0\) and spreads horizontally. The coordinates remain tied to full scale rather than being normalized independently on every frame, so a quiet signal is not made to look artificially loud.

The field keeps the latest 250 ms but bounds its GPU data. For \(W_f=\lceil0.25F_s\rceil\) captured frames, the worker selects one pair every:

\[ d=\left\lceil\frac{W_f}{4096}\right\rceil \]

frames. At 48 kHz, \(W_f=12000\), \(d=3\), and the cloud contains about 4,000 uniformly spaced points. Their opacity fades with age. This decimation changes only the picture; it does not change the correlation measurement.

The adjacent correlation value uses every sample, with 300 ms exponentially weighted averages:

\[ \begin{aligned} \alpha&=e^{-1/(F_s\cdot0.300)},\\ E_n[z]&=\alpha E_{n-1}[z]+(1-\alpha)z[n]. \end{aligned} \]\[ \rho[n]=\frac{E_n[LR]}{\sqrt{E_n[L^2]E_n[R^2]}}. \]

The three running values \(E[L^2]\), \(E[R^2]\), and \(E[LR]\) advance in source-sample order on the real-time side, and the implementation clamps the final ratio to \([-1,1]\) against numerical error. A value near \(+1\) means the channels are strongly alike, \(0\) means little linear relationship, and a negative value warns that mono playback may cancel important content. If either averaged channel power is below \(10^{-9}\), equivalent to \(-90\) dBFS RMS, correlation is reported as unavailable rather than dividing by a nearly zero value. A genuinely mono input is labeled MONO and likewise does not receive a synthetic \(+1\) correlation.

Loudness: how loud does it feel over time?

Raw peak level is not perceived loudness. Audio Insight implements BS.1770-5 K-weighting with the Momentary, Short-term, and Integrated semantics commonly used with EBU R128. K-weighting is a pair of filters: a high-frequency shelf models the head’s acoustic effect, and a high-pass stage reduces the contribution of very low frequencies. The code derives their coefficients for the current sample rate.

If \(y_c[n]\) is the K-weighted output of channel \(c\), Audio Insight forms the per-sample energy and a window mean:

\[ \begin{aligned} e[n]&=\sum_c y_c[n]^2,\\ z_W&=\frac{1}{N_W}\sum_{n\in W}e[n]. \end{aligned} \]

For the supported mono and stereo layouts, every actual channel has unit weight. Mono therefore contributes once; it is never duplicated into synthetic left and right channels. Surround layouts and their channel weights are outside the current scope. Window energy becomes LUFS (Loudness Units relative to Full Scale) using the BS.1770 offset:

\[ L_W=-0.691+10\log_{10}z_W. \]

Momentary loudness covers 400 ms.
Short-term loudness covers 3 seconds.
Integrated loudness uses 400 ms blocks completed every 100 ms—75% overlap—from the latest Reset within the current uninterrupted, visible analysis interval.

Momentary and Short-term are simple ungated window measurements. Integrated loudness applies gates: thresholds that exclude blocks from the long-term average. For each 400 ms block \(i\), let its mean-square energy be \(z_i\) and its loudness be \(L_i=-0.691+10\log_{10}z_i\). The absolute-passing set is:

\[ \begin{aligned} \mathcal A&=\{i\mid L_i>-70\ \mathrm{LUFS}\},\\ \mu_{\mathcal A}&=\frac{1}{|\mathcal A|}\sum_{i\in\mathcal A}z_i. \end{aligned} \]

At each Integrated update, one non-iterative relative threshold is calculated 10 LU below the preliminary absolute-gated mean of the history accumulated so far:

\[ \begin{aligned} \Gamma_{\mathrm{rel}}&=-0.691+10\log_{10}\mu_{\mathcal A}-10,\\ \mathcal R&=\{i\in\mathcal A\mid L_i>\Gamma_{\mathrm{rel}}\}. \end{aligned} \]

Finally:

\[ \begin{aligned} \bar z_{\mathcal R}&=\frac{1}{|\mathcal R|}\sum_{i\in\mathcal R}z_i,\\ L_I&=-0.691+10\log_{10}\bar z_{\mathcal R}. \end{aligned} \]

Both comparisons are strict \(>\), and the relative gate is not iterated repeatedly. This two-stage gate prevents silence and very quiet passages from dragging the program average down indefinitely. The tile’s Reset command restarts Integrated loudness while ready Momentary/Short-term values and K-weighting continuity remain intact. Editor reactivation, an audio discontinuity, or a format change resets the complete loudness analyzer.

The empty cases are explicit too. If \(\mathcal A\) contains no blocks, the preliminary mean and relative gate remain unavailable. If \(\mathcal R\) is empty, Integrated loudness remains \(-\infty\). The implementation never divides by an empty set.

The implementation does not claim complete EBU Mode compliance: it does not yet include LRA or true peak, for example. The label describes its M/S/I measurement semantics, not a certification.

There is an interesting performance problem hiding in Integrated loudness. Within one uninterrupted visible measurement, the exact answer can cover 24 hours: up to 864,000 blocks. Rescanning every qualifying block every 100 ms would make the cost grow throughout the measurement.

The implementation uses a preallocated sorted index called a B+ tree. It contains finite block energies above the absolute gate and keeps aggregate counts and sums in its branches; all completed blocks still count toward the 24-hour limit. A new relative-gate boundary can be answered by finding one boundary leaf and combining a bounded number of branch totals. Capacity for the worst case occupies about 7.25 MiB on arm64, and the structure never allocates while processing.

Smooth is a timing property, not an FPS number

The renderer uses CAMetalDisplayLink, which supplies a drawable in step with a display. While visible, Audio Insight requests the active display’s exact reported maximum refresh rate, with a 60 Hz fallback. That request is best effort—Core Animation and the compositor still control actual presentation—so measured presentation timestamps are the truth.

This distinction became important repeatedly. A counter can say 120 callbacks per second while the screen still changes only 60 times. An average can say 120 FPS while an occasional doubled interval makes scrolling visibly hitch. Smoothness is about the complete chain from callback to presentation and about the distribution of frame intervals, not just one large number.

The plugin that crashed its host

The first AU build appeared for a moment in SoundSource and then disappeared. The host reported only that its Audio Unit hosting service had crashed.

The detailed log led to an assertion in timed drawable presentation. A normal Metal application may call an API such as timed present, but a drawable delivered by CAMetalDisplayLink has different presentation ownership. Combining the two caused the hosting process to assert. The correct sequence is to commit the command buffer and call plain present() on that drawable, while using the display link’s target timestamp only for telemetry and scheduling.

This is one reason plugin development needs testing in real hosts. SoundSource exposed an API misuse that a successful build or unit test had not.

Why 120 display callbacks produced 60 frames

After the crash was fixed, the display link was firing close to 120 times per second, but only about 60 frames were submitted. The built-in metrics captured the pattern:

Counter	Before the fix
Display-link callbacks	6,453
Metal submissions	3,230
GPU-backpressure drops	3,223
Sampled display-link callback rate	~111/s
Sampled Metal submission rate	~59.6/s

The cumulative counters cover the full telemetry epoch; the two rates are a sample from its final roughly 0.25 seconds. Almost exactly every other display-link callback was being rejected by the in-flight buffer pool.

The surprising part was that the GPU was not necessarily too slow. Reusable vertex buffers were retained until the drawable was actually presented. The compositor may hold a drawable for several refresh periods even after GPU execution has completed, so all reusable buffers became occupied and the next callback had nowhere to write.

The fix was to separate two lifetimes. GPU command completion now releases the reusable buffers immediately. A small, independent object survives only to correlate the later presentation timestamp. The renderer no longer holds large working resources hostage to compositor timing.

In a later point-in-time M1 Max capture from another development build, with the Metrics panel visible, the drawable was 2,400×1,496 pixels at 2× backing scale. The run recorded 1,188 display-link callbacks, 1,188 submissions, and zero GPU-backpressure drops. Across the most recent 240 presented intervals, the average was 8.438 ms, or 118.52 Hz; 237 intervals were the normal 8.333 ms and three doubled to 16.667 ms. Telemetry also counted 11 skipped presentations over the run. That capture used the then-selected 16,384-point FFT; today’s default is 8,192.

The capture demonstrates that the buffer-lifetime bottleneck and its GPU-backpressure drops were gone. It is evidence of approximately display-rate presentation in that run, not a perfect-pacing claim or a controlled comparison with another plugin.

Making 60 Hz data scroll on a 120 Hz display

The Spectrogram exposed a second kind of stutter. New analysis columns arrive at 60 Hz. If the image moves forward only when a complete column arrives, it necessarily steps every other frame on a 120 Hz display.

The solution was not to double the FFT workload. The renderer advances a fractional scroll head from the target presentation clock while keeping the actual dB cells discrete. A one-slice cushion absorbs ordinary analysis scheduling jitter. If a texture upload is briefly busy, the renderer postpones that upload while continuing to draw the rest of the dashboard.

The result is much smoother motion from the same 60-slices-per-second target. This also explains why raising thread priority would have been the wrong first response: the main issue was the relationship between two clocks, not a shortage of real-time privileges.

The random resets that were not random

During longer sessions, all graphs would occasionally reset. The recovery was intentional—when audio history has a real gap, temporal analyzers must not pretend the samples on either side were adjacent—but the handoff overflowed far too easily even while host audio was continuous. Sequence tracking then correctly detected the resulting loss.

The original capture queue had 16 logical slots and consumed one for each host callback. Its time capacity therefore depended on the host’s block size. A metrics capture reached all 16 ready slots, discarded 20 queued chunks to make room for newer audio, and recorded three consumer discontinuities followed by three Loudness resets.

The redesigned queue packs audio across callback boundaries into 128 slots of 256 frames, retaining 32,768 frames regardless of host callback size. That is about 683 ms at 48 kHz, 341 ms at 96 kHz, or 171 ms at 192 kHz. The capacity and overflow behavior are covered by implementation tests; longer post-redesign host runs remain part of validation. A sufficiently long stall can still overflow it. When that happens, latest data wins and temporal analyzers reset, because joining unrelated pieces of audio would produce convincing but false measurements.

Building observability into the plugin

Apple’s Metal HUD is useful for applications that enable it before creating their first Metal device. A plugin usually arrives after its host has already done that, so it cannot reliably switch the HUD on from a settings button. I replaced that idea with a built-in performance panel available in Release builds.

It reports exact frame pacing over the latest 240 presentation intervals, derived from 241 timestamps; CPU, submit, GPU, and compositor latency composition; display-link scheduling; audio-callback histograms; queue occupancy and discontinuities; analyzer freshness; and raw copyable metrics for offline inspection. The stacked latency bar covers the pipeline from a display-link callback to presentation. It is not a breakdown of an 8.33 ms frame budget: several frames can overlap in flight, so its total can exceed one refresh interval without reducing presentation cadence.

Graphs move at vblank, headline numbers refresh at no more than 10 Hz, and the full text table refreshes at 4 Hz. That keeps the visual feedback immediate without rebuilding lots of text 120 times per second. Instrumentation turned several vague reports—“it looks a bit laggy,” “it seems to reset”—into specific, actionable failures.

An agent-assisted, human-tested development loop

I used coding agents to implement much of Audio Insight. That did not remove the need for a tight feedback loop; it made the loop more important.

The agents could design the threading model, inspect crash logs, add instrumentation, and reason from raw captures. They could build the plugin, but they could not reliably judge how motion felt inside my particular SoundSource setup. At runnable milestones I installed the AU, watched it on the M1 Max, adjusted settings, and returned observations, screenshots, logs, or copied metrics. Those reports led directly to the presentation-lifetime fix, the fractional Spectrogram scroll, dynamic axis labeling, and the queue redesign.

For visual and real-time software, “the code is correct” and “the product feels right” are different claims. An instrumented implementation plus a person looking at the actual display proved far more useful than guessing at either one in isolation.

What is open source today

The current code identifies itself as Audio Insight 0.1.0. It targets macOS 15 on arm64 and builds AUv2 and VST3. It uses C++20, CMake, a pinned JUCE submodule for the plugin shell, CPU FFT analysis accelerated by Apple’s vDSP on macOS, and a native Metal renderer. Project-owned code is licensed under AGPL-3.0-or-later; JUCE retains its own upstream AGPL terms.

The current release policy is pragmatic for a small open-source project: builds use ad hoc signing, and Developer ID signing and notarization are out of scope. Users can build from source. For a downloaded bundle, the documented flow is to verify the published checksum, extract it, clear quarantine only on the intended bundle, apply an ad hoc signature, and verify that signature.

Older macOS versions, Intel/Universal builds, Windows, and AUv3 are architectural possibilities rather than current support promises. Logic compatibility, broader VST3 host coverage, multi-instance stress testing, and several formal performance gates also remain work in progress.

The source, build instructions, and current limitations are all in the Audio Insight repository. If you use analyzers but have never looked inside one, I hope the code makes the path from samples to pixels a little less mysterious. And if the Spectrogram glides across a 120 Hz display without drawing attention to the renderer, that is exactly the point.

Thoughts on Re-Encoded BD Animes

Tue, 17 Jan 2023 21:28:00 +0800

Introduction

Re-encoded BD anime series are my go-to when watching animes. They have great quality while being a fraction of the size of a Blu-ray disc. Let alone they come with all kinds of extra contents.

Nowadays, re-encoded BD anime series usually have video tracks compressed using 10-bit x265 (a software implementation of the H.265/HEVC video compression standard), and audio tracks compressed using FLAC (a lossless audio codec).

The video tracks are great, generally a good balance between size and quality, no complaints. The audio tracks are my main problem. Sometimes they take up too much space with no real benefit for most viewers.

Other versions like WEB-DL are not considered because they are already heavily compressed so they don’t have the space concerns.

Video Tracks

I am totally satisfied with the video tracks, because:

the x265 encoder is very efficient (especially at variable bitrate controlling, which most hardware encoders still suffer from, i.e. NVIDIA NVENC, Intel QuickSync, and AMD VCN), being able to compress video to a fraction of the size of the BD original while preserving video quality;
the encoding team usually does a great job doing manual optimizations to fix problems in the original video, such as aliasing, ringing, and color banding. (Some team even goes as far as AI upscaling the video, although a bit controversial.)

Despite being one third (or even less) of the original bitrate, you can barely tell the difference between the re-encoded video and the original one, even when comparing side-by-side. In fact, the re-encoded video is sometimes even better due to the manual optimizations by the encoding team. This saves a lot of space while preserving video quality.

Video encoding is a complex topic, involving a huge amount of trial and error, even x265-encoding alone has a TON of dials and knobs to tweak, and this is only part of the whole encoding process. I will not go into details here, simply because of my lack of knowledge.

Samples

For example, you can see what the encoding team (VCB-Studio) does to optimize the original video quality from the note of Haiyore! Nyaruko-san / 潜行吧！奈亚子 :

For Season 1, the source is of nothing special. Its native resolution is 720p, then upscaled by a poor algorithm. This leads to many defects: lines featuring severe aliasing and ringing as well as serious ringing on the image borders. Dark scenes are widely spread throughout Season 1, together with the plentiful use of dark fades, resulting in heavy colour banding. For the lines, most of the problems can be solved by descaling and reconstruction, then slight AA and moderate de-ringing as supplements. For colour banding, since most scenes have only a little colour banding, in order to deal with severe colour banding in dark scenes, we designed a complex mask for protection in combination with luminance information. We also use a colour banding detection algorithm. Thanks to all those, we finally adaptively controlled the strength of de-banding. At the end, we added some grains to improve the viewing exprience.

第一季原盘画质一般。原生分辨率为 720p，被比较劣质的算法拉升上来，导致线条带有严重锯齿和振铃，而且画面的边框部分有很强的拉升带来的边缘振铃。画面暗场极多，而且大量运用暗式淡入淡出，这些场景出现了严重的色带。对于线条，大部分问题都可以通过逆向拉升再重构解决，然后补上轻微的抗锯齿处理和中等强度的去振铃处理。对于色带，由于大部分场景的色带都很少，只有少部分暗场有较严重的色带，我们结合亮度信息设计了复杂的保护手段，并使用了一个色带检测算法，根据检测的结果对去色带的力度进行了自适应调整。最后加上了一定强度的动态噪点，以改善原盘噪点的观感。

MediaInfo: Haiyore! Nyaruko-san episode 1 from VCB-Studio

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


General
Unique ID : 276260962046491867529669363000855506406 (0xCFD5ED098E39DF78A31330DA2C4C2DE6)
Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[VCB-Studio] Haiyore! Nyaruko-san/[VCB-Studio] Haiyore! Nyaruko-san [Ma10p_1080p]/[VCB-Studio] Haiyore! Nyaruko-san [01][Ma10p_1080p][x265_flac].mkv
Format : Matroska
Format version : Version 4
File size : 946 MiB
Duration : 23 min 56 s
Overall bit rate mode : Variable
Overall bit rate : 5 524 kb/s
Encoded date : UTC 2022-05-28 10:09:48
Writing application : mkvmerge v48.0.0 ('Fortress Around Your Heart') 64-bit
Writing library : libebml v1.4.0 + libmatroska v1.6.0

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L4.1@High
Codec ID : V_MPEGH/ISO/HEVC
Duration : 23 min 56 s
Bit rate : 4 028 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.081
Stream size : 690 MiB (73%)
Writing library : x265 3.5+97-ga456c6e73+1-g9859a8cb5:[Windows][clang 14.0.0][64 bit] Kyouko 10bit+8bit+12bit
Encoding settings : rc=crf / crf=14.0000 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=38000 / vbv-bufsize=40000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / no-lossless / no-cu-lossless / aq-mode=3 / aq-strength=1.00 / aq-bias-strength=1.00 / cbqpoffs=-2 / crqpoffs=-2 / ipratio=1.40 / pbratio=1.20 / psy-rd=2.00 / psy-rdoq=1.00 / deblock=-1:-1 / ref=5 / limit-refs=0 / no-limit-modes / bframes=10 / b-adapt=2 / bframe-bias=0 / b-pyramid / b-intra / weightp / weightb / min-keyint=1 / max-keyint=360 / rc-lookahead=80 / gop-lookahead=0 / scenecut=40 / hist-scenecut=0 / radl=0 / max-cu-size=32 / min-cu-size=8 / me=3 / subme=5 / merange=38 / rdoq-level=1 / rd=5 / rdpenalty=0 / dynamic-rd=0.00 / rd-refine / ----- / cutree / no-sao / rect / no-amp / no-open-gop / wpp / no-pmode / no-pme / no-psnr / no-ssim / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-tu-size=16 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / qg-size=32 / qpmax=69 / qpmin=0 / ----- / cpuid=1111039 / frame-threads=4 / numa-pools=+ / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / no-allow-non-conformance / no-repeat-headers / no-aud / no-hrd / info / hash=0 / no-temporal-layers / lookahead-slices=0 / no-splice / no-intra-refresh / no-ssim-rd / signhide / tskip / max-merge=5 / temporal-mvp / no-frame-dup / no-hme / no-analyze-src-pics / no-sao-non-deblock / selective-sao=0 / no-early-skip / no-rskip / no-fast-intra / no-tskip-fast / no-splitrd-skip / zone-count=0 / no-strict-cbr / no-rc-grain / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

Audio
ID : 2
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 23 min 56 s
Bit rate mode : Variable
Bit rate : 1 494 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 256 MiB (27%)
Writing library : libFLAC 1.3.2 (UTC 2017-01-01)
Language : Japanese
Default : Yes
Forced : No

Menu
00:00:00.000 : en:Chapter 01
00:01:14.992 : en:Chapter 02
00:02:45.040 : en:Chapter 03
00:13:08.997 : en:Chapter 04
00:22:10.037 : en:Chapter 05
00:23:40.002 : en:Chapter 06

As of video quality, you can verify yourself. Here are some frames extracted from Haiyore! Nyaruko-san / 潜行吧！奈亚子 (VCB-Studio). Can you tell any difference between the original and the re-encoded video? Remember that the re-encoded video has a bitrate of only 4028 kbps, while the original video will at least triple that.

This blog uses Responsive Images to improve experience, so what you see here may be scaled down. To view the original image, you can download the original file from the links provided.

Original	Re-encoded
original-0	re-encoded-0
original-1	re-encoded-1
original-2	re-encoded-2
original-3	re-encoded-3

Can’t notice any difference? Or the re-encoded one look better? Save the image locally and zoom in further :p. Yes, I DO put the correct pictures in the correct places.

The static images are already hard to tell apart. The difference will be even smaller when you are watching the video, simply because there are motions in the video and you will not be able to focus on the details.

So yeah, the video tracks are great. I will not touch them.

Audio Tracks

Lossless Compression Problem

However, the audio part is where I have a problem. It’s not about quality, but about size. Let me explain.

Unlike the video tracks, which is compressed using a lossy codec (H.265/HEVC), the audio tracks are usually losslessly compressed, which means that its quality is 100% same to the original, literally. This is great for archival purposes, but it’s a overkill for most people. Because what comes with great quality is size. FLAC tracks usually have a bitrate of 1000+ kbps. One FLAC track in a single episode will typically be over 200 MB in size, as you can seen from the example below. By compressing them to 200 kbps, for example, you save 800+ kbps per track, or put it simply, cut a 200 MB track to 40 MB, which saves a lot of space. This is especially true for animes with multiple FLAC tracks. An extreme example is Kobayashi-san Chi no Maidragon / Miss Kobayachi’s Dragon Maid / 小林家的龙女仆 from team AI-Raws. It has three FLAC tracks, with a total bitrate of 3955 (1432+1294+1229) kbps. Compressing them to 600 (200*3) kbps will cut the entire episode size from 1.89 GB to 1.3 GB, which is a 36% reduction in size, saving 400 MB per episode. That’s a lot.

MediaInfo: Kobayashi-san Chi no Maidragon episode 1 from AI-Raws

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109


General
Unique ID : 39274225615593709220200463866578723060 (0x1D8BF0D2BC4C9D643B9E040209C9E0F4)
Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[AI-Raws][Miss Kobayachi's Dragon Maid][BDRip][MKV]/[AI-Raws] 小林さんちのメイドラゴン #01 (BD HEVC 1920x1080 yuv444p10le FLAC 日本語字幕)[6FDC51A9].mkv
Format : Matroska
Format version : Version 4
File size : 1.89 GiB
Duration : 25 min 11 s
Overall bit rate mode : Variable
Overall bit rate : 10.8 Mb/s
Encoded date : UTC 2022-01-24 14:17:11
Writing application : mkvmerge v33.1.0 ('Primrose') 64-bit
Writing library : libebml v1.3.7 + libmatroska v1.5.0

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Format Range@L4@High
Codec ID : V_MPEGH/ISO/HEVC
Duration : 25 min 11 s
Bit rate : 6 700 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Chroma subsampling : 4:4:4
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.135
Stream size : 1.18 GiB (62%)
Writing library : x265 2.9+8-27d8424c799d:[Windows][MSVC 1900][64 bit] 10bit
Encoding settings : cpuid=1111039 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=3 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / no-lossless / cbqpoffs=6 / crqpoffs=6 / rc=crf / crf=14.5 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=25600 / vbv-bufsize=10240 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=31 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709

Audio #1
ID : 2
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 432 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 258 MiB (13%)
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : Yes
Forced : No

Audio #2
ID : 3
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 294 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 233 MiB (12%)
Title : キャストコメンタリー
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : No
Forced : No

Audio #3
ID : 4
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 229 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 221 MiB (11%)
Title : スタッフコメンタリー
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : No
Forced : No

Text
ID : 5
Format : PGS
Muxing mode : zlib
Codec ID : S_HDMV/PGS
Codec ID/Info : Picture based subtitle format used on BDs/HD-DVDs
Duration : 24 min 30 s
Bit rate : 277 kb/s
Count of elements : 2364
Stream size : 48.5 MiB (3%)
Language : Japanese
Default : Yes
Forced : No

Lossy Compression

But compressing them so hard will certainly bring quality losses… Or will it? Well, yes, there must be quality loss when using lossy codecs, but the question is how much? Is it acceptable? You have seen that the example video tracks are compressed to 4028 kbps, which is a lot less than the original video. But the video tracks are still great. So, is the quality loss acceptable for audio tracks?

In fact, using a modern codec at 256 kbps (or even half that), the quality loss is so little that I can almost guarantee that you will NOT hear a difference for two reasons:

most codecs are considered “transparent” at 256 kbps or above (better codes will be “transparent” at even lower bitrate), which means that you cannot tell the difference between the original and the compressed one (more on this later);
when watching anime, you are usually not paying attention to details in the audio, but the video. This makes the difference in audio tracks even less noticeable.

So we can safely compress audio without worrying about quality loss. But how much should we compress? Well, that depends on your own preference. Generally, you can choose a codec and a bitrate that is better than what is “transparent” to you.

But hold on, how do I know what is “transparent” to me?

Audio Transparency

There is a concept called “transparent” in audio encoding, which means the compressed audio is so good that a person cannot tell a difference between the original and the compressed one. This is usually done through ABX blind tests, where the listener is not told which one is the original and which one is the compressed one. If the listener cannot tell the difference, then the audio is considered transparent. Of course, this differs from person to person. You have to do the test with your own ears with different encoders at different bitrates to find your “transparent” threshold (which codec at which bitrate). When you find your threshold, any better codec at any better bitrate will also be transparent to you.

About Bitrate and quality:

They cannot be compared directly. For example, 320 kbps MP3 is not the same as 320 kbps AAC. (AAC is generally considered a better codec than MP3, so it can achieve the same quality with a lower bitrate.)

When dealing with the same codec (same encoder, same settings), the higher the bitrate, the better the quality. For example, MP3 at 320 kbps wll have better quality than MP3 at 256 kbps.

But when dealing with different codecs, the bitrate is not the only factor that affects the quality. The codec itself, the encoder, encoding settings, and etc. all affect quality. For example, AAC (qaac, vbr, default settings) at 256 kbps is often said to have similar quality to MP3 at 320 kbps.

But the general consensus is that Opus is transparent at 160kbps and above, and AAC is transparent at 192kbps and above:

A blind test of multiple codecs at ~192kbps VBR shows that codecs at 192k is transparent to the test subject. Opus being the most transparent codec at 192k;
Tests of different encoders at different bitrates published by SoundExpert show that most codecs at 128kbps and above are transparent to most test subjects. Detailed results can be found at http://soundexpert.org/encoders-128-kbps;
A topic from HydrogenAudio forum even shows that Opus at ~80kbps is transparent to an average listener;

It’s interesting to observe that 4 members have mentioned Opus@80kbps as point where is hard to spot artifacts for them.

Opus ~80 kbps is roughly equivalent to LAME ~130 kbps (V5) which lands in an “excellent” area of quality (MOS 4.5+) http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm

So one could say that Opus 80 kbps is “excellent” at least for an average listener. It’s clear that an experienced listeners can spot artifacts at much higher rates.

I never noticed any artifacts at 80kbps though, though I haven’t tried to find any either. Quote from: noiselab on 2017-09-18 06:49:15

I tried it and for me it’s 80 kbit. Quote from: hlloyge on 2017-09-18 11:49:01

I’ll be honest, I struggled to ABX at 64kbps. 80kbps is enough for me. Quote from: Funkstar De Luxe on 2017-09-18 13:35:01

80kbps is pretty much my limit as well, can’t be bothered listening to killer samples all the time. Quote from: bstrobl on 2017-09-18 14:39:30
All iTunes music is delivered in AAC at 256kbps (before the arrival of Apple Music Lossless), which is an indication that most people cannot tell the difference as Apple is a company that cares about quality;

As you can see I am only focusing on AAC and Opus (as of lossy codecs), because AAC offers good sound quality and great compatibility while Opus has great sound quality and “okay” compatibility. Other codecs like MP3 are just not good enough to compete with them, so they will not be my choice later.

Listening Samples

Now bring your best audio equipment. Same as before, let’s listen to some samples to see if you can tell a difference, or you can do blind ABX tests by yourself to find your “transparent” threshold.

All lossy codecs are encoded using VBR (CBR is not considered because it is not size-efficient) with a quality setting to match the resulting bitrates of different encoders as close as possible.

Encoder versions

1
2
3
4


opusenc 0.2-3-gf5f571b; libopus 1.3.1
OggEnc v2.88; libvorbis 1.3.6
qaac 2.73; CoreAudioToolbox 7.10.9.0
lame 3.100

Note that if you cannot playback some audio tracks, make sure your browser supports this type of audio or you can download the audio tracks and play them locally. Latest versions of Chromium-based browsers should be fine. Safari might have some issues with Opus and Ogg.

Orange / オレンジ

For the first example, I will use Orange / オレンジ by 7!! — the 2nd ending song from Shigatsu wa Kimi no Uso / 四月は君の嘘 / Your Lie in April / 四月是你的谎言 for its female voices. The audio is not complex (simpley put, do not contain many instruments), so it is easier for the encoders to achieve good results.

Vocal

Reference track:

flac, best, 893k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 276k	qaac, q109, 225k	oggenc2, q8, 234k	lame, v0, 249k
~192k	opusenc, vbr192, 212k	qaac, q91, 171k	oggenc2, q6, 178k	lame, v2, 182k
~160k	opusenc, vbr160, 180k	qaac, q82, 142k	oggenc2, q5, 153k	lame, v4, 148k
~128k	opusenc, vbr128, 146k	qaac, q64, 113k	oggenc2, q4, 131k	lame, v5, 126k
~96k	opusenc, vbr96, 111k	qaac, q45, 86k	oggenc2, q2, 92k	lame, v7, 98k
~64k	opusenc, vbr64, 76k	qaac, q27, 66k	oggenc2, q0, 61k	lame, v9, 69k

Vocal with simple instruments

Reference track:

flac, best, 920k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 274k	qaac, q109, 236k	oggenc2, q8, 239k	lame, v0, 256k
~192k	opusenc, vbr192, 210k	qaac, q91, 180k	oggenc2, q6, 183k	lame, v2, 183k
~160k	opusenc, vbr160, 179k	qaac, q82, 149k	oggenc2, q5, 158k	lame, v4, 147k
~128k	opusenc, vbr128, 144k	qaac, q64, 119k	oggenc2, q4, 134k	lame, v5, 126k
~96k	opusenc, vbr96, 109k	qaac, q45, 90k	oggenc2, q2, 95k	lame, v7, 100k
~64k	opusenc, vbr64, 75k	qaac, q27, 66k	oggenc2, q0, 62k	lame, v9, 70k

Vocal with more complex instruments

Reference track:

flac, best, 1089k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 262k	qaac, q109, 270k	oggenc2, q8, 266k	lame, v0, 288k
~192k	opusenc, vbr192, 200k	qaac, q91, 200k	oggenc2, q6, 201k	lame, v2, 206k
~160k	opusenc, vbr160, 169k	qaac, q82, 165k	oggenc2, q5, 170k	lame, v4, 154k
~128k	opusenc, vbr128, 136k	qaac, q64, 129k	oggenc2, q4, 142k	lame, v5, 134k
~96k	opusenc, vbr96, 102k	qaac, q45, 96k	oggenc2, q2, 97k	lame, v7, 101k
~64k	opusenc, vbr64, 69k	qaac, q27, 69k	oggenc2, q0, 62k	lame, v9, 68k

Vision

The second song will be a more complex, or demanding one — Vision by 中島岬:

Vision reference track:

flac, best, 1058k

Vision compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 260k	qaac, q109, 297k	oggenc2, q8, 290k	lame, v0, 288k
~192k	opusenc, vbr192, 197k	qaac, q91, 218k	oggenc2, q6, 212k	lame, v2, 209k
~160k	opusenc, vbr160, 165k	qaac, q82, 178k	oggenc2, q5, 175k	lame, v4, 156k
~128k	opusenc, vbr128, 133k	qaac, q64, 142k	oggenc2, q4, 139k	lame, v5, 135k
~96k	opusenc, vbr96, 100k	qaac, q45, 109k	oggenc2, q2, 99k	lame, v7, 109k
~64k	opusenc, vbr64, 66k	qaac, q27, 77k	oggenc2, q0, 63k	lame, v9, 73k

You can clearly see the AAC and MP3 encoders tend to give a higher bitrate than target bitrate in VBR mode to handle complex songs.

Want to listen to your own song at differnet bitrates? Use this script to encode it to all formats. This is exactly the same script that I wrote to encode the songs and generate the table above.

Now, you should have your own understanding of the different codecs: above which bitrate of which codec is transparent to you.

My take on the codecs

Here is my opinion: since ~192k Opus is already transparent to me (and to most people), why use 1000+ kbps FLAC? I can save a lot of space, without even tell a difference! Unless you have outstandingly gifted ears with high-end equipments, or you do heavy post-processing on audio, you should and will be fine with a transparent lossy encoding.

Also, most audio in animes is voice, which is usually not complex for encoders, so the actual bitrate can be even lower to sound good.

Still writing…

The rest is still being written…

Audio on Charlie Chiang's blog

Building Audio Insight: An Audio Analyzer That Actually Renders Smoothly

What an audio plugin actually does

Turning samples into pictures

Spectrum: what frequencies exist now?

Spectrogram: how did the spectrum change?

Peak and RMS: how strong is the signal?

Stereo: how are left and right related?

Loudness: how loud does it feel over time?

Smooth is a timing property, not an FPS number

The plugin that crashed its host

Why 120 display callbacks produced 60 frames

Making 60 Hz data scroll on a 120 Hz display

The random resets that were not random

Building observability into the plugin

An agent-assisted, human-tested development loop

What is open source today

Thoughts on Re-Encoded BD Animes

Introduction

Video Tracks

Samples

Audio Tracks

Lossless Compression Problem

Lossy Compression

Audio Transparency

Listening Samples

Orange / オレンジ

Vocal

Vocal with simple instruments

Vocal with more complex instruments

Vision

My take on the codecs

Still writing…