Song Structure & Hierarchical Harmony — Research Report

00Summary

The hierarchy idea is well supported by how published chord and key systems work (transition-penalised decoding, whole-piece context for local key) but is not yet proven on our data (ESTIMATE); most of it can be built on CPU from pieces we already run. The biggest wins are not a higher whole-song key score. They are cleaner chord charts (bar-level decoding, no flicker), real section boundaries instead of four equal slices, and per-section keys and modulation flags that we can trust.

~0.70MEASURED strict boundary hit-rate F (±0.5 s, HR.5F .703) on SongFormBench-Harmonix — SongFormer (HX), their paper/README

~0.78MEASURED HR3F (±3 s) .784, same benchmark and model

~0.81MEASURED frame-level label ACC .807 on SongFormBench-Harmonix (7 label classes; a different SongFormer training variant, HX+E+H+G)

84%MEASURED ChordFormer MajMinor WCSR 84.09, Sevenths 72.28 — 1,217-song Isophonics+Billboard+MARL set, 5-fold CV

Recommendations

Start with CPU-only wins (phases 0–2): a benchmark harness, a bar-level chord Viterbi on the chroma we already compute, and self-similarity sections snapped to SANCT bar lines. No new licences, no GPU.
Treat deep section models as a second opinion, behind a licence gate (phase 3). allin1 is MIT code; SongFormer is stronger on paper but depends on MuQ, whose published weights are CC-BY-NC. That is a commercial-use blocker until cleared.
SANCT stays sacred. Everything here reads beat_times/downbeat_times and never rewrites them. Section edges are snapped to the grid, not the other way round.
Hierarchy is not “shorter windows”. The plan adds priors and pooling over bar-level states; it does not shrink the analysis window, which would make chroma noisier. See 4b.
Measure first. The repo has no key or downbeat ground-truth benchmark today (section 1). Phase 0 fixes that; without it every gain below is an estimate.

How to read the numbers

Accuracy figures are from papers that report them on specific datasets. They do not transfer 1:1 to our users’ uploads (stems from Demucs, loop-based beats, mixed genres). Treat them as ceilings, not promises. All projected gains (roadmap, compute adders, “expected” anything) are ESTIMATE. Anything I could not fetch today is tagged unverified and listed at the bottom. A claim-by-claim ledger with the sources re-checked is in Claim ledger.

01How our key detection works today

Read from the repo (read-only). Code lives in afcut_engine/harmonic.py; webapp/app.py:_detect_key is now a thin wrapper that delegates to it. verified in repo

Stage	What it does	Where
Input	Full mix (the pack’s mixed audio). No stems are used for key. Mono downmix, resampled to 22.05 kHz, capped at the first 120 s.	harmonic.py ANALYSIS_SR, MAX_ANALYSIS_SECONDS; app.py:7543, :28930
Harmonic isolation	HPSS (STFT n_fft 1024, kernel 17, margin 1/3) keeps the harmonic part, so drums do not smear the chroma.	_harmonic_component
Chroma	CQT chroma on the harmonic part. Frames are RMS-weighted and gated at 5% of the 95th-percentile energy, then averaged to one 12-bin vector.	_harmonic_chroma, _weighted_chroma
Profiles	Krumhansl-Kessler major/minor only. No Temperley, edma/edmm or learned profiles.	KRUMHANSL_MAJOR / MINOR
Scoring	Rank (Spearman-style) correlation against all 24 rotated profiles. Rank transform makes it robust to the overall loudness/shape of the chroma vector.	_rank_keys
Confidence	Margin between best and second-best, `margin / 0.125`, deliberately no floor. Returns the top 3 alternatives. Code comments say it was tuned on a small calibration set and that a wrong answer still reached 0.39 — i.e. confidence is a hint, not a probability.	_calibrate_confidence, _alternatives
Cross-checks	CENS chroma on an 11 kHz / 45 s excerpt gives an `key_agreement` flag; a triad-pair “rival tonality” test (threshold 0.16) flags relative/neighbour confusions; a guard returns “indeterminate” for silence/DC/degenerate scores.	detect_key_advanced, _rival_tonality_gain, _degenerate_key_reason
Per-section key	Not structural. `detect_per_section_keys` splits the audio into 4 equal-length slices and runs the same scorer on each.	harmonic.py:734
Chords	`chord_timeline`: for each beat window (from the SANCT grid, read-only) take triad-template cosine argmax over 12 roots × a few qualities. Merge identical neighbours, cap at 200. No HMM, no transition cost, no bar awareness, no 7ths. Confidence from template margin.	harmonic.py:1099
Where it runs	`_harmonic_layer` on the worker cache-miss path only (never on the API origin). Measured ~13.6 s per ~4 min song, ~0.9–1.1 GB peak on the cx33 worker box.	app.py:~32335; docs/reports/2026-10-09-song-intelligence-strategy.md
Section prototype	Offline-only `tools/song_intel_demo.py`: per-bar energy (RMS + onset density + brightness), novelty peaks with a 4-bar minimum, labels from energy rank (Intro/Verse/Build/Drop…). Not wired into the product.	tools/song_intel_demo.py

Why it is accurate (and where that is a claim, not a measurement)

It reads harmonic content only (HPSS), uses CQT chroma and a rank-based comparison — all robust choices that avoid the classic failure of full-mix chroma polluted by drums.
It reports alternatives and a margin, so relative-key confusion (e.g. “D# minor, could also be F# major” in the 2026-10-09 demo) is surfaced instead of hidden.
It is an honest baseline, not state of the art: one profile family and a single global vector. Literature suggests profile choice alone matters by genre (section 4).

Data we have to benchmark it

Essentially none. No tests/ directory, no ground-truth key file, no evidence folder with labelled keys. The code comments mention two published-key demo tracks used for calibration (one still came out wrong: a C♯ minor piece detected as D minor, per scout notes unverified: not re-checked). mir_eval==0.8.2 is already pinned in requirements.txt, so scoring tools are free. The founder’s “very accurate” is a plausible but unmeasured claim — phase 0 of the roadmap turns it into a number.

Also noted: the Track 12 plan (docs/plans/2026-09-14-track12-bpmandkey-harmonic-analysis.md) still lists chord_timeline as to-be-built and cites _detect_key at app.py:4772; both are stale (it now lives at app.py:5741, and chords exist). Not edited here.

02Section / structure segmentation

Libraries and models

System	What	Reported accuracy	Licence	Notes
allin1 (All-In-One)	Joint beats, downbeats, section boundaries and labels (intro/verse/chorus/bridge/outro…). Takes demixed stems. WASPAA 2023.	MEASURED Harmonix, 8-fold CV (their paper): beat F1 .958, downbeat F1 .915, boundary HR.5F .660, label pairwise-F .738 (on SongFormBench-Harmonix, measured by the SongFormer authors: HR.5F .596, HR3F .730, ACC .740)	Code MIT verified (GitHub licence API). Weights licence not stated in the README unverified. Needs NATTEN and madmom (from git).	Speed per README: 33 min of audio in 73 s on RTX 4090 + 14-core CPU. Trained on Harmonix only (pop/electronic skew).
SongFormer (Oct 2025)	Fuses MuQ + MusicFM representations; trained on 14k+ songs (SongFormDB) with mixed label schemas. arXiv 2510.02797.	MEASURED SongFormBench-Harmonix (200 Harmonix + 100 Chinese-pop songs for the full bench): HR.5F .703, HR3F .784 (HX-trained variant); ACC .807 (HX+E+H+G variant) — best figures across the paper’s variants, not one model. RWC-Pop (HX+E+H+G variant): HR.5F .650, HR3F .800, ACC .812	Code CC-BY-4.0 (README “License” section) verified. The README tells users to fetch MuQ-related weights; MuQ’s README states its published weights (MuQ-large-msd-iter, MuQ-MuLan-large) are CC-BY-NC 4.0 verified. We did not establish which exact checkpoint SongFormer loads or its own weights licence, so the NC conflict is likely, not confirmed commercial blocker until cleared	README: 2–4 s per song on an L40 GPU (excl. model load). Beats Gemini 2.5 Pro on strict boundaries (HR.5F .423) but Gemini is slightly better at ±3 s (.813).
MSAF	Classical framework: many boundary/label algorithms (novelty, spectral clustering, etc.).	Older algorithms; no 2024–26 numbers fetched unverified	MIT	Useful as a reference/ablation, not as the main engine.
librosa	Recurrence matrix, self-similarity, agglomerative clustering (`librosa.segment`). We already ship it.	Boundary quality depends on features; no benchmark of our own	ISC unverified	Cheap CPU baseline. Our building block for SSM-on-bar-grid.
MusicFM / MuQ / MERT	Self-supervised music encoders; the strongest structure models fine-tune on top of them (SongFormer, and the “MusicFM-Zhang” and “MuQiter” baselines in its table).	MusicFM-Zhang: ACC .725, HR.5F .640; MuQiter ACC .772 (cited in SongFormer, Harmonix subset)	MuQ code MIT, weights CC-BY-NC 4.0; MusicFM: repo LICENSE file says dual MIT / Apache-2.0 (GitHub shows NOASSERTION) verified; weights licence not checked unverified	Encoders are heavy; need a GPU in practice.
LinkSeg	Recent supervised structure model (baseline in SongFormer paper).	ACC .780, HR.5F .630, HR3F .762 (Harmonix subset)	unverified	Not evaluated by us.

Realistic accuracy and the honest ceiling

What the numbers mean

On the best public benchmark, a strict ±0.5 s boundary F-measure tops out near 0.70; relaxed ±3 s is near 0.78–0.80. An F of ~0.70 means a substantial share of strict boundaries are still missed or spurious even for the 2025 state of the art (F mixes precision and recall, so “one in three” is only a rough reading). MEASURED SongFormBench-Harmonix, HR.5F/HR3F.
Frame-level label accuracy ~0.80 MEASURED (ACC, SongFormBench-Harmonix) is with a seven-class label set. It drops as labels get finer.
Snapping predicted boundaries to the nearest bar line helps strict F, because annotators mark bar starts (estimate — to be proven in phase 0).

By genre — what we can and cannot say

The benchmarks are Harmonix (mostly pop/rock/electronic, unverified composition), RWC-Pop (J-pop) and 100 Chinese pop songs. No fetched number covers hip-hop, loop-based electronic, jazz or classical.
Our own 2026-10-09 prototype note: sections were “believable” on a beat-driven track and wrong (“Verse”) on orchestral music.
estimate Loop-based music (hip-hop, house) has structure carried by layers entering and leaving, not by harmony or melody changing. Chroma self-similarity is weak there; energy/stem-activity features matter more. Expect lower label accuracy and boundaries that are real but ambiguous (drop vs. build).
estimate Pop/rock with verse-chorus form and vocals is the best case.

The human ceiling is also below 1.0: annotators disagree on where sections start and what to call them (see risks). I could not fetch a sourced inter-annotator figure today unverified.

Commercial and competitor approaches

Product	What they do	Evidence
rekordbox	“Phrase analysis” marks track phrases; auto hot-cue/memory-cue placement uses it and is skipped when phrases cannot be analysed or the track is too short.	verified rekordbox 7 FAQ
Moises	“Chord Finder”: chords synced to playback, Easy/Medium/Advanced voicings, capo mode, editable, exportable chart. No accuracy figure published on the page.	verified moises.ai
Spotify audio analysis	Spotify’s 2024-11-27 notice lists Audio Analysis among endpoints removed for new apps (registered on/after that date) and for development-mode apps without a pending extension request; apps that already had access are unaffected. Not an option for us.	verified Spotify blog
Mixed In Key, Serato, Ableton, Logic, Cubase, Chordify, Songsterr, Cyanite, Musiio, AudioShake, LANDR, Beatport	Not fetched. Our own earlier note (2026-10-09) describes MIK/rekordbox/Splice features “based on established product knowledge”. Chordify’s site returned 403 to automated fetch.	unverified see unverified list

Take-away: phrase/section bands on the waveform are now table stakes in DJ software, but nobody publishes accuracy. Our differentiator is honest confidence + bar-exact edges on our own grid.

03Chord recognition & harmonic rhythm

System	Approach	Reported accuracy	Licence	Notes
Chordino / NNLS-Chroma (Mauch)	NNLS-chroma + HMM/Viterbi over a chord dictionary; Vamp plugin.	No fresh number fetched unverified	GPL-2.0 (repo) verified	Classic, interpretable, CPU. GPL means keep it a separate process or skip it.
madmom chords (deep chroma + CRF)	CNN chroma + CRF decoding.	unverified	Code BSD-style; model files CC BY-NC-SA 4.0 verified LICENSE	Non-commercial weights ⇒ off-limits for the paid product.
BTC (Park et al., ISMIR 2019)	Bi-directional Transformer; maj/min or large vocabulary.	MEASURED as a baseline in the ChordFormer table: “BTC+CNN” MajMinor 49.00 (the paper’s own re-run on its 1,217-song split; not BTC’s own published result). Training sets listed from memory unverified	Code MIT verified repo; weights/training audio status unverified	Old but usable; librosa-based features.
ChordFormer (Feb 2025)	Conformer on CQT, structured chord representation, re-weighted loss, CRF decoding to cut spurious transitions.	MEASURED 1,217-song Isophonics+Billboard+MARL set, 5-fold CV, WCSR: Root 84.7, MajMin 84.1, Sevenths 72.3, MIREX 83.6 (CNN+BLSTM baseline 83.4 / 82.6 / 69.8 / 81.5)	Weights/code release unverified	Gains over the CNN+BLSTM baseline in the same table are +1.3 (root), +1.5 (MajMin), +2.5 (7ths) points MEASURED. our reading progress is incremental.
Chord Recognition with Deep Learning (Dec 2025)	Thesis revisiting methods.	Findings: poor on rare chords; pitch augmentation helps; generative-model features do not	n/a	Confirms the plateau.
Essentia chord detection, autochord, CREMA	Template / deep options.	unverified	Essentia is AGPL-3.0 verified	AGPL is a network-copyleft risk for a hosted service.

How to stop beat-by-beat flicker

Decode on bars, not beats. Aggregate beat-synchronous chroma into per-bar vectors using the SANCT downbeats, then run Viterbi over bars. A bar is the natural unit of harmony in most pop/electronic music.
Hierarchical resolution choice. Per bar, allow three hypotheses — one chord, two half-bars, or four beats — and charge a penalty for each extra change. The finer split only wins when the evidence margin is large.
Phrase-aware prior. Changes are cheaper at bar 1 of a 4/8-bar phrase than mid-phrase; repeated phrases should decode to the same chords (use the section SSM to tie them).
Transition and self-loop costs (the HMM/CRF trick used by Chordino/madmom/ChordFormer). ChordFormer’s paper explicitly credits CRF decoding with reducing excessive transitions.

Harmonic-rhythm priors — what is and is not sourced

“Most pop is one chord per bar or per two bars” matches practice and our own demo (about 115 chords for a ~4-minute song, which is too dense), but I did not fetch a corpus study (e.g. McGill Billboard statistics, de Clercq & Temperley) unverified. Plan: measure chords-per-bar on the Billboard annotations in phase 0 and set the penalty from data, per genre bucket.

Annotation ambiguity caps all scores: even expert annotators disagree on chord labels, which ChordFormer cites as a cause of overlapping class boundaries. A sourced agreement percentage was not retrieved unverified, so the often-quoted “80–85% ceiling” is not used here.

04Local / segmental key detection

Topic	What we know	Tag
Key CNN (Korzeniowski & Widmer)	Genre-agnostic “AllConv” model, MIREX weighted score `w = r_c + 0.5 r_fifth + 0.3 r_relative + 0.2 r_parallel`. GiantSteps (electronic, Beatport previews): w 74.6 (correct 67.9, fifth 7.0, relative 8.1, parallel 4.1, other 12.9). Billboard: 85.1 (correct 79.9). Classical: 96.6. arXiv 1808.05340.	verified
Genre sensitivity	An earlier paper by the same authors finds systems tuned on pop perform subpar on electronic music, and that errors concentrate on relative and parallel keys. arXiv 1706.02921.	verified
Profile families	Essentia’s `KeyExtractor` exposes `diatonic, krumhansl, temperley, weichai, tonictriad, temperley2005, thpcp, shaath, gomez, noland, faraldo, pentatonic, edmm, edma, bgate, braw`; default bgate. Profile names such as `edma/edmm` suggest electronic-music-specific profiles; the docs page itself gives no accuracy comparison, and the claim that KK fits electronic worse rests on the Key-CNN authors’ cross-genre finding below, not on Essentia our inference. Essentia docs.	verified (names); accuracy by profile unverified
Local key / modulation	The Key-CNN paper itself notes that local-key performance on short excerpts depends on the harmonic coherence of the whole piece (1808.05340 abstract) — i.e. local windows need global context. Dedicated modulation-detection papers were not found in my search unverified. Truck-driver modulations (final chorus +1 or +2 semitones) are well known in pop unverified frequency.	mixed
Stem-aware chroma	No source fetched. Reasoning: bass stem gives a strong root estimate; “other”/harmonic stems remove drums and vocal melody-only passages. HPSS already approximates the drum removal. A test is in the ablation list.	estimate
Tuning	Our CQT chroma path uses librosa defaults; whether tuning is estimated per track was not confirmed in code unverified. Cheap to add: estimate tuning offset and re-centre before chroma.	unverified
MIREX audio key results	music-ir.org wiki pages were not reachable from this environment (connection failed). The key-CNN paper cites MIREX 2017 results.	unverified

What section structure adds to key (the core idea)

Pool across repeats. If chorus 1, 2 and 3 are the same section, sum their (energy-weighted) bar chroma before scoring. Noise from vocals/fills averages out; this is a bigger effective sample for the same key.
Per-section key from real sections, not four equal slices. A verse in the relative minor no longer drags a quarter-slice vote.
Modulation by transposed repetition. If the last chorus matches earlier choruses after rotating its chroma by k semitones (k=1,2) with high similarity, that is strong evidence of a key change, much stronger than a lone key-vote flip.
Bidirectional reconciliation. Whole-song key is the prior for each section (high self-transition in a 24-state key HMM over bars); a section overrides it only after ≥8 bars of consistent evidence.

4b · Hierarchical inference is not the same as shorter windows

These are two different ways to get “finer” harmonic output, and they pull in opposite directions. The report proposes the first and rejects the second.

Hierarchical harmonic inference (what we propose)

Keep the evidence window at a musical unit (a bar, from the SANCT grid) and decode a sequence of bar-level states with Viterbi.
Priors flow down the hierarchy: song key → section key → phrase position → bar. A change is cheap on bar 1 of a phrase and dear mid-phrase; diatonic chords are cheaper than chromatic ones; a harmonic-rhythm prior defaults to one chord per bar.
Evidence is pooled across repeats: choruses 1–3 share chroma/posteriors, so noise averages out as the effective sample grows.
A finer split (half-bar, beat) is accepted only when its likelihood gain beats a penalty. Resolution is earned, not assumed.

Just shrinking the window (what we avoid)

Cutting the analysis window (say from a bar to a beat, or a 4-slice key vote to 16 slices) gives each estimate less audio, so chroma is noisier, passing tones and vocal melody dominate, and the argmax flips.
With no transition cost or prior, each window is decided independently ⇒ flicker (this is exactly today’s chord_timeline: beat windows, argmax, no HMM, ~115 chords in a ~4-minute song).
For key, shorter slices make the vote less reliable, not more local: the Key-CNN authors note local-key classification of short segments has to consider the harmonic coherence of the whole piece.

Claim	Evidence	Label
Per-frame/window argmax over-produces chord changes; adding transition-penalised decoding fixes it	ChordFormer (arXiv 2502.11840): taking the top class per frame “often results in excessive transitions between adjacent chords, as it lacks a mechanism to impose penalties for immediate changes”; they add a linear-chain CRF for “more coherent chord transitions”. Chordino/NNLS also decodes with an HMM/Viterbi. We did not extract an isolated CRF-on/off score.	MEASURED (mechanism stated by authors) / no isolated number unverified
Local key needs whole-piece context	Korzeniowski & Widmer (arXiv 1808.05340), abstract: one must “consider the harmonic coherence of the whole piece when classifying the local key of short segments of audio”.	verified (statement), not a head-to-head
Shorter windows = noisier chroma	Signal-processing reasoning: fewer frames per estimate means higher variance, and a beat holds fewer harmonic cycles than a bar. No window-length sweep was fetched or run.	ESTIMATE (reasoned, not measured)
Pooling repeated sections improves key/chord posteriors	Standard averaging argument; no paper fetched that measures it for key. Depends on repeat clustering being right.	ESTIMATE
Harmonic-rhythm prior (about one chord per bar in pop/electronic)	Plausible and consistent with our own dense demo, but no corpus statistic was fetched.	unverified → ESTIMATE
Bar-level Viterbi beats beat-level argmax on our audio	Not measured. Falsifying test, to run in phase 0/1 (ablation 7): same chroma, same beat grid; compare (a) beat argmax, (b) shorter windows (half-beat), (c) bar Viterbi with priors. Report MajMin WCSR and chord changes per bar. The hierarchy claim fails if (c) does not beat (a) on flicker at equal or better WCSR.	ESTIMATE — hypothesis with a stated test

One caution cuts the other way: hierarchical priors can over-smooth real fast changes (jazz, passing chords) and wrong section clusters can tie unrelated bars together. That is why the sketch lets half-bar/beat hypotheses win on strong evidence and why low-confidence songs fall back to beat-level output.

05Beat / downbeat tracking as the grid

Tracker	Reported	Licence	Notes
SANCT beat_engine (ours)	No benchmark in repo. Downbeats come from a beat-phase vote over bars of 4 beats.	Ours	Sacred. librosa-based (HPSS tempo probes); no madmom/Beat This! dependency seen in `beat_engine.py`. Exposes `beat_times` and `downbeat_times`.
Beat This! (ISMIR 2024)	MEASURED GTZAN test-only set (models trained with 8-fold CV on other data): beat F1 89.1, downbeat F1 78.3 without DBN (77.4 with DBN, but better downbeat continuity: CMLt 73.3 vs 67.3). ~20 M params; the paper’s smaller model scores 88.8 / 77.2. Their ablation table on a validation split reports higher numbers (92.6 / 85.4) — do not mix the two.	MIT code and weights verified; note training data partly copyrighted	Authors admit failures on underrepresented genres and non-periodic beats; weaker continuity metrics. arXiv 2407.21658.
allin1 joint model	Harmonix: beat F1 .958, downbeat F1 .915 (different data from the row above — not comparable)	MIT code	Joint training helps all four tasks (ablation in paper).
madmom DBN	Our internal notes quote ~0.83–0.88 beat accuracy unverified source	Code BSD-style; models CC BY-NC-SA 4.0 verified	PyPI release only supports Python<3.10 and numpy<1.20, per Beat This! README verified; install from git.
Beat Transformer, SpecTNT-TCN, BEAST	MEASURED Beat Transformer: “up to 4% point improvement in downbeat tracking accuracy over the TCN architectures” (abstract; separate finding: demixed input beats non-demixed); BEAST: online, <50 ms latency, beat F1 80.04, downbeat F1 46.78	Abstracts only checked unverified beyond that	Research baselines; 2209.07140, 2312.17156.

Why downbeats matter so much here

Everything in the hierarchy — sections on bar lines, bar-level chords, 4-bar phrases — assumes the bar phase is right. If the downbeat is one beat off, every bar chroma straddles two real bars and chords smear. Public downbeat F-measures (MEASURED ~0.78 on GTZAN for Beat This!, ~0.92 on Harmonix for allin1; different datasets) say a meaningful fraction of songs will have an off-phase or wrong-meter grid. Phase 0 must measure SANCT downbeat accuracy against annotated data (read-only), and the pipeline should expose a bar-phase confidence so low-confidence songs fall back to beat-level or whole-song output.

SANCT rule for this project

New code consumes the SANCT grid; it never re-derives, snaps or reorders it. Other trackers (Beat This!, allin1) may run as diagnostics to flag grids that look wrong; they must not feed back into cut points. Section/chord boundaries are expressed as indices into the SANCT bar list, so they cannot drift from the grid by construction.

06Proposed architecture for ProduceHits

Hierarchy: song → sections → phrases (4/8 bars) → bars → half-bars → beats. Each level constrains the one below it with a prior and passes a confidence up.

Green = sacred input, blue = new CPU code, orange dashed = optional GPU step.

Bar-level chord decoding (sketch)

for each bar b:   x_b      = mean(beat chroma in bar b)          # 12-d, RMS weighted
                  x_b,h    = [mean(first half), mean(second half)]
                  x_b,beat = beat chroma (4)
score(chord c, span s)     = cosine(x_span, template_c)           # triads now, 7ths later
cost(bar b, hypothesis h)  =  - sum_span score  +  lambda_change * (#changes in bar)
                             + lambda_phrase * [change not on phrase bar 1 or bar 3]
Viterbi over bars with transition cost  T(c_prev, c) = 0 if same, mu if diatonic to key, mu + nu otherwise
choose finer split (2 or 4) only if cost drops by more than a margin  =>  default is 1 chord per bar
tie repeated phrases: bars in the same section-cluster share chord posteriors (soft sum)

Parameters (lambda_change, lambda_phrase, mu, nu) are fitted by grid search on Billboard/Isophonics-style annotations, per genre bucket. design proposal

“Don’t over-segment” rules

Minimum section length 4 bars; boundaries only on SANCT bar lines.
Cap sections to roughly one per ~8 bars on average; merge neighbours whose bar-SSM similarity exceeds a threshold.
Keep a boundary only if supported by novelty above a threshold or by two independent voters (SSM, allin1, SongFormer).
Chords: minimum duration of one beat, default one per bar; show chords faded by confidence (already the product guidance from 2026-10-09).
Low-confidence songs (odd meter, no stable bar phase, very short) return whole-song key + beat-level chords only, labelled as such.
Don’t force Verse/Chorus words. If the label model and the SSM disagree, show neutral labels (A, B, C… or Low/Rising/High energy for beatless material).

6a · Recommendation table

Library	Task	Accuracy (reported)	Licence	Speed	Verdict
librosa (ours)	chroma, SSM, clustering	n/a (feature tool)	ISC unv.	CPU, seconds	USE backbone of phases 1–2
mir_eval (pinned)	scoring: segment, chord, key, beat	n/a	MIT unv.	trivial	USE benchmark harness
allin1	sections + labels + downbeats	HR.5F .60–.66, ACC .74 (Harmonix)	MIT code; weights unstated	~27 s of audio/s on 4090; ~9–12 s/song L40 (SongFormer README)	PILOT on Modal T4 after weights-licence check
SongFormer	sections + labels	HR.5F .70, HR3F .78, ACC .81	Code CC-BY-4.0; MuQ weights CC-BY-NC	2–4 s/song on L40 (README)	GATE evaluate offline; no prod until licence cleared
MSAF	classical segmentation	older, unmeasured here	MIT	CPU	REFERENCE ablation baseline only
Beat This!	beats/downbeats	beat F1 89.1, downbeat F1 78.3	MIT code+weights	~20 M params; CPU speed unv.	DIAGNOSTIC cross-check of SANCT bar phase only
madmom	beats, chords, key	various	models CC BY-NC-SA	CPU	AVOID in prod (NC models; install friction)
Chordino / NNLS	chords	unverified	GPL-2.0	CPU	AVOID linking; could be an offline reference only
Essentia	key profiles, chords	unverified	AGPL-3.0	CPU	AVOID as a dependency; borrow the idea of edma/edmm/bgate profiles (re-derive from papers)
BTC	chords (maj/min, large vocab)	baseline in ChordFormer table	MIT code; weights unv.	small model	PILOT as a learned alternative to templates
ChordFormer	chords (large vocab)	MajMin 84.1, 7ths 72.3	release unv.	unv.	WATCH check for released weights
Key CNN (Korzeniowski)	global key	GiantSteps w 74.6; Billboard 85.1	unv.	small CNN	WATCH use paper as the benchmark to beat
Spotify audio analysis	sections	n/a	closed to new apps	n/a	NO

6b · Ablations and benchmark plan

Ablations

Key: full-mix HPSS (today) vs harmonic stem vs bass+other stems vs no HPSS.
Key: KK vs Temperley vs edma/edmm-style profiles vs an ensemble vote.
Key: flat global vs pooled-by-repeat-cluster vs bar-level key HMM.
Key: with and without per-track tuning estimation.
Sections: equal slices (today) vs energy novelty (demo) vs bar-SSM vs allin1 vs SongFormer vs 2-of-3 fusion.
Sections: effect of snapping to SANCT bars on HR.5F.
Chords: beat argmax (today) vs bar Viterbi vs bar+half-bar+beat hypotheses vs + phrase prior vs + key prior.
Bar phase: SANCT downbeats vs Beat This! downbeats vs allin1 downbeats (agreement and chord accuracy impact).

Datasets and metrics

Keys: GiantSteps Key (Beatport previews, electronic) and Billboard-type sets as used in the key-CNN paper; metric mir_eval.key.weighted_score plus the raw correct/fifth/relative/parallel split.
Chords: Isophonics/Beatles, McGill Billboard, RWC-Pop; mir_eval.chord root, majmin, sevenths WCSR; plus our own chords-per-bar and flicker rate.
Sections: Harmonix Set / SongFormBench (200 Harmonix + 100 Chinese songs), SALAMI, RWC-Pop; mir_eval.segment detection P/R/F at 0.5 s and 3 s, pairwise F, V-measure.
Beats: Harmonix beats/downbeats; mir_eval.beat F, CMLt, AMLt, downbeat F.
Internal set (new): 100–150 tracks from the founder’s own catalog and licensed/public-domain audio, hand-labelled for key, modulations and sections, balanced across pop/rock, hip-hop, house/techno, orchestral/other.

Most public sets distribute annotations only; audio must be sourced separately. Keep any benchmark audio private and out of the repo and out of demos.

6c · Compute and licence notes

Where it runs: everything is worker-side. Phases 1–2 are CPU on the auto-scaled Hetzner workers, adding an estimated <1–2 s per song on top of the measured ~13.6 s / ~1 GB baseline (estimate: SSM on ~100–200 bars and a 24-state Viterbi are tiny). The optional deep step is GPU-only on Modal T4. The API origin never touches audio.
Demucs reuse: allin1 runs Demucs internally. We already produce stems for paid users, so a pilot should check whether stems can be passed in to avoid a second separation unverified. Free-tier songs have no stems; the CPU path must work from the mix.
Pinned versions: requirements.txt has torch 2.6.0 while modal_gpu.py pins torch 2.4.1; SongFormer’s requirements pin torch 2.4.0. allin1 needs NATTEN builds matched to torch/CUDA. Budget integration time for a dedicated Modal image.
Licence flags: (1) MuQ weights CC-BY-NC → SongFormer commercial use unresolved. (2) madmom models CC BY-NC-SA. (3) Essentia AGPL-3.0 and NNLS-Chroma GPL-2.0 are copyleft. (4) Weights of allin1/BTC/ChordFormer are not stated in what I fetched. (5) Beat This! is MIT but notes some training data is copyrighted or under limited CC licences. None of this is legal advice; get a one-hour review before any model ships.
Credits / storage laws: new analysis is part of the existing bpmandkey_analyze job; results go into the existing analysis JSONB on processed_songs (additive fields). No new job type is needed (Track 12 plan). Credit pricing for any extra GPU step must go through cost-guard measurement first.

6d · Phased roadmap

Phase	Deliverable	Effort	Expected gain ESTIMATE (not measured)	Gate
0 · Measure	mir_eval harness; baseline for current key, chord flicker, SANCT downbeat vs Harmonix; internal labelled set	3–5 days est.	A real accuracy number for key (replaces “very accurate”); bar-phase reliability	Datasets sourced
1 · Bar-level chords	Viterbi with 1/2/4-resolution, phrase prior, key prior; behind a flag, same JSON shape plus `bar`	1 week	Large drop in chord changes per bar (target ≤1.2 on pop); MajMin gain of a few points likely est.	Phase 0 metrics
2 · CPU sections + pooled key	Bar-SSM sections snapped to bars, repeat clusters, pooled per-section key, key HMM, modulation flag	1–2 weeks	Per-section keys meaningful; whole-song key +1–3 pts on mixed material est., unproven; detect final-chorus modulations	Phase 0 + 1
3 · Deep second opinion	allin1 and SongFormer offline eval on Modal T4; fusion rules; licence decision	1–2 weeks	Boundary F at 0.5 s from ~0.5–0.6 (CPU, est.) toward 0.65–0.70; real labels	Licence cleared
4 · Better chords	Learned model (BTC / ChordFormer if weights are usable) vs templates; 7ths; bass-stem root prior	2–3 weeks	Sevenths and inversions; at most low single-digit WCSR over a tuned template Viterbi est.	Phase 1 baseline
5 · Product surface	Section bands, per-section key chips, modulation badge, confidence fading, CSV with bars	1 week	Trust & usability	ux-polish gate

07Honest risks

Label ambiguity

Functional labels (verse/pre-chorus/chorus/bridge) are subjective and schema-dependent. SongFormer’s whole premise is training with “schema-mismatched labels”. Expect disagreement even among experts; ship neutral labels when models disagree.

Electronic and loop-based music

Sections are defined by layers and energy, not by harmony. Chroma-SSM finds a loop repeated for 64 bars and calls it one section. Key profiles tuned on pop fit electronic worse (key-CNN papers). A single-chord or drone track makes chord Viterbi and key pooling nearly meaningless.

Tempo changes and time signatures

SANCT assumes stable bar groups of 4 beats. 3/4, 6/8, 5/4, meter changes and rubato break the bar tensor. Beat This! authors themselves note failures on non-periodic pieces. Detect and fall back.

Lyrics-driven structure

Verses often share a melody and chords but differ in lyrics; choruses can be defined by hook text. Audio-only models partially capture this. Gemini-style lyric-aware models score higher at ±3 s but worse strict boundaries (SongFormer table).

Over-segmentation and false modulations

Every extra boundary or key flip that a user sees and finds wrong costs more trust than a missed one. Hence the 4-bar floor, the 8-bar evidence rule and the whole-song fallback.

Benchmark and licence risk

No existing key ground truth; models trained on Harmonix may overfit pop/EDM; some weights are non-commercial. Estimated gains in this report are not measurements until phase 0 runs.

—Claim ledger (decision-driving claims, re-checked 2026-10-18)

Each source was re-opened (paper PDF/abstract, repo README or LICENSE, product page) and the number or licence compared with the claim. “Fixed” = the report text was corrected or softened in this validation pass.

Claim in report	Label	Source check	Status
SongFormer HR.5F .703, HR3F .784, ACC .807 on SongFormBench-Harmonix	MEASURED	SongFormer README table (also arXiv 2510.02797): HX variant .703/.784; ACC .807 belongs to the HX+E+H+G variant	Confirmed; variant caveat added
allin1 Harmonix 8-fold: beat F1 .958, downbeat .915, HR.5F .660, label PWF .738	MEASURED	arXiv 2307.16425 Table 1 (All-In-One 300K: .958 / .915 / .660 / .738)	Confirmed
allin1 on SongFormBench-Harmonix: HR.5F .596, HR3F .730, ACC .740	MEASURED (by SongFormer authors)	SongFormer README	Confirmed
allin1 speed 33 min audio in 73 s on RTX 4090	MEASURED (README)	allin1 README	Confirmed
SongFormer 2–4 s/song on L40; allin1 9–12 s	MEASURED (SongFormer authors, README)	README speed table (excl. model load)	Confirmed; GPU is L40, not our Modal T4 unverified on T4
ChordFormer MajMin 84.09, Sevenths 72.28, Root 84.69, MIREX 83.62; baseline CNN+BLSTM 82.62/69.78	MEASURED	arXiv 2502.11840 Table II; dataset 1,217 songs (Isophonics, Billboard, MARL), 5-fold CV, WCSR	Confirmed; dataset/CV wording added. Earlier note that the baseline row was “83.4/82.6/69.8/81.5” matches
Beat This! beat F1 89.1, downbeat F1 78.3 (77.4 with DBN)	MEASURED	arXiv 2407.21658 Table 2 (GTZAN test set)	Fixed: dataset was mislabelled as “broad 8-fold set”; removed unverified “78 MB”
Key-CNN GiantSteps w 74.6 / Billboard 85.1 / Classical 96.6	MEASURED	arXiv 1808.05340 results table	Confirmed
Local key needs whole-piece harmonic coherence	verified	1808.05340 abstract	Confirmed; now cited in 4b
Pop-tuned key systems underperform on electronic; errors are relative/parallel	verified	arXiv 1706.02921 abstract	Confirmed
MuQ weights CC-BY-NC 4.0, code MIT	verified	MuQ README “License”	Confirmed. Gap: SongFormer’s exact checkpoint not traced, so “commercial blocker” softened to “likely”
madmom models CC BY-NC-SA 4.0; code BSD	verified	madmom LICENSE file	Confirmed
NNLS-Chroma GPL-2.0; Essentia AGPL-3.0	verified	COPYING files in each repo	Confirmed. The “AVOID” verdicts are our legal-risk judgement, not legal advice
allin1 / BTC / Beat This! / MSAF code MIT	verified	GitHub licence API / LICENSE / Beat This! README (MIT code and weights; notes some training data copyrighted)	Confirmed
Spotify Audio Analysis closed to new apps	verified	Spotify 2024-11-27 blog	Reworded to match the post exactly
rekordbox auto cues use Phrase analysis; skipped if phrases cannot be analysed / track too short	verified	rekordbox 7 FAQ	Confirmed
Moises Chord Finder, no published accuracy number	verified	moises.ai page: only “most users find… quite accurate”	Confirmed
Our pipeline: ~13.6 s and ~0.9–1.1 GB per song	MEASURED (internal)	docs/reports/2026-10-09-song-intelligence-strategy.md table (13.6 s; 888 MB / 1,065 MB peak)	Confirmed
Phase 1–4 expected gains; +<1–2 s CPU; CPU boundary F 0.5–0.6; whole-song key +1–3 pts; chords/bar ≤1.2	ESTIMATE	No source. Projections only; the phase-0 harness exists to test them	Labelled
Bar snapping raises strict boundary F	ESTIMATE	No source	Labelled
Hierarchical decoding beats shorter windows (4b)	ESTIMATE with partial support	Mechanism stated by ChordFormer and Key-CNN authors; no head-to-head	Stated as hypothesis + falsifying test

Not checked: MIREX results, weights licences for allin1/SongFormer/BTC/ChordFormer, any competitor beyond the three listed (see Unverified list).

—Sources

All fetched on 2026-10-18 unless noted.

Kim & Nam, All-In-One Metrical and Functional Structure Analysis (WASPAA 2023): arxiv.org/abs/2307.16425; code github.com/mir-aidj/all-in-one (MIT).
SongFormer: arxiv.org/abs/2510.02797; code github.com/ASLP-lab/SongFormer (CC-BY-4.0 code).
MuQ: github.com/tencent-ailab/MuQ (code MIT, weights CC-BY-NC 4.0). MusicFM: github.com/minzwon/musicfm.
MSAF: github.com/urinieto/msaf (MIT).
Beat This!: arxiv.org/abs/2407.21658; github.com/CPJKU/beat_this (MIT).
Beat Transformer 2209.07140; SpecTNT 2205.14701; BEAST 2312.17156 (abstract-level only).
madmom: github.com/CPJKU/madmom, LICENSE (code BSD-style; models CC BY-NC-SA 4.0).
ChordFormer: arxiv.org/abs/2502.11840. Chord Recognition with Deep Learning: arxiv.org/abs/2512.22621. BTC: arxiv.org/abs/1907.02698, code (MIT).
NNLS-Chroma/Chordino: github.com/c4dm/nnls-chroma (GPL-2.0).
Essentia: github.com/MTG/essentia (AGPL-3.0); KeyExtractor.
Korzeniowski & Widmer, Genre-Agnostic Key Classification with CNNs: arxiv.org/abs/1808.05340; End-to-End Musical Key Estimation: arxiv.org/abs/1706.02921.
Spotify Web API changes, 2024-11-27: developer.spotify.com.
rekordbox 7 FAQ (phrase analysis): rekordbox.com. Moises Chord Finder: moises.ai.
Repo (internal): afcut_engine/harmonic.py, webapp/app.py (_detect_key, _harmonic_layer), webapp/beat_engine.py (read-only), tools/song_intel_demo.py, docs/plans/2026-09-14-track12-bpmandkey-harmonic-analysis.md, docs/reports/2026-10-09-song-intelligence-strategy.md, requirements.txt, modal_gpu.py.

—Unverified list

MIREX structure, chord and key results (music-ir.org not reachable from the research environment).
Weights licences for allin1, SongFormer, BTC, ChordFormer; whether allin1 loads madmom model files at runtime.
Accuracy of Chordino, madmom chords, Essentia, autochord, CREMA; any 2024–26 chord or key model other than ChordFormer and the 2025 thesis.
Commercial tools beyond rekordbox, Moises and Spotify: Mixed In Key, Serato, Ableton, Logic, Cubase, Chordify, Songsterr, Cyanite, Musiio, AudioShake, LANDR, Beatport.
Inter-annotator agreement for sections and chords; harmonic-rhythm statistics by genre; frequency of pop modulations; any dedicated modulation-detection literature.
Harmonix Set genre composition; librosa, mir_eval, Demucs licences (from memory); Beat This! CPU speed; allin1 stems pass-through; T4 vs L40 speed for SongFormer.
Whether our CQT path estimates tuning; accuracy of SANCT downbeats; the two calibration tracks’ details.
All “expected gain” figures in the roadmap are estimates, not measurements.

Repo	Licence (GitHub)	Last push	Stars	Open issues*	Notes
mir-aidj/all-in-one	MIT	2024-05-09	862	20	Last commit 2024-05-09; open issues/PRs run into 2026, so the community is live (not abandoned). Open: NATTEN incompatibility (#30, PR #36 Aug 2025 / PR #39 Sep 2026), “uses too much CPU” (#38), commercial use question (#40 Oct 2026), Harmonix label inaccuracies (#34), RTX 5090 incompatibility (#35). Weights licence unverified.
ASLP-lab/SongFormer	NOASSERTION	2026-05-14	181	6	Active-ish. #9 checkpoint licensing unanswered; #5 duplicate section labels; PR #8 VRAM work in draft.
urinieto/msaf	MIT	2026-05-13	560	22	Classical algorithms (SF, Foote, C-NMF). No chords or key.
CPJKU/beat_this	MIT	2026-05-28	428	4	Beats/downbeats only. Possible cross-check for the SANCT grid, not a replacement.
jayg996/BTC-ISMIR19	MIT	2020-05-23	215	10	Chord model, 6 years untouched; training data licence unverified.
CPJKU/madmom	NOASSERTION	2026-03-20	1727	81	Models CC BY-NC-SA (Research tab).
tencent-ailab/MuQ	MIT (code)	2025-08-04	390	3	Weights CC-BY-NC per Research tab.
MTG/essentia	AGPL-3.0	2026-10-14	3775	443	AGPL: do not link into the product.

Item	What it is	What it is NOT
#34 Label inaccuracies in the Harmonix Dataset	One unresolved reporter questions Harmonix section annotations ANECDOTAL	Not proof of dataset-wide label inaccuracy; no measurement, no sample size
#35 use of the NVIDIA GeForce RTX 5090	torch 2.5 + cu121 / RTX 5090 unsupported compatibility report	Not an accuracy result; says nothing about model quality
PR #36 (Aug 2025) NATTEN compatibility fix and working Dockerfiles	An open, unmerged proposed fix	Not merged, not released, not verified by us
PR #39 (Sep 2026) pure-PyTorch CPU/MPS neighborhood attention, NATTEN optional	Unmerged proposal for a CPU/MPS fallback	Two proposed fixes (#36/#39) plus 2026 activity show a live community, not proof the project is maintained or that a release ships
#40 (Oct 2026) commercial use in an iOS app	An unanswered licensing question	Not proof permission was denied or granted; the weights licence stays unverified

#	File (existing)	Change	Needed?	Risk
T0	none	Poller inside the new job: a periodic `song_structure_scan` picks `processed_songs` rows with no `song_structure` row.	Alternative to T1	Low. Zero edits to the SANCT path. Costs up to a poll interval of latency.
T1	`shared/worker_daemon.py`	Register the new job type’s handler in the dispatcher (an import and a mapping entry). Required in any option: the daemon is where jobs are claimed.	Yes	Medium. Shared by every job; a bad import would break all workers. Mitigate: lazy import inside the handler, a unit test that the daemon imports, and the existing kill switch pattern (env flag).
T2	`shared/scaler.py`	Add `song_structure` to `LIGHT_JOB_TYPES` (so it never rents a worker-hour alone, same reasoning as `track_preview`). Without this the job either never wakes a cold fleet or, if put in HEAVY, spends worker-hours.	Yes	Medium. Wrong class = stuck queue or a billing leak. Choose LIGHT and run the cost-guard check.
T3	`shared/models.py` + new alembic revision	New `song_structure` model/table only (db-migration skill). No change to existing models.	Yes	Low if additive. Not created now.
T4	`webapp/app.py`	One read-only GET route and, for any paid tier, a `CREDIT_COSTS` entry. Router could live in a new file mounted by one line.	Yes	Low. Reads a new table. Must not compute (origin never processes).
T5	`webapp/static/js/bpmandkey.js`, SongChop view	Render the lanes; fetch is additive.	Yes (UI)	Low-medium. Must obey the frontend law and not alter loop cut logic.
T6 (optional)	end of the `full_pack` handler	One enqueue line instead of the poller.	No: choose T0	Medium. It sits next to SANCT code; avoid.

Law	How the design complies
API origin enqueue-only	Compute only inside the queued job on an auto-scaled worker. The GET route reads a table. No BackgroundTasks, no subprocess on the origin.
Credits law	Cache by `audio_hash` (unique key on the new table); idempotent `idem:` ref_id if ever charged. Recommended: free by design like `track_analyze`, cost shown as “Already analysed”. Result persisted in Postgres.
Storage routing	Reads the stored FLAC from StorageBox (never served to browser). Result is JSON in Postgres; no audio written, no new WAV/FLAC.
SANCT law	No change to cut points or slicing. SongChop uses sections only as labels and suggestions over existing bar indices.
DB law	New table via alembic (db-migration skill), additive, reversible.

Option	Extra CPU-s	Peak RAM	Wall	€/song	Basis
Current BPM&Key full analysis	~13.5	0.9–1.1 GB	~14 s	~€0.0001	MEASURED (strategy report)
Own analyzer, CPU (fresh chroma pass, shares decode)	+4 to +8	+0.3 to +0.8 GB (child process, same cap)	+4–8 s	+€0.00003–0.00006	ESTIMATE: existing chord step is 2.6 s, section-key 2.6 s, our synthetic maths 1.8 s (I6); real CQT costs more than the prototype’s STFT
Own analyzer, reusing saved harmonic features	+1 to +3	~+0.3 GB	+1–3 s	<€0.00003	ESTIMATE. Needs the chroma to be saved by the existing path, which would add a touch point; not assumed.
allin1 on CPU	60–200	2–4 GB	1–3 min	€0.0004–0.0014	ESTIMATE. Not benchmarked. Issue #38 reports heavy CPU use (title read only). Whether it re-runs stem separation by default: unverified.
SongFormer on Modal GPU (T4)	GPU 5–20 s + cold start	GPU VRAM several GB (PR #8 is about VRAM)	10–60 s with cold start	~€0.002–0.01	ESTIMATE. T4 hourly rate and model speed not checked. Needs a gateway job and licence clearance.

Stage	Seconds
STFT	0.99
Beat-sync chroma	0.09
SSM + novelty	0.002
Chord Viterbi (120 bars × 24 states)	0.002
Per-section key	0.01
Process CPU total (incl. signal synthesis)	1.78
Peak RSS	877 MB

Product	Consumes	Shows
BPMandKey	sections[], key_segments[], chords per bar	Structure timeline, per-section key chips (with “modulates to …” marker), chord-per-bar strip, confidence shading. Whole-song key stays as today, and is shown next to the sectioned one so users can compare.
SongChop	sections[] edges (bar indices), existing loops	Section bands over the waveform; “suggest loops from the chorus” picks existing SANCT bar ranges inside a section. Never writes cut points, never moves a loop boundary; a suggestion is a pair of bar indices handed to the existing loop code.
SongTranscribe	chords per bar	Cleaner chord chart (Research tab’s main win).
Remix / PodDJ / others	sections	Arrangement blueprints: the strategy report names remix blueprint prep as a consumer ESTIMATE (not scoped here).

Failure mode	Design mitigation
Key wrong on electronic / modal / ambiguous music	Return top-3 with margin; allow “indeterminate” (the existing engine already has an indeterminate path); show confidence, never a bare label.
Relative major/minor confusion	Always output the relative key and its margin; disambiguate with chord-root prevalence and bass/tonic-chord evidence per section; mark low margin visibly.
Over-segmentation	Minimum section length of 4 bars, edges only at downbeats, novelty threshold adaptive to the song, merge sections with the same repetition cluster; hierarchy (phrase → section).
Chord flicker	Bar-level states, high self-transition prior, beat-level change only when evidence is strong, majority pooling across repeats of a section.
Wrong downbeat (bar shifted by 1–3 beats)	Sidecar cannot fix SANCT. Correct SANCT beats do not automatically establish downbeats or bar phase, so the phase-0 audit must verify downbeats independently — without altering the grid. Detect inconsistency (chord changes landing mid-bar systematically) and flag `grid_confidence` low; offer a UI “bar offset” that only shifts analysis labels, never audio cut points.
Tempo changes / free tempo	Work in bars from the grid; if beat intervals drift beyond a threshold, split into segments and analyse each, or mark “unsupported”.
Modulation false positives	Require a sustained change over ≥ 8 bars, win by margin, and relative-key/chord support before showing a modulation.
Stale or corrupt result	Version field; recompute on mismatch; `status: failed` shown as “analysis unavailable”, never blocking SongChop.

Phase	Scope	Effort ESTIMATE	Gate to proceed
0 Benchmark & ablation	Private labelled set (Harmonix annotations plus ~30 own tracks, audio kept out of repo). Score: boundary F, chord WCSR, key accuracy. Ablate hierarchy vs shorter windows (open question from the Research tab), own segmenter vs allin1 vs SongFormer (offline only), audit SANCT downbeat accuracy.	3–5 days	Own segmenter within ~10 pts of the best on HR3F, or a clear plan; licence decision for SongFormer.
1 Sections	Sidecar job, table, migration, touch points T1–T4, GET route, BPMandKey timeline. Bar-snapped sections, cluster labels, unknown allowed.	5–8 days	Beta songs pass visual review; queue and memory unchanged (measured p50, p95).
2 Chords	Bar-level Viterbi, simplified/extended toggle, chord strip UI, SongTranscribe chart.	4–6 days	Flicker rate and WCSR at or above current chord timeline on the benchmark.
3 Per-section key / modulation	Pooled section key, relative-key margin, modulation marker.	3–5 days	Beats whole-song key on a modulating subset; no regression on the rest.
4 Per-stem (later)	Bass-stem root evidence for chords and key, drums for downbeat cross-check. Needs stems (paid, GPU). Not started until 1–3 prove value.	1–2 weeks	Measured gain vs the extra stem cost.
Backfill	See I10; after phase 1 gate.	1–2 days	Cost-guard sign-off.

00Summary

Recommendations

How to read the numbers

01How our key detection works today

Why it is accurate (and where that is a claim, not a measurement)

Data we have to benchmark it

02Section / structure segmentation

Libraries and models

Realistic accuracy and the honest ceiling

What the numbers mean

By genre — what we can and cannot say

Commercial and competitor approaches

03Chord recognition & harmonic rhythm

How to stop beat-by-beat flicker

Harmonic-rhythm priors — what is and is not sourced

04Local / segmental key detection

What section structure adds to key (the core idea)

4b · Hierarchical inference is not the same as shorter windows

Hierarchical harmonic inference (what we propose)

Just shrinking the window (what we avoid)

05Beat / downbeat tracking as the grid

Why downbeats matter so much here

SANCT rule for this project

06Proposed architecture for ProduceHits

Bar-level chord decoding (sketch)

“Don’t over-segment” rules

6a · Recommendation table

6b · Ablations and benchmark plan

Ablations

Datasets and metrics

6c · Compute and licence notes

6d · Phased roadmap

07Honest risks

Label ambiguity

Electronic and loop-based music

Tempo changes and time signatures

Lyrics-driven structure

Over-segmentation and false modulations

Benchmark and licence risk

—Claim ledger (decision-driving claims, re-checked 2026-10-18)

—Sources

—Unverified list

Legend (same as the Research tab)

I0Verdict: HYBRID, own analyzer first

Recommendation

I1Evidence for the verdict

Why not “adopt everything”

I1b · Bounded issue-level evidence (root web_fetch, 2026-10-18)

Bounded mitigations

I2Sidecar design

I3Touch points: the ONLY places existing files would change

I4Repo laws check

I5Resource analysis

Current cost MEASURED (repo docs)

Incremental cost per song, by option

I6Prototype measurement MEASURED

What this does and does not show — read before quoting any number above

I7Product integration

I10Library-wide backfill

I8API response shape (design)

I9UI graph spec

I11Failure modes → mitigations

I12Phased roadmap

I13Sources, what is verified and what is not