Matching guitar signal with tablature, near real-time algorithms? I'm looking for candidate algorithms that might work for a use-case comparable to Rocksmith's analysis system. To compare a clean electric guitar signal with a tablature reference.

A higher latency is acceptable, as well as compromises in accuracy, but the goal is to have near real-time information on whether what you actually played, is within a threshold of what you were expected to play.

The output would be a probability value per expected strum (or pluck or tap). Which is then matched against a configurable threshold.

My current line of thinking is to take multiple algorithms and predetermine which algorithm should be used per segment of the song.


One for chords (compare lowest note &amp; 12-bin PCPs?)
One for sustained / single notes (FFT cross-correlation?)
One for fast or polyphonic riffs (Supervised training? Classification models? Don't know.)
One for bends... etc.


My two question are:


Is a (context-sensitive) combination of sub-algorithms required for such a use-case?
Which algorithms would make good candidates for the individual tasks, given this use-case?


Bonus question:
do any "master" algorithms already exist to do such switching / tweaking of sub-algorithms?



Extra details,

What is considered acceptable processing time depends on the reference. For example if you're instructed to play a sustained single note for 3 bars, it's perfectly acceptable to do a slow analysis of the plucking envelope.

However if you're instructed to play a solo, alternating picking and hammering, 1/16th notes and bends. You're working with shorter samples, more human error and residual noises while playing. But we still need to attribute a binary decision if it was "close enough" for each note within, say ~200ms after each note was supposed to be released.

The guitar signal comes directly from a dedicated USB guitar audio interface, making it single instrument and a good quality signal.

The tablature is highly detailed. Assume a modern guitar pro format. Making relatively accurate training samples possible, but they will not contain the characteristics of the exact guitar, pickups, noise, etc. that the guitar signal would have.
Assuming that the tablature is well defined; let's say it's a MIDI file complete with pitch bend commands.  you can make a synthesizer where each note is actually a tracking comb filter tuned to the midi note (and offset by the pitch bend command) with notches at each integer harmonic (including the fundamental).  if those tracking comb filters processes a sound file playing that MIDI file and the two are properly aligned in time, the tracking comb filters should null out most of the input audio, even polyphonic.

Then magnitude-square-LPF the result and a minimum value means a pretty good match.

Just an idea.  No equations, but I have posted about pitch detection before. And I thought I did about tuned comb filters, too (you need a precision delay line to tune the comb filter to whatever fine-tuned pitch that is specified).