Context-Independent Polyphonic Piano Onset Transcription with an Infinite Training Dataset
Samuel Li
stat.ML cs.SD

	arXiv Preprint
Samuel Li

	Spectroscopy of very hot plasma in non-flaring parts of a solar limb active region: spatial and temporal properties
    Helen E. Mason
    December 30, 2023
===================================================================================================================



    
    
    
    	Many of the recent approaches to polyphonic piano note onset transcription require training a machine learning model on a large piano database.
        However, such approaches are limited by dataset availability; additional training data is difficult to produce, and proposed systems often perform poorly on novel recording conditions.
        We propose a method to quickly synthesize arbitrary quantities of training data, avoiding the need for curating large datasets.
        Various aspects of piano note dynamics — including nonlinearity of note signatures with velocity, different articulations, temporal clustering of onsets, and nonlinear note partial interference — are modeled to match the characteristics of real pianos.
        Our method also avoids the disentanglement problem, a recently noted issue affecting machine-learning based approaches.
        We train a feed-forward neural network with two hidden layers on our generated training data and achieve both good transcription performance on the large MAPS piano dataset and excellent generalization qualities.
    

    
    music information retrieval (MIR), neural networks, data modeling
    



    

§ INTRODUCTION

    Polyphonic music transcription involves extracting a musical score or equivalent representation from an audio recording.
    In particular, the problem of polyphonic piano onset transcription involves extracting the onset time and pitch of many potentially simultaneous piano notes.
    Deep neural networks have been successfully applied to this area, but current approaches require the use of large, painstakingly annotated datasets as training data <cit.> — more often than not, the extensive MAPS piano database <cit.>.
    However, curating additional training data can be both time-consuming and challenging <cit.>, and the original setup used to create these datasets cannot be accurately reproduced should additional samples be needed.
    In addition, many of these machine learning approaches are both trained and evaluated on samples drawn from the same database <cit.>, weakening claims about generalization behavior; networks trained on one dataset tend to overfit its specific timbre and perform relatively poorly on newly generated data <cit.>.
    It has even been recently noted that neural networks face a fundamental issue when applied to polyphonic note transcription — they suffer from the entanglement problem, memorizing chords or combinations of notes rather than learning to report the onset of each note individually <cit.>.
    
    We circumvent all of these problems by generating our training data procedurally.
    No annotated piano database is used as training data.
    Although we evaluate our approach on the MAPS piano database, the instruments and recording conditions used for testing are completely unknown to the network, providing a high degree of confidence in our model's generalization capabilities.
    Our approach is completely context-independent — that is, we require no prior information about the instrument or recording being transcribed, allowing a broader field of application.
    Furthermore, we solve the disentanglement issue presented in <cit.> by randomly generating arbitrary combinations of notes, forcing the network to learn to identify individual notes.
    
    

§ PROPOSED MODEL

    

 §.§ Data Representation

    
    We use a constant-Q transform (CQT) spectrogram as the fundamental time-frequency representation of our audio signals <cit.>.
    We use bins ranging from the note G_1 ≈49Hz to C_8 ≈4186Hz, with a spacing of 12 bins per octave, for a total of 79 frequency bins.
    We use a Q-factor of 32.
    
    Our spectrogram frames are spaced 1024 audio samples apart; for the audio in the MAPS database, which has a sample rate of 44100Hz, this leads to a frame rate of about 43Hz.
    
    For our machine learning model, we use a simple feed-forward neural network.
    The network's input consists of an 8-frame wide “reading window” of the magnitude of the CQT spectrogram, normalized to have a maximum value of 1 (see Figure <ref>), yielding a total of 8 × 79 = 632 input values.
    The network's output is an 88-dimensional vector; each component corresponds to a specific piano key, and represents the presence or absence of a note onset at the 5 frame in the reading window.
    The output layer uses the sigmoid activation function, yielding values in the interval (0, 1).
    We use two hidden layers of 512 neurons each with the softsign activation function <cit.>.
    
    

 §.§ Data Generation

    
    
    Our data generation method is based upon the linearity of the CQT transform.
    By superimposing spectrograms of individual notes at various locations and intensities, we can model many characteristics of real piano music.
    
    For each training example, we generate an 8 × 79 magnitude spectrogram, representing the reading window, and its associated label.
    Each spectrogram is generated using a spectral basis: a collection of 88 spectrograms, one for each piano note, each with a labeled onset time.
    
    For each of the MIDI instruments listed in Appendix <ref>, we generate a spectral basis for each of the four MIDI velocities 30, 60, 90, and 120.
    Spectrograms are generated from audio recordings of each note approximately 3.5 seconds in length.[For SoundFont instruments, we use 0.5 seconds of silence followed by 3 seconds of sustained sound, leading to a 151-frame spectrogram. Spectrograms for SFZ instruments are 172 frames long due to a quirk in rendering, but this does not significantly affect our data generation.]
	This yields a total of 7 × 4 = 28 sets of spectral bases.
	Since loudly and softly played piano notes differ fundamentally in their harmonic composition, generating a new spectral basis for various MIDI velocities allows us to model the nonlinearities in note partials and decay associated with each dynamic level.
    
    Note that although the CQT transform is linear, its magnitude is not.[However, the magnitude of the transform is approximately linear, justifying NMF-based techniques <cit.>. Despite this, we find that many characteristics of real data, such as nonlinear note partial interference, are more effectively modeled by keeping complex phase information.]
    For this reason, we opt to store the complex-valued spectral basis and perform all calculations in the complex domain.
    
    

 §.§ Algorithm Description

    At the start of the generation of each training example, one of the spectral bases is randomly chosen.
    
    We label the 5 frame in the reading window as Frame 0.
    Several chord onset times are randomly picked between Frame -130 and Frame 10; the number of onset times is chosen from a Poisson distribution with λ = 6.
    
    For each chord, we randomly pick several notes uniformly from A_0 to C_8 without replacement.
    The number of notes in each chord is chosen out of a geometric distribution with p = 0.4.
    The spectrogram for the chord is modeled as a superposition of the spectrograms for each note.
    
    To improve robustness, we scale the basis spectrogram for each note by a factor randomly chosen between 0.1 and 1, and randomly shift the overall complex phase.
    The spectrogram of each note is also temporally shifted by a number of frames drawn from a discrete Gaussian distribution with σ = 0.5.
    
    To model note offsets, the spectrogram for a randomly chosen 5% of notes begins to decay exponentially by a factor of e per frame, starting on a frame randomly chosen from the reading window.
    
    Finally, the spectrogram for each chord is scaled by a factor randomly chosen between 0.1 and 1 and placed at its corresponding chord onset frame; the overall spectrogram is formed by superimposing all the chord spectrograms.
    
    We take the magnitude of the overall spectrogram[If no value within the reading window has a magnitude exceeding 10^-3, we discard the data and start over to avoid “silent” frames.] and add white noise of magnitude 0.003.
    We then use the reading window from Frame -4 to Frame 3, normalized to a maximum value of 1, as our final generated data sample.[In practice, of course, we perform only the calculations necessary to compute the values within the reading window.]
    
    The labels are 88-dimensional vectors whose components are ternary logical values.
    We assign a value of true () if a note onset occurs at exactly the 5 frame of the reading window, a value of unknown () if an onset occurs at frames 3, 4, 6, or 7 of the reading window, and a value of () otherwise.
    In practice, we encode the values , , and  as 1, 1/2, and 0, respectively.
    
    

§ TRAINING

    

 §.§ Setup

    Training data was generated using the 24 spectral bases associated with the , , , , , and  MIDI instruments.
    The network was trained for 1.5 million iterations using the ADAM optimizer <cit.> with an initial learning rate of 10^-3.
    We use a mini-batch size of 32 and an L^2 regularization parameter of 5 × 10^-10.
    
    We use a modified version of the cross-entropy loss function, where  labels do not contribute to the cost regardless of the network's output.
    This is to avoid penalizing ambiguous cases where a note onset occurs, but is not perfectly centered in the reading frame — it is difficult to define a precise temporal threshold between the presence and absence of an onset.
    
    Detailed information about the system configuration used for training is available in Appendix <ref>.
    
    

 §.§ Validation

    Three validation datasets were used, each of which contained 32768 samples:
    
    	
  * Data generated using the 4 spectral bases associated with the  MIDI instrument.
        
  * Synthesized piano recordings from the MAPS database.
        
  * Real piano recordings from the MAPS database.
    
    
    For MAPS validation data, we precomputed the spectrogram for each piece and normalized each to a maximum value of 1.
    We randomly selected 1000 recordings for validation.
    Each sample was generated by selecting a reading window randomly from the validation spectrograms.
    Reading windows whose magnitudes did not exceed 10^-3 were discarded; otherwise, they were normalized to a maximum of 1 and used as validation samples.
    Corresponding ternary labels were constructed using the MAPS onset annotations in the same manner as for synthesized data.
    Network performance on these validation datasets, as well as on a similarly generated dataset of 32768 training samples, is discussed in Appendix <ref>.
    
    

§ EVALUATION

    Our network was evaluated on the 28910 MAPS recordings not used for validation.
    For each spectrogram, we iterated over all positions of the reading window; windows were filtered and normalized as for the validation datasets, then fed through the network, generating a raw onset piano-roll representation.
    
    The onset piano-roll was thresholded at 0.8 to obtain a binary activation piano-roll.
    For each of the 88 notes, every run of consecutive activations was considered to be a single note onset event; the frame at which each event occurred was taken to be the average of all frames involved in a run.
    We convert the frame of each event into an onset time and evaluate our network predictions using the  tool <cit.>.
    
    Onsets are considered to be correct if their onset time is within 50ms of a ground truth onset at the same pitch; each label can only be matched with one network prediction.
    Unmatched labels are counted as false negatives, and unmatched predictions are counted as false positives.
    Standard note-based precision (), recall (), accuracy (), and  () metrics are computed as described in <cit.>.

	

§ RESULTS

    
    
    
    
    Our results, as well as those of other machine learning systems, are shown in Table <ref>.[Poliner used a detection tolerance of 100ms rather than 50ms.]
    Note that all other approaches listed use the MAPS dataset as both training and testing data, while we did not train on the MAPS dataset.
    In addition, our feed-forward neural network is much simpler than the models proposed by other authors.
    
    Nevertheless, we manage to exceed the , precision, and accuracy of the convolutional neural network used in <cit.> when evaluated on the MAPS dataset.
    Although <cit.> also gives results for a network trained on synthesized pianos and tested on real recordings, their results under this configuration are significantly worse: training and testing on different datasets, they achieve an onset  of only 54.89.
    
    Since we do not train on MAPS data, our results demonstrate our system's ability to transcribe recordings under novel conditions.
    Our approach is also context-independent, allowing it to be immediately applied to any recording without prior adjustment or fine-tuning.
    
    Detailed results for the overall MAPS dataset, as well as for each individual component, are listed in Table <ref>.
    Note that our network achieves the best  on the MUS component, which consists of real musical compositions.
    A sample comparison between our network's predictions and the MAPS ground truth labels is shown in Figure <ref>.
    
    While our network performs significantly worse on the real Disklavier samples, these recordings can have annotation discrepancies of up to 100ms <cit.>.
    In addition, we found several issues with the MAPS Disklavier dataset, including omitted notes and recordings consisting entirely of percussive keybed and pedal noises.
    However, even taking these discrepancies into account, we do expect our network to perform worse on real piano recordings; software samples are unable to capture the piano's full range of acoustic subtleties.

    

§ CONCLUSION

    We have presented a method of generating spectral data which accurately models many aspects of piano music, avoiding the need for an annotated musical database.
    We trained a simple machine learning model on our procedurally generated data and achieve good transcription results under novel recording conditions, highlighting our network's context-independence and excellent generalization qualities.
    Our approach outperforms that of <cit.>, even though they train and test on samples drawn from the same database and propose a much more complex network architecture.
    
    

 §.§ Future Work

    Several improvements to our data generation method could be implemented:
    
    	
  * We could quite easily model the full range of pianos more accurately by adding more MIDI instruments or increasing the number of MIDI velocities sampled.
    	
  * Note offsets are modeled in a rather crude fashion by applying exponential decay to the note spectrogram; a sample-based approach could more accurately model various articulations.
    	
  * No distinction is made between notes played with and without sustain pedal — lifting the dampers alters note timbre slightly and could be modeled with additional samples.
    	
  * Rather than model chord onset times and notes randomly, a musical model could be introduced to take into account the temporal patterns and harmonic progressions present in Western music.
    
    
    In addition, our method could easily be extended to generate training data for recursive, convolutional, or LSTM networks.
    Considering that we were able to train a very simple network network using our proposed method and achieve good transcription performance, using a more complex machine learning model would likely yield better results.
    
    The use of generated training data as input to a neural network brings to mind the principle behind generative adversarial networks, which have not yet been applied to piano onset transcription.
    It may be possible to train a neural network to generate data which closely resemble those of real pianos, effectively creating an infinite stream of high-quality spectral data indistinguishable from those of an annotated database.

    

§ MIDI INSTRUMENTS

    
    
    The SoundFont/SFZ instruments used are listed in Table <ref>; all of them are available online, free of charge.
    Audio samples for spectral basis generation were synthesized by FluidSynth for SoundFont 2 instruments and by Plogue sforzando for SFZ instruments.
    Both software synthesizers are available free of charge.
    
    
    

§ TRAINING AND VALIDATION PERFORMANCE

    
    
    
    
    
    For evaluation on training and validation datasets, the network's output is interpreted as a ternary logical value; a value greater than 0.8 is reported as a note onset (), a value less than 0.2 is reported as a lack thereof (), and a value of  is reported otherwise.
    We evaluate the network's precision (), recall (), accuracy (), and  (), defined using the ternary confusion matrix (Figure <ref>) as follows:
    
    
    = HTP + STP/HTP + STP + HFP
       = HTP/HTP + SFN + HFN
       = HTP + STP/HTP + STP + HFP + SFN + HFN
       = 2 ××/ +


	Our network's performance on training and validation data can be found in Table <ref>.
    Since these metrics are computed in a non-standard manner, they cannot be directly compared with those of other approaches.
    However, we observe that the  for our generated validation data and the synthesized MAPS validation data are nearly identical, highlighting our network's generalization abilities.
    
    

§ TRAINING SETUP

    
    Our network was trained on the CPU of a Lenovo  laptop.
    The system has a 1.40GHz Intel Celeron 2957U CPU and 4.00GB of DDR3 RAM.
    We use TensorFlow for Python 3.5.3 on Windows 10.
    Training took a total of 44 hours.

    