Studying the speech-generation model and have question about the confusing nature of model input and outputs
There are two &quot;inputs&quot; into Wavenet:

the previously generated samples of the waveform, which are usually encoded into multiple channels, like into 256 channels using 8-bit mu-law encoding
local conditioning, which can be things like linguistic features such as phoneme classes (used in the original wavenet paper), or frequencies like mel spectrogram values (used in the Tacotron 2 paper)

The local conditioning signal usually has a much lower resolution than the waveform itself. For example, the Tacotron 2 paper mentioned using &quot;50 ms frame size, 12.5 ms frame hop&quot; to derive its mel spectogram values. At 24 kHz, each waveform sample has a duration of just 0.0416 ms. So in order to using the spectogram information to condition the waveform generation, it would have to be upsampled to spread out along the time dimension. (If you were locally conditioning using letters, you might turn &quot;dog&quot; into &quot;dddddoooooooooggg&quot; in order to use the letter &quot;d&quot; to generate 5 samples in the output waveform, etc.).


What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1 floating point value for raw audio?


In the implementation you linked to, it looks like variables that start with &quot;c&quot; refer to &quot;conditioning&quot; signals. So &quot;hparams.cin_channels&quot; indicates how many channels the input conditioning signal has. If that signal is mel spectogram with 80 channels, then it would be set to 80.
It looks like the &quot;inference/input_convolution&quot; layer is processing the actual waveform, not the spectogram conditioning signal.


Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?


I suspect (though I'm not too familiar with tensorflow conventions) that the second dimension is time, which is why you're seeing the model upsample the conditioning signal in that direction. Those values may be required to match the resolution of the waveform being generated.


Why is the input-channels in residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs?


The &quot;c&quot; in &quot;cin_conv&quot; appears to mean that it is for the conditioning signal, which is why is has a different dimension than the waveform's embedding size.
I hope this helps. You could also try opening up an issue on that github repo to try getting help directly from the author.