Confusion about Keras' skipgram and sampling table utilities I'm fairly new to ML, so as a learning exercise to get familiar with Keras I'm trying to learn some word2vec style embeddings from a dataset. I'm confused about the behavior of the skipgram utility, in particular the sampling table argument that's supposed to be populated from make_sampling_table. I understand the idea behind it, but I'm seeing some strange outputs.

As a toy example, let's pretend that we have the following parameters:

vocab_size = 3
sequence = [0,1,0,1,1,0]


So a small vocabulary with 3 words in it, only 2 of which show up in the sequence. Generating the skipgrams &amp; labels works fine without a sampling table at all:

pairs, labels = skipgrams(sequence, vocab_size)
print("pairs: " + str(pairs) + " labels: " + str(labels))

OUTPUT: pairs: [[1, 1], [1, 2], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 1]] labels: [0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1]


But when I try to pass in a sampling table so as to not oversample the common words, I get no outputs at all:

pairs, labels = skipgrams(sequence, vocab_size, sampling_table=make_sampling_table(vocab_size))
print("pairs: " + str(pairs) + " labels: " + str(labels))

OUTPUT: pairs: [] labels: []


My question is: what's going on here? Why does the act of passing a sampling table into the skipgrams method make it produce no output?

Thanks for the help!
After looking through the code for the skipgram method, I figured it out: it has to do with the sampling_factor parameter. It defaults to 1e-5, which means that for such a small sampling factor it's almost never going to actually pick a sample unless your dataset &amp; vocabulary size is big enough to actually get some samples. If you have a small vocabulary (like in the toy example), you should set that to something much higher: when I set it to 1.0 (i.e. take a sample as often as possible) I got lots of pairs, even for the toy example.