!pip install "thinc>=8.0.0a0"

"""
Explanation: Intro to Thinc's Model class, model definition and methods
Thinc follows a functional-programming approach to model definition. Its approach is especially effective for complicated network architectures, and use cases where different data types need to be passed through the network to reach specific subcomponents. This notebook shows how to compose Thinc models and how to use the Model class and its methods.
End of explanation
"""


import numpy
from thinc.api import Linear, zero_init

n_in = numpy.zeros((128, 16), dtype="f")
n_out = numpy.zeros((128, 10), dtype="f")

model = Linear(nI=n_in.shape[1], nO=n_out.shape[1], init_W=zero_init)
nI = model.get_dim("nI")
nO = model.get_dim("nO")
print(f"Initialized model with input dimension nI={nI} and output dimension nO={nO}.")

"""
Explanation: Thinc provides a variety of layers, functions that create Model instances. Thinc tries to avoid inheritance, preferring function composition. The Linear function gives you a model that computes Y = X @ W.T + b (the function is defined in thinc.layers.linear.forward).
End of explanation
"""


model = Linear(init_W=zero_init)
print(f"Initialized model with no input/ouput dimensions.")

X = numpy.zeros((128, 16), dtype="f")
Y = numpy.zeros((128, 10), dtype="f")
model.initialize(X=X, Y=Y)
nI = model.get_dim("nI")
nO = model.get_dim("nO")
print(f"Initialized model with input dimension nI={nI} and output dimension nO={nO}.")

"""
Explanation: Models support dimension inference from data. You can defer some or all of the dimensions.
End of explanation
"""


from thinc.api import chain, glorot_uniform_init

n_hidden = 128
X = numpy.zeros((128, 16), dtype="f")
Y = numpy.zeros((128, 10), dtype="f")

model = chain(Linear(n_hidden, init_W=glorot_uniform_init), Linear(init_W=zero_init),)
model.initialize(X=X, Y=Y)
nI = model.get_dim("nI")
nO = model.get_dim("nO")
nO_hidden = model.layers[0].get_dim("nO")
print(f"Initialized model with input dimension nI={nI} and output dimension nO={nO}.")
print(f"The size of the hidden layer is {nO_hidden}.")

"""
Explanation: The chain function wires two model instances together, with a feed-forward relationship. Dimension inference is especially helpful here.
End of explanation
"""


from thinc.api import concatenate

model = concatenate(Linear(n_hidden), Linear(n_hidden))
model.initialize(X=X)
nO = model.get_dim("nO")
print(f"Initialized model with output dimension nO={nO}.")

"""
Explanation: We call functions like chain combinators. Combinators take one or more models as arguments, and return another model instance, without introducing any new weight parameters. Another useful combinator is concatenate:
End of explanation
"""


from thinc.api import clone

model = clone(Linear(), 5)
model.layers[0].set_dim("nO", n_hidden)
model.initialize(X=X, Y=Y)
nI = model.get_dim("nI")
nO = model.get_dim("nO")
print(f"Initialized model with input dimension nI={nI} and output dimension nO={nO}.")

"""
Explanation: The concatenate function produces a layer that runs the child layers separately, and then concatenates their outputs together. This is often useful for combining features from different sources. For instance, we use this all the time to build spaCy's embedding layers.
Some combinators work on a layer and a numeric argument. For instance, the clone combinator creates a number of copies of a layer, and chains them together into a deep feed-forward network. The shape inference is especially handy here: we want the first and last layers to have different shapes, so we can avoid providing any dimensions into the layer we clone. We then just have to specify the first layer's output size, and we can let the rest of the dimensions be inferred from the data.
End of explanation
"""


from thinc.api import Relu, Dropout

def Hidden(dropout=0.2):
    return chain(Linear(), Relu(), Dropout(dropout))

model = clone(Hidden(0.2), 5)

"""
Explanation: We can apply clone to model instances that have child layers, making it easy to define more complex architectures. For instance, we often want to attach an activation function and dropout to a linear layer, and then repeat that substructure a number of times. Of course, you can make whatever intermediate functions you find helpful.
End of explanation
"""


from thinc.api import with_array

model = with_array(Linear(4, 2))
Xs = [model.ops.alloc2f(10, 2, dtype="f")]
model.initialize(X=Xs)
Ys = model.predict(Xs)
print(f"Prediction shape: {Ys[0].shape}.")

"""
Explanation: Some combinators are unary functions: they take only one model. These are usually input and output transformations. For instance, the with_array combinator produces a model that flattens lists of arrays into a single array, and then calls the child layer to get the flattened output. It then reverses the transformation on the output.
End of explanation
"""


from thinc.api import add, chain, concatenate, clone
from thinc.api import with_array, reduce_max, reduce_mean, residual
from thinc.api import Model, Embed, Maxout, Softmax

nH = 5

with Model.define_operators({">>": chain, "|": concatenate, "+": add, "**": clone}):
    model = (
        with_array(
            (Embed(128, column=0) + Embed(64, column=1))
            >> Maxout(nH, normalize=True, dropout=0.2)
        )
        >> (reduce_max() | reduce_mean())
        >> residual(Relu() >> Dropout(0.2)) ** 2
        >> Softmax()
    )

"""
Explanation: The combinator system makes it easy to wire together complex models very concisely. A concise notation is a huge advantage, because it lets you read and review your model with less clutter – making it easy to spot mistakes, and easy to make changes. For the ultimate in concise notation, you can also take advantage of Thinc's operator overloading, which lets you use an infix notation. Operator overloading can lead to unexpected results, so you have to enable the overloading explicitly in a contextmanager. This also lets you control how the operators are bound, making it easy to use the feature with your own combinators. For instance, here is a definition for a text classification network:
End of explanation
"""


from thinc.api import Linear, Adam
import numpy

X = numpy.zeros((128, 10), dtype="f")
dY = numpy.zeros((128, 10), dtype="f")

model = Linear(10, 10)

"""
Explanation: The network above will expect a list of arrays as input, where each array should have two columns with different numeric identifier features. The two features will be embedded using separate embedding tables, and the two vectors added and passed through a Maxout layer with layer normalization and dropout. The sequences then pass through two pooling functions, and the concatenated results are passed through 2 Relu layers with dropout and residual connections. Finally, the sequence vectors are passed through an output layer, which has a Softmax activation.

Using a model
Define the model:
End of explanation
"""


model.initialize(X=X, Y=dY)

"""
Explanation: Initialize the model with a sample of the data:
End of explanation
"""


Y = model.predict(X)
Y

"""
Explanation: Run the model over some data
End of explanation
"""


Y, backprop = model.begin_update(X)
Y, backprop

"""
Explanation: Get a callback to backpropagate:
End of explanation
"""


dX = backprop(dY)
dX

"""
Explanation: Run the callback to calculate the gradient with respect to the inputs. If the model has trainable parameters, gradients for the parameters are accumulated internally, as a side-effect.
End of explanation
"""


optimizer = Adam()
model.finish_update(optimizer)

"""
Explanation: The backprop() callback only increments the parameter gradients, it doesn't actually change the weights. To increment the weights, call model.finish_update(), passing it an optimizer:
End of explanation
"""


dim = model.get_dim("nO")
W = model.get_param("W")
model.attrs["hello"] = "world"
model.attrs.get("foo", "bar")

"""
Explanation: You can get and set dimensions, parameters and attributes by name:
End of explanation
"""


dW = model.get_grad("W")
model.inc_grad("W", dW * 0.1)

"""
Explanation: You can also retrieve parameter gradients, and increment them explicitly:
End of explanation
"""


model_bytes = model.to_bytes()

"""
Explanation: Finally, you can serialize models using the model.to_bytes and model.to_disk methods, and load them back with from_bytes and from_disk.
End of explanation
"""


