A CNN encoder block mainly consists of a stack of identical causal or dilated convolutional layers. Each convolution has stride 1 and prior to processing prepends k -1 padding tokens to inputs, where k is a kernel size, so that outputs of convolutions have dimensions equal to the inputs'. This way we save positional information (and therefore, pooling is also redundant). Kernels can be interpreted as fragment detectors. The maximum path length is O(τ /k) or O(log k τ ), depending on the type of convolutions. The computational complexity of convolutions is O(kτ d 2 ), as opposed to O(τ d 2 ) of RNNs, but the sequential operations can be parallelized. Optionally, for deeper models, we employ residual or highway connection between the outputs of an embedding block and the last convolutional layer.