How much neural network theory required to design one?
Here is a good way to get going:


C++ or c#
spend a week learning vectors and matrices.
Be able to code a "class vector" and "class matrix"
Get intuition on how matrices represent weights between 2 layers
Learn the chain rule and the "tree diagram" when computing composite (nested) functions
Spend a week learning backprop. Be able to compute derivatives for all layers, using pen an paper a toy example with 4 layers, 3 neurons each.
be able to make code that reads a .txt file character by character. 
Learn about one-hot, to describe your character to the network (it only understands vectors)  - or see how your byte-problem might be used instead of characters.
Learn about hard-max softmax and their derivatives,  tanh and sigmoid
code fwd prop
code bkprop
test if it can predict the next char
debug
debug
DEBUG
Gradient checking