Neural network without matrices
Matrix multiplication is just a simplified notation for a particular set of addition and multiplication operations. You can absolutely represent a neural network without invoking matrix notation, it'll just be really tedious (and it will run slower).

Start with a single perceptron and build your way up from there.