Why are the nodes (or neurons) in neural networks depicted as circles?
I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.