Is GNU Octave a perfect place to code neural networks
No.

The key to implementing neural networks easily and fast is having automatic differentiation. This is the key feature that allows programs using Python libraries like Pytorch or Tensorflow (and Keras), to implement only the forward pass of a neural network and have the backward pass computed automatically.

Octave does not have automatic differentiation. While Matlab's deep learning toolbox seems to include automatic differentiation, I think it is not included with Octave. And even if it were, the deep learning community is mainly focused on the Python ecosystem, and the rest of the environments like R or Matlab are paid almost no attention in comparison.