Best practice to test (unit and integration) a neural network implementation
Make your code testable. You should be able to write plenty of tests for individual components without having to mock up an entire training pipeline for each one. A "property-based testing" library like Hypothesis can be helpful for this.
Train a standard model on a standard dataset and make sure the accuracy numbers are similar.
See here for an in-depth discussion, with more and better advice than I've been able to provide.