Neural Network Simulator with OpenMP
It is generally advisable to parallelize loops at the absolute outermost level possible. Creating new OS threads, dividing up the relevant loop, and allocating data in the private address space for each thread all take up time, as does synchronization once the parallel task is over. Often, this overhead can outweigh any benefit that parallelism might have conferred in the first place. Whenever you use OpenMP to parallelize a task or loop, it has to be large enough to amortize the cost of spawning threads.

I would hoist as many parallel directives to the outermost loop as possible, and avoid having nested parallel directives. You may have to switch from using omp parallel for to just using omp parallel at the outermost level, then having some logic to explicitly decide which part of the data each thread works on depending on its thread number.

You'll have to be the judge of whether this is actually the problem or not; I only did a shallow read of your code, and the work that gets parallelized in the inner loops might actually be large enough that this is not the consideration at all.

Finally, there are alternatives to OpenMP for shared-memory parallel programming, e.g. Intel's Threading Building Blocks. OpenMP is pretty crude in the grand scheme of things, so you might find task-based parallelism easier or more fun to work with. You get to pretend like you have as many parallel threads of execution as you want, and the library manages farming out the tasks you define to the existing OS threads.