Why don't O3 less than O1 in gpu memory?
i'm training EfficientDet-D7(head_only=True) in 2080TI * 1.
And i'm using NVIDIA/APEX:amp.
When i use opt_level=O1, although the memory is definitely reduced compared to when apex is not used.
But, when I use opt_level=O2orO3, more memory is consumed.
I am experimenting with the same 2080 Ti, each with a separate GPU by creating two containers with the same docker image. The learning code also copied the code using O1 as it is and changed it to O3, and the args required for learning are all the same. (The batch size and d7 are also the same.)
Why happen this... TT
Additionally, Can you recommend the book about this?(ex. deep learning Memory, gpu ... etc)
Thanks!

You're optimizing for speed. Some speed optimizations will reduce memory usage. Other will increase them.

An example of when speed optimization reduces memory usage is when unnecessary variables and function calls are removed.

An example of the opposite is loop unrolling.


There's no reason to expect optimization to either reduce or increase memory usage. That's not the goal when optimizing for speed. Any increase or decrease is just a byproduct.
If you really want to find out why it happens in your particular case, you can study the documentation for your compiler and inspect the assembly code.
