Instruction: clang: error: unsupported option '-fopenmp' (SparseConvNe build error)

Input:
Thank you first for your help and time. I am trying to run and build SparseConvNet (https://github.com/facebookresearch/SparseConvNet)on my Mac, however, I get the following error after running bash develop.sh on my terminal:
running develop
running egg_info
creating sparseconvnet.egg-info
writing sparseconvnet.egg-info/PKG-INFO
writing dependency_links to sparseconvnet.egg-info/dependency_links.txt
writing top-level names to sparseconvnet.egg-info/top_level.txt
writing manifest file 'sparseconvnet.egg-info/SOURCES.txt'
package init file 'sparseconvnet/SCN/__init__.py' not found (or not a regular file)
reading manifest file 'sparseconvnet.egg-info/SOURCES.txt'
writing manifest file 'sparseconvnet.egg-info/SOURCES.txt'
running build_ext
building 'sparseconvnet.SCN' extension
creating build
creating build/temp.macosx-10.6-x86_64-3.5
creating build/temp.macosx-10.6-x86_64-3.5/sparseconvnet
creating build/temp.macosx-10.6-x86_64-3.5/sparseconvnet/SCN
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/sahar/opt/anaconda3/envs/mypython3/include -arch x86_64 -I/Users/sahar/opt/anaconda3/envs/mypython3/include -arch x86_64 -I/Users/sahar/Documents/test_sparse/SparseConvNet/sparseconvnet/SCN/ -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/TH -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/THC -I/Users/sahar/opt/anaconda3/envs/mypython3/include/python3.5m -c sparseconvnet/SCN/pybind.cpp -o build/temp.macosx-10.6-x86_64-3.5/sparseconvnet/SCN/pybind.o -std=c++14 -fopenmp /usr/local/bin/g++-11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=SCN -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: /usr/local/bin/g++-11: 'linker' input unused [-Wunused-command-line-argument]
clang: error: unsupported option '-fopenmp'
error: command '/usr/bin/clang' failed with exit status 1

after some search I figured out that the problem may be with the clang so I installed gcc as follow:
brew install gcc
and I added the path in my bash_profile as follow:
export CC=/usr/local/bin/gcc-11
export CXX=/usr/local/bin/g++-11
I also modified the following line in setub.py
from
extra = {'cxx': ['-std=c++14', '-fopenmp','-O3'], 'nvcc': ['-std=c++14', '-Xcompiler', '-fopenmp', '-O3']}
to
extra = {'cxx': ['-std=c++14', '-fopenmp', "/usr/local/bin/g++-11"], 'nvcc': ['-std=c++14', '-Xcompiler', '-fopenmp', '-O3']}
After doing the aforementioned steps I get the following error
/usr/local/bin/gcc-11 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/sahar/opt/anaconda3/envs/mypython3/include -arch x86_64 -I/Users/sahar/opt/anaconda3/envs/mypython3/include -arch x86_64 -I/Users/sahar/Documents/test_sparse/SparseConvNet/sparseconvnet/SCN/ -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/TH -I/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/THC -I/Users/sahar/opt/anaconda3/envs/mypython3/include/python3.5m -c sparseconvnet/SCN/sparseconvnet_cpu.cpp -o build/temp.macosx-10.6-x86_64-3.5/sparseconvnet/SCN/sparseconvnet_cpu.o -std=c++14 -fopenmp /usr/local/bin/g++-11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=SCN -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command-line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/core/Device.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/core/Allocator.h:6,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h: In function 'const char* torch::jit::toString(torch::jit::AttributeKind)':
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h:21:42: warning: expression does not compute the number of elements in this array; element type is 'const char*', not 'torch::jit::AttributeKind' -Wsizeof-array-div]
   21 |   AT_ASSERT(size_t(kind) < sizeof(names) / sizeof(AttributeKind));
      |                            ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:148:39: note: in definition of macro 'C10_EXPAND_MSVC_WORKAROUND'
  148 | #define C10_EXPAND_MSVC_WORKAROUND(x) x
      |                                       ^
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:167:34: note: in expansion of macro 'C10_UNLIKELY'
  167 | #define C10_UNLIKELY_OR_CONST(e) C10_UNLIKELY(e)
      |                                  ^~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:204:7: note: in expansion of macro 'C10_UNLIKELY_OR_CONST'
  204 |   if (C10_UNLIKELY_OR_CONST(!(cond))) {       \
      |       ^~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:360:32: note: in expansion of macro 'TORCH_INTERNAL_ASSERT'
  360 |     C10_EXPAND_MSVC_WORKAROUND(TORCH_INTERNAL_ASSERT(__VA_ARGS__)); \
      |                                ^~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h:21:3: note: in expansion of macro 'AT_ASSERT'
   21 |   AT_ASSERT(size_t(kind) < sizeof(names) / sizeof(AttributeKind));
      |   ^~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h:21:44: note: add parentheses around 'sizeof (torch::jit::AttributeKind)' to silence this warning
   21 |   AT_ASSERT(size_t(kind) < sizeof(names) / sizeof(AttributeKind));
      |                                            ^~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:148:39: note: in definition of macro 'C10_EXPAND_MSVC_WORKAROUND'
  148 | #define C10_EXPAND_MSVC_WORKAROUND(x) x
      |                                       ^
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:167:34: note: in expansion of macro 'C10_UNLIKELY'
  167 | #define C10_UNLIKELY_OR_CONST(e) C10_UNLIKELY(e)
      |                                  ^~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:204:7: note: in expansion of macro 'C10_UNLIKELY_OR_CONST'
  204 |   if (C10_UNLIKELY_OR_CONST(!(cond))) {       \
      |       ^~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/c10/util/Exception.h:360:32: note: in expansion of macro 'TORCH_INTERNAL_ASSERT'
  360 |     C10_EXPAND_MSVC_WORKAROUND(TORCH_INTERNAL_ASSERT(__VA_ARGS__)); \
      |                                ^~~~~~~~~~~~~~~~~~~~~
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h:21:3: note: in expansion of macro 'AT_ASSERT'
   21 |   AT_ASSERT(size_t(kind) < sizeof(names) / sizeof(AttributeKind));
      |   ^~~~~~~~~
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/ir.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/autograd/generated/variable_factories.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:7,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/jit/attributes.h:19:22: note: array 'names' declared here
   19 |   static const char* names[] = {
      |                      ^~~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 1]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 1]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:566:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 3>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 3>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 2]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 2]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:569:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 4>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 4>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 3]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 3]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:572:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 5>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 5>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 4]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 4]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:575:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 6>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 6>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 5]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 5]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:578:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 7>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 7>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
In file included from sparseconvnet/SCN/sparseconvnet_cpu.cpp:33:
sparseconvnet/SCN/CPU/SparseToDense.cpp: In instantiation of 'void cpu_SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with T = float; int Dimension = 6]':
sparseconvnet/SCN/sparseconvnet_cpu.cpp:387:51:   required from 'void SparseToDense_updateOutput(at::Tensor&, Metadata<Dimension>&, at::Tensor&, at::Tensor&, long int) [with int Dimension = 6]'
sparseconvnet/SCN/sparseconvnet_cpu.cpp:581:1:   required from here
sparseconvnet/SCN/CPU/SparseToDense.cpp:48:29: error: cannot convert 'std::array<long int, 8>' to 'c10::IntArrayRef' {aka 'c10::ArrayRef<long long int>'}
   48 |     output_features.resize_(sz);
      |                             ^~
      |                             |
      |                             std::array<long int, 8>
In file included from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Tensor.h:12,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/Context.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/ATen.h:5,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/torch/extension.h:4,
                 from sparseconvnet/SCN/sparseconvnet_cpu.cpp:12:
/Users/sahar/opt/anaconda3/envs/mypython3/lib/python3.5/site-packages/torch/include/ATen/core/TensorMethods.h:961:45: note:   initializing argument 1 of 'at::Tensor& at::Tensor::resize_(c10::IntArrayRef) const'
  961 | inline Tensor & Tensor::resize_(IntArrayRef size) const {
      |                                 ~~~~~~~~~~~~^~~~
error: command '/usr/local/bin/gcc-11' failed with exit status 1

I tested my code both by poetry and anaconda and in both, I get the same error.

Output:
I'm unsure about the error you received after adjusting the setub.py file (may be due to changing the compiler version being called), but your original error seems to relate to the MacOS version of clang not natively supporting fopenmp. A solution for this was posted here:
Enable OpenMP support in clang in Mac OS X (sierra & Mojave)


Instruction: Input to nn.Linear(in_features=16*4*4, out_features=100)

Input:
I am performing a CNN on the MNIST dataset with the following model:
class ConvNet(nn.Module):
  def __init__(self, mode):
    super(ConvNet, self).__init__()

    # Define various layers here, such as in the tutorial example
    # self.conv1 = nn.Conv2D(...)
    #First Convolution Kayer
    #input size (28,28), output size = (24,24)
    self.conv1 = nn.Conv2d(1,6,5)
    self.reLU1 = nn.ReLU(inplace=True)
    self.MaxPool1 = nn.MaxPool2d(kernel_size=2)

    #Second Convolution Layer
    #input size (12,12), output_size = (8,8)
    self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5)
    self.reLU2 = nn.ReLU(inplace=True)
    self.MaxPool2 = nn.MaxPool2d(kernel_size=2)

    #Affine operations
    self.fc1 = nn.Linear(in_features = 16*4*4, out_features = 100)
    self.sig = torch.nn.Sigmoid()
    self.fc2 = nn.Linear(in_features=100, out_features=10)

My forward pass is defined below.
def forward_pass(self, X):
    #Conv Layer #1
    X = self.conv1(X)
    X = self.reLU1(X)
    X = self.MaxPool1(X)
    #Conv Layer #2
    X = self.conv2(X)
    X = self.reLU2(X)
    X = self.MaxPool2(X)

    print(Tensor.size(X))
    #X = X.view()
    X = self.fc1(X)
    X = self.sig(X)
    X = self.fc2(X)

    return X

I get an error when trying to pass the Tensor into the fully connected layer #1 (fc1). This is due to a mismatch in the in_features from my last Convolutional Layers.
When I print out the size of the tensor X before my fully connected layer I get tensor.Size([10,16,4,4]). Can anyone explain to me what would be the proper way to calculate the input to the first fully connected layer?

Output:
Your classifier's input is shaped (10, 16, 4, 4), discarding the first dimension which corresponds to the batch size, you end up with 16*4*4 elements. So this is correct, but the shape isn't: you need to flatten the spatial dimension before feeding the tensor to fc1. You can do using nn.Flatten:
class ConvNet(nn.Module):
    def __init__(self, mode):
        super(ConvNet, self).__init__()
        ## layer definitions
        self.flatten = nn.Flatten()

    def forward(self, X):
        ## inference on CNN
        X = self.flatten(X)
        ## inference on fully-connected layers

Here is a inference example:
>>> model = ConvNet(mode=None)
>>> model(torch.rand(10, 1, 24, 24))
torch.Size([10, 10])

Side note please name your function forward instead of forward_pass, it is standard practice.


Instruction: Error: Some NCCL operations have failed or timed out

Input:
While running a distributed training on 4 A6000 GPUs, I get the following error:
[E ProcessGroupNCCL.cpp:630] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1803710 milliseconds before timing out.
                                                                                [E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.

terminate called after throwing an instance of 'std::runtime_error'
what():  [Rank 2] Watchdog caught collective operation timeout:
WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1804406 milliseconds before timing out.

[E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.

I use standard NVidia PyTorch docker. Interesting thing is that training works fine for small datasets but for the bigger datasets, I get this error. So I can confirm that the training code is correct and does work.
There is no actual runtime error or any other information to get an actual error messages anywhere.

Output:
Following two have solved the issue:

Increase default SHM (shared memory) for CUDA to 10g (I think 1g would have worked as well). You can do this in docker run command by passing --shm-size=10g. I also pass --ulimit memlock=-1.
export NCCL_P2P_LEVEL=NVL.

Debugging Tips
To check current SHM,
df -h
# see the row for shm

To see NCCL debug messages:
export NCCL_DEBUG=INFO

Run p2p bandwidth test for GPU to GPU communication link:
cd /usr/local/cuda/samples/1_Utilities/p2pBandwidthLatencyTest
sudo make
./p2pBandwidthLatencyTest

For A6000 4 GPU box this prints:

The matrix shows bandwith betweeb each pair of GPU and with P2P, it should be high.


Instruction: GPU is not available for Pytorch

Input:
I installed Anaconda, CUDA, and PyTorch today, and I can't access my GPU (RTX 2070) in torch. I followed all of installation steps and PyTorch works fine otherwise, but when I try to access the GPU either in shell or in script I get
>>> import torch
>>> torch.cuda.is_available()
False
>>> torch.cuda.device_count()
0
>>> print(torch.version.cuda)
None

Running conda list shows this as my installed package
cudatoolkit               11.3.1               h59b6b97_2

and running numba -s in the conda environment shows
__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 11030
CUDA Detect Output:
Found 1 CUDA devices
id 0    b'NVIDIA GeForce RTX 2070'                              [SUPPORTED]
                      compute capability: 7.5
                           pci device id: 0
                              pci bus id: 1
Summary:
        1/1 devices are supported

and all of the tests pass with ok. CUDA 11.3 is one of the supported compute platforms for PyTorch and by my GPU and that is the version that I installed.
I already tried reinstalling CUDA, I am on Windows 10, nvcc --version shows that CUDA is installed Build cuda_11.3.r11.3/compiler.29745058_0
Any suggestions would be helpful
Edit: I am using PyTorch 1.10 installed from the generated command on their website. Using python 3.9.7. I also installed PyTorch again in a fresh conda environment and got the same problem.

Output: If you use conda, try to update conda. It works for me to install PyTorch 1.10 with CUDA 10.2.


Instruction: How to split folder with images into train, val and test?

Input:
I am using colab and I have a folder with images. How to split them into three folders with images with random splitting? I want there to be 0.8 in train, 0.1 in val and 0.1 in test. I tried splitfolders library:
splitfolders.ratio("content/data", output="output", seed=1337, ratio=(.8, .1, .1), group_prefix=None)

but new folder didn't appear. and its not clear what would their names be. How to do that? Are there any other solutions?
Pytorch solutions are very welcomed

Output:
Try this in your computer.
I'm using this code in my project
import splitfolders

#### input dataset that want to split
input_folder = 'D:/Raw_DS'

output_folder= 'D:/Splitted_DS'

splitfolders.ratio(input_folder, output= output_folder, seed=1337, ratio = (0.8, 0.1, 0.1))


Instruction: test_train_split ValueError: Found input variables with inconsistent numbers of samples: [200000, 6]

Input:
I've looked at a couple of other posts with this issue, and I cannot figure out what I'm getting wrong here.
I have X_data, and Y_data, and they both have the shape (200000,6). Sample data output from them looks like this:
X_data:
(200000, 6)
[[ 0.00237987  0.00237987 -0.00075756 -0.00221595 -0.00368199  0.00019625]
 [ 0.00171481  0.00171481  0.00176989  0.00125255  0.00275689 -0.00111833]
 [ 0.00190234  0.00190234  0.00333571  0.00127516  0.00146631 -0.00240469]
 ...
 [ 0.00211437  0.00211437  0.00221987  0.0002214   0.00273094 -0.00114419]
 [ 0.00185682  0.00185682  0.00352099  0.00064055 -0.00051575  0.00335213]
 [ 0.00155133  0.00155133 -0.00368774 -0.00200935  0.00225988 -0.00161371]]

Y_data:
(200000, 6)
[[1.         0.14713856 0.04063819 0.03123633 0.00239176 0.01674091]
 [1.         0.35532772 0.09834969 0.19631962 0.0153588  0.10071312]
 [1.         0.17015225 0.04700213 0.04208244 0.00322773 0.02244747]
 ...
 [1.         0.14534398 0.04014234 0.03046259 0.0023313  0.01633189]
 [1.         0.18606737 0.05138638 0.0368341  0.00281708 0.01979553]
 [1.         0.31199003 0.0863072  0.14879644 0.01157114 0.07705023]]

As soon as I do test_train_split, as follows:
    ts1 = 0.2
    rs1 = 42
    X_train, X_test, Y_train, Y_test = train_test_split(X_data, Y_data[0], test_size = ts1, random_state = rs1)

My code crashes with the value error.  I have no idea where I'm going wrong.

Output:
It seems like the first column of your Y_data matrix is the label for your x data, (I'm not sure what the other 5 columns in your Y_train represent). You are currently getting the first row, which isn't correct (note the size is 6 but you would like one y-value for each x input). So the code I think you want is
X_train, X_test, Y_train, Y_test = train_test_split(X_data,\
    Y_data[:, 0], test_size = ts1, random_state = rs1)


Instruction: Torch not compiled with CUDA enabled - reinstalling pytorch is not working

Input:
My code which I'm trying to run gives me an error: AssertionError: Torch not compiled with CUDA enabled.
I was trying to search for an solution to this problem and I found a lot of solutions saying the same. Just use code:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
And then it should work, since previously pytorch was installed without CUDA enabled.
However, my code still returns me exactly same error and I don't know why. I've also tried to change cudatoolkit=10.2 to cudatoolkit=10.1 but result is identical.
For more specific info I upload some specifics obtained with nvidia-smi command:

Additionally I'm uploading list of my packages produced by conda list
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             4.5                       1_gnu
_pytorch_select           0.1                       cpu_0
absl-py                   0.15.0             pyhd8ed1ab_0    conda-forge
aiohttp                   3.7.4.post0      py39h3811e60_0    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
blas                      1.0                         mkl
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py39h3811e60_1001    conda-forge
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.17.1               h27cfd23_0
ca-certificates           2021.9.30            h06a4308_1
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
certifi                   2021.10.8        py39h06a4308_0
cffi                      1.14.6           py39h400218f_0
chardet                   4.0.0            py39hf3d152e_1    conda-forge
charset-normalizer        2.0.0              pyhd8ed1ab_0    conda-forge
click                     8.0.3            py39hf3d152e_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
cpuonly                   2.0                           0    pytorch
cryptography              35.0.0           py39hbca0aa6_0    conda-forge
cudatoolkit               10.2.89              hfd86e86_1
dataclasses               0.8                pyhc8e2a94_3    conda-forge
ffmpeg                    4.3                  hf484d3e_0    pytorch
freetype                  2.10.4               h5ab3b9f_0
fsspec                    2021.10.1          pyhd8ed1ab_0    conda-forge
future                    0.18.2           py39hf3d152e_3    conda-forge
giflib                    5.2.1                h7b6447c_0
gmp                       6.2.1                h2531618_2
gnutls                    3.6.15               he1e5248_0
google-auth               1.35.0             pyh6c4a22f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
grpcio                    1.37.1           py39hff7568b_0    conda-forge
idna                      3.1                pyhd3deb0d_0    conda-forge
importlib-metadata        4.8.1            py39hf3d152e_0    conda-forge
intel-openmp              2019.4                      243
jpeg                      9d                   h7f8727e_0
lame                      3.100                h7b6447c_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.35.1               h7274673_9
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.3.0               h5101ec6_17
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libgomp                   9.3.0               h5101ec6_17
libiconv                  1.15                 h63c8f33_5
libidn2                   2.3.2                h7f8727e_0
libmklml                  2019.0.5                      0
libpng                    1.6.37               hbc83047_0
libprotobuf               3.15.8               h780b84a_0    conda-forge
libstdcxx-ng              9.3.0               hd4cf53a_17
libtasn1                  4.16.0               h27cfd23_0
libtiff                   4.2.0                h85742a9_0
libunistring              0.9.10               h27cfd23_0
libuv                     1.40.0               h7b6447c_0
libwebp                   1.2.0                h89dd481_0
libwebp-base              1.2.0                h27cfd23_0
lz4-c                     1.9.3                h295c915_1
markdown                  3.3.4              pyhd8ed1ab_0    conda-forge
mkl                       2020.2                      256
mkl-service               2.3.0            py39he8ac12f_0
mkl_fft                   1.3.0            py39h54f3939_0
mkl_random                1.0.2            py39h63df603_0
multidict                 5.1.0            py39h27cfd23_2
ncurses                   6.2                  he6710b0_1
nettle                    3.7.3                hbbd107a_1
ninja                     1.10.2               hff7bd54_1
numpy                     1.19.2           py39h89c1606_0
numpy-base                1.19.2           py39h2ae0177_0
oauthlib                  3.1.1              pyhd8ed1ab_0    conda-forge
olefile                   0.46               pyhd3eb1b0_0
openh264                  2.1.0                hd408876_0
openssl                   1.1.1l               h7f8727e_0
packaging                 21.0               pyhd8ed1ab_0    conda-forge
pillow                    8.4.0            py39h5aabda8_0
pip                       21.2.4           py39h06a4308_0
protobuf                  3.15.8           py39he80948d_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20                       py_2
pydeprecate               0.3.1              pyhd8ed1ab_0    conda-forge
pyjwt                     2.3.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_3    conda-forge
python                    3.9.7                h12debd9_1
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.10.0              py3.9_cpu_0    pytorch
pytorch-lightning         1.4.9              pyhd8ed1ab_0    conda-forge
pytorch-mutex             1.0                         cpu    pytorch
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1            py39h3811e60_0    conda-forge
readline                  8.1                  h27cfd23_0
requests                  2.26.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
rsa                       4.7.2              pyh44b312d_0    conda-forge
scipy                     1.6.2            py39h91f5cce_0
setuptools                58.0.4           py39h06a4308_0
six                       1.16.0             pyhd3eb1b0_0
sqlite                    3.36.0               hc218d9a_0
tensorboard               2.6.0              pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.6.0            py39h3da14fd_0    conda-forge
tensorboard-plugin-wit    1.8.0              pyh44b312d_0    conda-forge
tk                        8.6.11               h1ccaba5_0
torchmetrics              0.5.1              pyhd8ed1ab_0    conda-forge
torchvision               0.11.1                 py39_cpu  [cpuonly]  pytorch
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
typing-extensions         3.10.0.2             hd3eb1b0_0
typing_extensions         3.10.0.2           pyh06a4308_0
tzdata                    2021a                h5d7bf9c_0
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
werkzeug                  2.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.37.0             pyhd3eb1b0_1
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.6.3            py39h3811e60_2    conda-forge
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.9                haebb681_0

Could you please help me with resolving the issue?

Output: Seems you have the wrong combination of PyTorch, CUDA, and Python version, you have installed PyTorch py3.9_cpu_0 which indicates that it is CPU version, not GPU. What I see is that you ask or have installed for PyTorch 1.10.0 which so far I know the Py3.9 built with CUDA 11 support only. See list of available (compiled) versions for CUDA 10.2 and CUDA 11.3.


Instruction: DCGAN understanding generator update step

Input:
Here is some DCGAN example in Pytorch:
https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#training
I wonder why we don't zero discriminator gradients before we update generator? (I added line in original code # netD.zero_grad() # Q: why we don't do this?)
Is it because gradients accumulate in some separate variable and don't affect gradients calculation in generator and only can affect update of weights, but on generator update step we only update generator weights so it's fine?
Here is code section from training loop:
############################
# (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
###########################
## Train with all-real batch
netD.zero_grad()
# Format batch
real_cpu = data[0].to(device)
b_size = real_cpu.size(0)
label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
# Forward pass real batch through D
output = netD(real_cpu).view(-1)
# Calculate loss on all-real batch
errD_real = criterion(output, label)
# Calculate gradients for D in backward pass
errD_real.backward()
D_x = output.mean().item()

## Train with all-fake batch
# Generate batch of latent vectors
noise = torch.randn(b_size, nz, 1, 1, device=device)
# Generate fake image batch with G
fake = netG(noise)
label.fill_(fake_label)
# Classify all fake batch with D
output = netD(fake.detach()).view(-1)
# Calculate D's loss on the all-fake batch
errD_fake = criterion(output, label)
# Calculate the gradients for this batch, accumulated (summed) with previous gradients
errD_fake.backward()
D_G_z1 = output.mean().item()
# Compute error of D as sum over the fake and the real batches
errD = errD_real + errD_fake
# Update D
optimizerD.step()

############################
# (2) Update G network: maximize log(D(G(z)))
###########################
netG.zero_grad()
# netD.zero_grad() # Q: why we don't do this?
label.fill_(real_label)  # fake labels are real for generator cost
# Since we just updated D, perform another forward pass of all-fake batch through D
output = netD(fake).view(-1)
# Calculate G's loss based on this output
errG = criterion(output, label)
# Calculate gradients for G
errG.backward()
D_G_z2 = output.mean().item()
# Update G
optimizerG.step()

Output:
.. gradients accumulate in some separate variable ..

Yes. That is correct. They are two storages (read "variables") - one for parameters and another for its gradients.

.. don't affect gradients calculation in generator ..

Also correct. Let's look at the computation graph for the relevant part
(x) -----> netG ----> (fake) -> netD -> (output)
             ^                   ^
(G_params) --|                   |
                                 |
                (D_params) ------|

Here, things inside () are tensors and rest are "pure computation units". I have explicitly depicted the parameters as inputs to their own computation units. When you hit .backward(), the following computations take place (read backwards)
(GRAD_x) <----- netG <---- (GRAD_fake) <- netD <- (GRAD_output)
                  |                         ^
(GRAD_G_params)<--|                         |
                                            |
                       (D_params) ----------|

I have replaced each variable with their gradients (i.e. GRAD_*) in the backward pass.
You see, there is no GRAD_D_params because its just not required to go further into netG since netG has no role in producing D_params (they are standalone entities associated only with netD).
Simply stated, just because the flow is "going through" netD does not mean it is required to compute/use gradient w.r.t its parameters. Computation of GRAD_fake does not require GRAD_D_params (however it requires D_params), which is why there is no reason to clear up the old GRAD_D_params. But technically, there is no harm either in clearing up GRAD_D_params.

Question in comment: How GRAD_fake is obtained with GRAD_output and D_params ?
Its simple chain rule. Let's rename the variable
fake -> x
netD -> f()
output -> y
D_params -> p
GRAD_output -> del_L/del_y

Then the forward computation simplifies to y = f(x, p). It doesn't matter what the form of f() is, computing gradient of x w.r.t some loss L

would require two things, 1. , which is basically GRAD_output; 2.  or equivalently , which is basically a complicated function involving (x, p) (not their derivatives).


Instruction: Issue plotting a simple tensor with Torch

Input:
Just beginning to use Pyorch, and I am trying to plot a very simple, 1-D array Tensor onto a histogram with Matplotlib.
 torch.manual_seed(8436)
 a = torch.Tensor(1000)

 a.normal_(0, 2.) #This will fill our array with a normal distribution
 plt.hist(a);

However, the result is strange..., and just consists of a bunch of vertical, multicolored lines.
The result I am supposed to get, which I do when entering:
plt.hist(a.numpy())

is the normal histogram.
Thanks in advance for any help!

Output:
ok, it seems to me that here is the reason:
cbook._reshape_2D is used to preprocess the data coming into plt.hist . in 3.4.3  it returns a list of arrays with only one element each, which obviously produces the wrong image above.
in 3.2.2 , however, it returns a list with one 1D array, basically a NumPy version of the tensor we provided. and this one is plotted as expected.
I downgraded the package and it worked. Would be interested to hear other solutions.


Instruction: RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int': Pytorch

Input:
So, I was trying to code a chatbot using Pytorch following this tutorial.
Code: (Minimal, Reproducible one)
tags = []
for intent in intents['intents']:
    tag = intent['tag']
    tags.append(tag)

tags = sorted(set(tags))

X_train = []
X_train = np.array(X_train)

class ChatDataset(Dataset):
    def __init__(self):
        self.n_sample = len(X_train)
        self.x_data = X_train

#Hyperparameter
batch_size = 8
hidden_size = 47
output_size = len(tags)
input_size = len(X_train[0])
learning_rate = 0.001
num_epochs = 1000

dataset = ChatDataset()
train_loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=0)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # using gpu
model = NeuralNet(input_size, hidden_size, output_size).to(device)

# loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(device)

        #forward
        outputs = model(words)
        loss = criterion(outputs, labels) #the line where it is showing the problem

        #backward and optimizer step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch +1) % 100 == 0:
        print(f'epoch {epoch+1}/{num_epochs}, loss={loss.item():.4f}')

print(f'final loss, loss={loss.item():.4f}')

Full Code (if needed)
I am getting this error while trying to get the loss function.
RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'
Traceback:
Traceback (most recent call last): File "train.py", line 91, in <module> loss = criterion(outputs, labels) File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 1150, in forward return F.cross_entropy(input, target, weight=self.weight, File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\functional.py", line 2846, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'
But looking into the tutorial, it seems to work perfectly there whereas it is not in my case.
What to do now?
Thanks.

Output:
In my case, I solved this problem by converting the type of targets to torch.LongTensor before storing the data into the GPU as follows:
for inputs, targets in data_loader:
    targets = targets.type(torch.LongTensor)   # casting to long
    inputs, targets = inputs.to(device), targets.to(device)
    ...
    ...

    loss = self.criterion(output, targets)


Instruction: How to apply torchvision transformations to zip file

Input:
I was trying to download CelebA data set and apply transformation to it via code:
from torchvision import transforms
from torchvision.datasets import CelebA

celeba_transforms = transforms.Compose([
    transforms.CenterCrop(140),
    transforms.Resize([64, 64]),
    transforms.ToTensor()
    ])
CelebA(root='path',
       split='train',
       download=True,
       transform=celeba_transforms)

However I obtained error: BadZipFile: File is not a zip file. I did some research about this error and it seems that its quite popular and not easy to solve (since the problem is with Google Drive which has a daily maximum quota for any file, which seems to be exceeded for the CelebA files).
My solution then was to simply download file from Kaggle in zip. However then I have plain non-transformed CelebA data.
Is there any possibility to still apply celeba_transforms to this data?
EDIT
CelebA(root='archive_celeba.zip',
       split='train',
       download=False,
       transform=celeba_transforms)

I obtained error:
RuntimeError: Dataset not found or corrupted. You can use download=True to download it

Do you know what I'm doing wrong?

Output:
According to documentation from pytorch:

download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.

you can set download to false and then it will still perform the transformations on the local dataset.


Instruction: How to design a joint loss function with two component with the aim of minimizing the first loss but maximizing the second loss?

Input:
I'm trying to do an experiment where there're two subtasks and the aim is to reduce the error rate of the 1st task and increase the error rate of the second task at the same time.
This setting may be similar with that of multi-task learning or adversarial learning.
And now my designed loss function is as follows:
total_loss = loss1 - alpha * loss2
where I just added a weight alpha to make sure that the second loss won't cover the influence of the loss1 totally.
And the result shows that, after training for a few epochs, the total loss get to be minus, and it
decrease in a quite high speed. I assume that is because loss1 is already close to 0 but loss2 is still getting smaller (Increasing the error rate is much more easier than reducing that).
I have never read a paper where a minus loss is added to the original loss function, so I am wondering is that appropriate to use such a loss function, or is that a better design for my experiment setting? And Is there any paper with similar aim of optimization?

Output:
First, let me explain why your loss will not work and will sharply drop to the negatives.
total_loss = loss1 - alpha*loss2

You want to minimize loss1 and maximize loss2, and subsequently combined the two opposing objectives into a single total_loss.
And then you are most likely training your model/system while minimizing total_loss. As it stands, it doesn't matter what alpha you use. The theoretical absolute minimum for a typical loss (crossentropy, mse) is 0. But because of the negative term in your loss, it can be minimized to negative infinity, so you can't stop it from exploding in the negative direction.
Now that we have the explanation, we can think of potential solutions. Since the problem is that your loss tends to converge to negative infinity, we have to find some other operation whose output tends to decrease with increasing input.
If we keep it simple, we could just try using an inverse.
total_loss = loss_1 + 1 / (loss_2 + epsilon)

The above objective should try to maximize loss_2 to make 1/loss_2 near 0.
Another option could be using tanh, which is bounded at (-1, 1). Sigmoid can be used as well.
total_loss = loss_1 + 1 - tanh(loss2)
total_loss = loss_1 + 1 - sigmoid(loss2)

There are probably other better ways to do this too.
Lastly, you need to revisit some questions:
Any learning problem has an end goal. What is your theoretical optimum? Is it system 1's loss is minimized to ~0 and system 2's loss to be maximized to infinity (or some large value)? Does system 2 start from a optimal position in the first place?
I believe you should also review your approach. Look into adversarial learning approaches, like GANs.


Instruction: Linear Regression in PyTorch

Input:
It's a simple regression problem. But no matter how much I try, I can't get the answer I want. I'm guessing the weight should be 32 (4 * 8) but, the code returns 25. Why is that?
This is my full source code:
import torch
import torch.nn as nn
import torch.optim as op

X = torch.FloatTensor([[1., 2.],[2., 4.],[3., 6.]])
Y = torch.FloatTensor([[2.],[8.],[18.]])

class TEST(nn.Module):
    def __init__(self):
        super(TEST,self).__init__()
        self.l1 = nn.Linear(2,1)

    def forward(self, input):
        x = self.l1(input)
        return x

epochs = 2000
lr = 0.001

model = TEST()
loss_func = nn.MSELoss()
optimizer = op.SGD(model.parameters(), lr=lr)

for epoch in range(epochs):
    optimizer.zero_grad()
    output = model(X)
    loss = loss_func(output, Y)
    loss.backward()
    optimizer.step()

    if epoch%10 == 0:
        print('loss[{}] : {}'.format(epoch, loss))

XX = torch.FloatTensor([[4., 8.]])

print(model(XX))

This is the output of the code:
loss[1920] : 0.8891088366508484
loss[1930] : 0.8890921473503113
loss[1940] : 0.8890781402587891
loss[1950] : 0.8890655636787415
loss[1960] : 0.8890505433082581
loss[1970] : 0.8890388011932373
loss[1980] : 0.889029324054718
loss[1990] : 0.8890181183815002
tensor([[25.3124]], grad_fn=<AddmmBackward>)

Output: You are trying to approximate y = x1*x2 but are using a single linear layer i.e. a purely linear model. Ultimately, what happens is you are learning weights a and b such that y = a*x1 + b*x2. However, this model cannot approximate the distribution of x1, x2 -> x1*x2.


Instruction: Runtime Error: mat1 and mat2 shapes cannot be multiplied in pytorch

Input:
I'm new to deep learning and I have created a model using the code below for the prediction of plant disease
class CNN_Model(nn.Module):
  def __init__(self):
    super(CNN_Model, self).__init__()
    self.cnn_model = nn.Sequential(
        nn.Conv2d(3, 16, 3),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(16, 32, 5),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),
    )

    self.fc_model = nn.Sequential(
        nn.Flatten(),
        nn.Linear(800, 300),
        nn.ReLU(),
        nn.Linear(300, 38),
        nn.Softmax(dim=1)
    )

  def forward(self, x):
      x = self.cnn_model(x)
      x = self.fc_model(x)

      return x

model = CNN_Model()

out = model(imgs)
out

When I'm trying to run the above code, I'm getting the error mat1 and mat2 cannot be multiplied. I have tried the answers posted for questions similar to this, but still, my issue is not solved.
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_66/1768380315.py in <module>
----> 1 out = model(imgs)
      2 out

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_66/1577403502.py in forward(self, x)
     26   def forward(self, x):
     27       x = self.cnn_model(x)
---> 28       x = self.fc_model(x)
     29
     30       return x

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    137     def forward(self, input):
    138         for module in self:
--> 139             input = module(input)
    140         return input
    141

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
     94
     95     def forward(self, input: Tensor) -> Tensor:
---> 96         return F.linear(input, self.weight, self.bias)
     97
     98     def extra_repr(self) -> str:

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1845     if has_torch_function_variadic(input, weight):
   1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1847     return torch._C._nn.linear(input, weight, bias)
   1848
   1849

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x119072 and 800x300)

Please someone help me to solve these errors.

Output:
The size mismatch error is shown as 32x119072 and 800x300. The first shape refers to the input tensor, while the second is the parameter of the layer. If you look into your model definition you will see that it matches the first fully connected layer, the one following the flatten. Indeed, nn.Linear(800, 300) was expecting 800-feature tensors, but got 119072-feature tensors instead.
You need to modify this linear layer to match the incoming tensor flattened spatial shape. But notice how this value will depend on the image that is fed to the CNN, ultimately this will dictate the size of the tensor fed to the classifier. The general way to solve this is to use adaptive layers: such as nn.AdaptiveMaxPool2d which will always provide the same output shape regardless of the input dimension size.


Instruction: Huggingface giving pytorch index error on sentiment analysis task

Input:
I am trying to run sentiment analysis on a dataset of millions of tweets on the server. I am calling a API prediction function that takes a list of 100 tweets and iterate over the test of each tweet to return the huggingface sentiment value, and writes that sentiment to a solr database. However, after the process of few hundred tweets, I get the below error, any suggestions?
API code:
from transformers import pipeline

model = pipeline(task = 'sentiment-analysis',model="finiteautomata/bertweet-base-sentiment-analysis")

# huggingface sentiment analyser
def huggingface_sent(sentence):
    text=preprocess(sentence)
    if (len(text)>0):
        predicted_dic = {'NEG': 'Negative','NEU':'Neutral', 'POS':'Positive'}
        return predicted_dic[model(text)[0]['label']]
    else:
        return 'Neutral'

def predict_list(tweets):
    print('Data Processing\n')
    predictions={}
    for t_id in tweets.keys():
        if(tweets[t_id]['language']=='en'):
            predictions[t_id] = huggingface_sent(str(tweets[t_id]['full_text']))
        else:
            predictions[t_id]='NoneEnglish'

    print('processed ', len(tweets.keys()))
    print('\n first element is ', predictions[t_id])
    return predictions

print('Running analyser ....\n')

Error log:

Token indices sequence length is longer than the specified maximum
sequence length for this model (211 > 128). Running this sequence
through the model will result in indexing errors [2021-11-01
12:24:20,649] ERROR in app: Exception on /api/predict [POST] Traceback
(most recent call last):   File
"/myusername/anaconda3/lib/python3.8/site-packages/flask/app.py", line
2447, in wsgi_app
response = self.full_dispatch_request()   File "/myusername/anaconda3/lib/python3.8/site-packages/flask/app.py", line
1952, in full_dispatch_request
rv = self.handle_user_exception(e)   File "/myusername/anaconda3/lib/python3.8/site-packages/flask/app.py", line
1821, in handle_user_exception
reraise(exc_type, exc_value, tb)   File "/myusername/anaconda3/lib/python3.8/site-packages/flask/_compat.py",
line 39, in reraise
raise value   File "/myusername/anaconda3/lib/python3.8/site-packages/flask/app.py", line
1950, in full_dispatch_request
rv = self.dispatch_request()   File "/myusername/anaconda3/lib/python3.8/site-packages/flask/app.py", line
1936, in dispatch_request
return self.view_functionsrule.endpoint   File "/mnt/raid1/diil/sentiment_api/analyser_main.py", line 11, in
api_predict_list
predictions = predict_list(tweets)   File "/mnt/raid1/diil/sentiment_api/analyser_core.py", line 84, in
predict_list
predictions[t_id] = huggingface_sent(str(tweets[t_id]['full_text']))   File
"/mnt/raid1/diil/sentiment_api/analyser_core.py", line 70, in
huggingface_sent
if model(text):   File "/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/text_classification.py",
line 126, in call
return super().call(*args, **kwargs)   File "/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/base.py",
line 915, in call
return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)   File
"/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/text_classification.py",
line 172, in run_single
return [super().run_single(inputs, preprocess_params, forward_params, postprocess_params)]   File
"/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/base.py",
line 922, in run_single
model_outputs = self.forward(model_inputs, **forward_params)   File
"/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/base.py",
line 871, in forward
model_outputs = self._forward(model_inputs, **forward_params)   File
"/myusername/anaconda3/lib/python3.8/site-packages/transformers/pipelines/text_classification.py",
line 133, in _forward
return self.model(**model_inputs)   File "/myusername/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 1051, in _call_impl
return forward_call(*input, **kwargs)   File "/myusername/anaconda3/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py",
line 1198, in forward
outputs = self.roberta(   File "/myusername/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 1051, in _call_impl
return forward_call(*input, **kwargs)   File "/myusername/anaconda3/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py",
line 841, in forward
embedding_output = self.embeddings(   File "/myusername/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 1051, in _call_impl
return forward_call(*input, **kwargs)   File "/myusername/anaconda3/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py",
line 136, in forward
position_embeddings = self.position_embeddings(position_ids)   File
"/myusername/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 1051, in _call_impl
return forward_call(*input, **kwargs)   File "/myusername/anaconda3/lib/python3.8/site-packages/tousername/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py",
line 2043, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) IndexError: index out of range in
selfusername/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py",
line 2043, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) IndexError: index out of range in self

Output:
As @Quang Hoang mentioned in the comment, it seems the problem is due to the length of your input tweet. Fortunately, you are able to determine the behavior of the tokenizer in pipeline class and truncate longer tweets explicitly. In addition, it's possible to set any other argument for pipeline elements.
MODEL_CHECKPOINT = "finiteautomata/bertweet-base-sentiment-analysis"
ner_pipeline = pipeline(task="sentiment-analysis", tokenizer=(MODEL_CHECKPOINT, {'model_max_length': 128}), model="finiteautomata/bertweet-base-sentiment-analysis")

As a side note, I recommend using the approach presented in this answer to accelerate the entire process.


Instruction: Is there any way to create a tensor with a specific pattern in Pytorch?

Input:
I'm working with linear transformation in the form of Y=Q(X+A), where X is the input tensor and Y is the output, Q and A are two tensors to be learned. Q is an arbitrary tensor, therefore I can use nn.Linear. But A is a (differentiable) tensor that has some specific pattern, as a short example,
A = [[a0,a1,a2,a2,a2],
     [a1,a0,a1,a2,a2],
     [a2,a1,a0,a1,a2],
     [a2,a2,a1,a0,a1],
     [a2,a2,a2,a1,a0]].

So I cannot define such a pattern in nn.Linear. Is there any way to define such a tensor in Pytorch?

Output:
This looks like a Toeplitz matrix. A possible implementation in PyTorch is:
def toeplitz(c, r):
    vals = torch.cat((r, c[1:].flip(0)))
    shape = len(c), len(r)
    i, j = torch.ones(*shape).nonzero().T
    return vals[j-i].reshape(*shape)

In your case with a0 as 0, a1 as 1 and a2 as 2:
>>> toeplitz(torch.tensor([0,1,2,2,2]), torch.tensor([0,1,2,2,2]))
tensor([[0, 1, 2, 2, 2],
        [1, 0, 1, 2, 2],
        [2, 1, 0, 1, 2],
        [2, 2, 1, 0, 1],
        [2, 2, 2, 1, 0]])

For a more detailed explanation refer to my other answer here.


Instruction: Pip install from source without building a wheel

Input:
I have a Python package that includes large PyTorch model checkpoints. I try including those in my setup.py as
package_data = {'mypackage': ['model_weights/*', 'model_weights/sequential_models*']},

Now the problem is whenever I try to install from the source via pip install mypackage/ --no-cache-dir I get a MemoryError. I tried debugging with --verbose and realized that this happens at
creating '/tmp/pip-wheel-bs29bp6a/tmpp0itbxn1/mypackage-1.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'mypackage/model_weights/distilled_model.pt'
adding 'mypackage-1.0.dist-info/RECORD'
Traceback (most recent call last):
...
File "/zhome/1d/8/153438/miniconda3/envs/testenv/lib/python3.9/zipfile.py", line 1127, in write
      data = self._compressor.compress(data)
MemoryError
Building wheel for mypackage (PEP 517) ... error
ERROR: Failed building wheel for mypackage

I really only want the installation to copy over the files in model_weights/ to the installation directory. Including them in the wheel appears to be impossible.
Is there a way to suppress this step when running pip install? The package will only be distributed as a source, never on PyPI, as the model_weights files are far too large anyway.

Output:
You can run
$ pip install mypackage/ --no-cache-dir --no-binary=mypackage

to skip wheel building (assuming mypackage is actually your distribution name - this is what you pass as name to setup() function).


Instruction: Is there a way to compute a circulant matrix in Pytorch?

Input:
I want a similar function as in https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.circulant.html to create a circulant matrix using PyTorch. I need this as a part of my Deep Learning model (in order to reduce over-parametrization in some of my Fully Connected layers as suggested in https://arxiv.org/abs/1907.08448 (Fig.3))
The input of the function shall be a 1D torch tensor, and the output should be the 2D circulant matrix.

Output:
You can make use of unfold to extract sliding windows. But to get the correct order you need to flip (later unflip) the tensors, and first concatenate the flipped tensor to itself.
circ=lambda v:torch.cat([f:=v.flip(0),f[:-1]]).unfold(0,len(v),1).flip(0)


Instruction: what is equivalent to torch.load() in tensorflow?

Input: I want to know if there is any way to see the parameters of models in tensorflow. there is a command in pytorch i.e. torch.load('/filepath').

Output:
Provided that you already have a model saved at MODEL_PATH, this should do the trick:
model = tf.keras.models.load_model(MODEL_PATH)
model.summary()

Check this out for more info on saving and loading models.


Instruction: Can't install Pytorch in Pycharm terminal, Python 3.10 .win 10

Input:
I go to pytorch site and take this
pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

I have windows 10 ,Python version is 3.10 ,CUDA version is 11.5
And I get this error

ERROR: Could not find a version that satisfies the requirement
torch==1.10.0+cu113 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.10.0+cu113

I really struggled and tried to solve it, Please help.

Output:
Check out the official issue on Pytorch's Github repository.
I've tried your exact command on python 3.9.5 and it works. I believe the issue is that PyTorch is not supported by python 3.10 yet.
Downgrading to any 3.9 version of Python should solve your problem.


Instruction: How can I read pytorch model file via cv2.dnn.readNetFromTorch()?

Input:
I am able to save a PyTorch custom model? (it can work any PyTorch version above 1.0)
However, I am not able to read the saved model. I am trying to read it via cv2.dnn.readNetFromTorch() so as to use the model in Opencv framework (4.1.0).
I saved the PyTorch model with different methods as follows to see whether this difference reacts to the reading function of cv2.dnn.
        torch.save(model.state_dict(), '/home/aktaseren/people-opencv/pidxx.pt')
        torch.save(model.state_dict(), '/home/aktaseren/people-opencv/pidxx.t7')
        torch.save(model, '/home/aktaseren/people-opencv/pidxx.t7')
        torch.save(model, '/home/aktaseren/people-opencv/pidxx.pth')

None of these saved file can be readable via cv2.dnn.readNetFromTorch().
The error I am getting is always the same on this issue, which is below.
cv2.error: OpenCV(4.1.0) ../modules/dnn/src/torch/torch_importer.cpp:1022: error: (-213:The function/feature is not implemented) Unsupported Lua type in function 'readObject'

Do you have any idea how to solve this issue?

Output:
OpenCV documentation states can only read in torch7 framework format. There is no mention of .pt or .pth saved by pytorch.
This post mentions pytorch does not save as .t7.

.t7 was used in Torch7 and is not used in PyTorch. If I’m not mistaken
the file extension does not change the behavior of torch.save.

An alternate method is to export the model as onnx, then read the model in opencv using readNetFromONNX.


Instruction: libtorch throws c10::error after build on Windows 10 (VS2019)

Input:
I've tried to build libtorch on Windows 10 using VS 2019 without CUDA and Python. Independent if I compile it with or without MKL, a simple test program crashes directly after start. After building the debug version, libtorch throws a c10:error in a function called torchCheckFail.
The function seems to complain about ATen/core/jit_type.h.
The problem is part of torch_cpu.dll. The problem disappears when I'm using the precompiled binaries for Windows.
Here's the function:
void torchCheckFail(
    const char* func,
    const char* file,
    uint32_t line,
    const std::string& msg) {
  throw ::c10::Error({func, file, line}, msg);
}

And here's the call stack:

Output:
I encountered the same exact error with the same environment.
A solution that worked for me was to take a release version of pytorch and not a non-release one (i.e. a release version + some commits).
Hope it helps.


Instruction: Is it possible to train ONNX models developed in tensorflow and pytorch with C++?

Input: I wonder if its possible to use tensorflow and pytorch models converted to onnx models to train them with the C++ Api like it is done in e.g. https://gist.github.com/asimshankar/5c96acd1280507940bad9083370fe8dc with a tensorflow model. I just found examples for inference with onnx. The idea is to be able to prototype with tensorflow and pytorch in python, convert to onnx models and to have a unified API in C++ to do inference and training. It would help quite a lot to get some (links to get) informaton.

Output:
ONNX's GitHub page suggests that it can be used for inference, but it doesn't seem reasonable to be able to train all models with it (from the development perspective).

Currently we focus on the capabilities needed for inferencing (scoring).

Although there are some difficulties, such as always writing backpropagation is more difficult than feedforwarding, and supporting it would double the framework size, which is not what ONNX is aiming for since there are already so many frameworks for this. To train you will need all the parameters, the derivatives of functions in the GPU and CPU(if its performance is lower than other frameworks, it will be a big problem since nobody will use it). And there are many other things that make a unified framework difficult(Supporting training on multiple GPUs over a network, for example).(So In our perspective, it's great, but in theirs it's so difficult)
But we can see that some functionality for training has been added to the framework, in this case it can train transformer models
Also, to training transformers in PyTorch you could see this link


Instruction: Finding out distance between output of two Convolutional Neural Network (CNN) i.e Siamese Network

Input:
I am trying to build a simple Siamese neural network for usage in Human re-identification.
For that, I have used MTCNN (https://github.com/timesler/facenet-pytorch) for face detection and official pytorch implementation of arcface algorithm (https://github.com/deepinsight/insightface/tree/master/recognition/arcface_torch) for CNN implementation.
I have used a pretrained model (ms1mv3_arcface_r50_fp16) trained on resnet 50 and a backbone CNN model from their repository for implementing the CNN. The CNN takes a 112x112 image and produces an array of 512x1.
So I get two arrays as a result of the network. I have tried to compare the two array using cosine similarity but It is not giving me the correct results all the time.
So, do I need to change my model parameters or do I need to use another metric for comparison?
My code: https://gist.github.com/desertSniper87/26f5f45f4cece9d0f3008e89cea94be8

Output:
I've tried and got fully enjoied Elasticsearch with standard dlib face vectors (128x1)... ES can store and search&compare such kind vectors super fast and accurate.
I've used somthing like this to creat a ES index:
from elasticsearch import Elasticsearch

es = Elasticsearch([{'host': ELASTIC_HOST, 'port': ELASTIC_PORT }],
                       timeout=30, retry_on_time=True, max_retries=3,
                       http_auth=(ELASTIC_NAME, ELASTIC_PSW)
                   )

mapping = {
     "mappings": {
     "properties": {
         "face_vector":{
             "type": "dense_vector",
             "dims": 128
              },
         "pic_file": {
             "type": "text"},
         "face_loc": {
             "type": "integer"}
             }
         }
     }
 es.indices.create(index="fr_idx", body=mapping)

and then
s_body = {"size": ELASTIC_MAX_RESULTS, "min_score": tolerance,
                          "query": {
                                "script_score": {
                                    "query": {
                                        "match_all": {}
                                    },
                            "script": {
                                "source": "1 / (1 + l2norm(params.query_vector, 'face_vector'))",
                                "params": {"query_vector": UNKNOWN_FACE_ENCOD}
                                        }
                                    }
                                }
                        }
res = es.search(index="fr_idx", body=s_body) #standard index
for hit in res["hits"]["hits"]:
...

to search similar vectors.


Instruction: Linear decay as learning rate scheduler (pytorch)

Input: I have read about LinearLR and ConstantLR in the Pytorch docs but I can't figure out, how to get a linear decay of my learning rate. Say I have epochs = 10 and lr=0.1 then I want to linearly reduce my learning-rate from 0.1 to 0 (or any other number) in 10 steps i.e by 0.01 in each step.

Output:
The two constraints you have are: lr(step=0)=0.1 and lr(step=10)=0. So naturally, lr(step) = -0.1*step/10 + 0.1 = 0.1*(1 - step/10).
This is known as the polynomial learning rate scheduler. Its general form is:
def polynomial(base_lr, iter, max_iter, power):
    return base_lr * ((1 - float(iter) / max_iter) ** power)

Which in your case would be called with polynomial(base_lr=0.1, max_iter=10, power=1).


Instruction: Does model evaluation during training affect final accuracy in PyTorch?

Input:
During a simple training loop for PyTorch a strange effect was observed.
If the evaluation function is called or not seems to have effects on the final performance of the model.
We train on the CIFAR10 using a very simple MLP model and Adam with 10 training epochs.
We try two Main loops:

After the end of each training epoch we measure the accuracy of validation set

We calculate the validation only once at the end of all training.

We show the difference in code here below:
Main Loop 1:
# Main Loop 1
num_epochs = 10
print(f"num_epochs: {num_epochs}")

for epoch in range(num_epochs):  # loop over the dataset multiple times
    print(f"\nStart Epoch {epoch}")

    model.train()

    train_loss, train_accuracy = training_epoch(trainloader,optimizer,model,criterion)
    print(f"Training Loss: {train_loss:.3f} - Training Accuracy: {train_accuracy:.3f}")

    model.eval()

    with torch.no_grad():
        val_loss, val_accuracy = val_epoch(testloader, model, criterion)
    print(f"Val Loss: {val_loss:.3f} - Val Accuracy: {val_accuracy:.3f}")

print('Finished Training')

Main Loop 2:
# Main Loop 2
num_epochs = 10
print(f"num_epochs: {num_epochs}")

for epoch in range(num_epochs):  # loop over the dataset multiple times
    print(f"\nStart Epoch {epoch}")

    model.train()

    train_loss, train_accuracy = training_epoch(trainloader,optimizer,model,criterion)
    print(f"Training Loss: {train_loss:.3f} - Training Accuracy: {train_accuracy:.3f}")

model.eval()
with torch.no_grad():
    val_loss, val_accuracy = val_epoch(testloader, model, criterion)
print(f"Val Loss: {val_loss:.3f} - Val Accuracy: {val_accuracy:.3f}")

print('Finished Training')

Though there shouldn't be any change, the final performance of model change.

Val Loss: 1.526 - Val Accuracy: 0.523
Val Loss: 1.501 - Val Accuracy: 0.528

Of course for reproducibility, we set all seeds. Moreover, this effect can already be observed at the beginning of the second training epoch.
I share the entire code as a Colab notebook:
https://colab.research.google.com/drive/1BODeKHZmcT8lH3r2bxYVHNR2KOpT9O9Y?usp=sharing

Output:
The observed difference would be due to variance because of stochasticity in the optimization algorithm. The evaluation you perform has no effect on the model's weights.
Also in the link you provided, you are re-initializing a SimpleMLP on both experiments. Since the module's weights get instantiated randomly the inference will naturally yield different results.


Instruction: How to use django rest api to do inference?

Input:
I was trying to build a website with Django rest API as the backend. when given a string Its gives the score from 1 to 10 for negativity.
The frontend part of the website was built using next.js. Previously I have made the same app without Django rest API by doing all inference in the views.py file. Now I am using Rest API I am confused about where should I need to include the machine learning inference code.
I have seen tutorials on the internet showing that inference code is attached in the models.py file. Previously when I included inference code in views.py the page used to get reload whenever I do the inference. I want to avoid it. What is the best practice to include inference code while using Django rest API.

Output:
I do it the following way:
from rest_framework.decorators import api_view
from rest_framework.response import Response

@api_view(['GET', 'POST'])
def snippet_list(request):
    """
    List all code snippets, or create a new snippet.
    """
    if request.method == 'GET':
        # if you have a GET request data do stuffs here, else remove get

    elif request.method == 'POST':
        data = request.data
        # run your inference code here and get the predictions
        context = {
           'score': score
        }
        return Response(context , status=status.HTTP_200_OK)


Instruction: What is the proper way to use pytorch and matplotlib with MKL on Windows?

Input:
Is there any clear instruction on how to actually make this work?
The error message I get is:

Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.

It seems, there are only workarounds, such as to opt out from MKL altogether (not to mention, say, nomkl is not available on Windows) or
import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

which is nasty.

Output:
If you have conda and pip installed on your machine. Please try installing by creating a new conda environment. You could try the below steps  (which I have validated from my end) to install matplotlib and pytorch (with mkl).
conda create -n myenv
conda activate myenv
conda install pytorch torchvision torchaudio cpuonly -c pytorch
C:\Users\{USERNAME}\.conda\envs\myenv\python.exe -m pip install matplotlib==2.2.5

Once installed I verified pytorch uses mkl with the below command
>>> import torch
>>> torch.__config__.show()
'PyTorch built with:\n  - C++ Version: 199711\n  - MSVC 192829337\n  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)\n  - OpenMP 2019\n  - LAPACK is enabled (usually provided by MKL)\n  - CPU capability usage: AVX2\n  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, \n'


Instruction: Pytorch Conv1D gives different size to ConvTranspose1d

Input:
I am trying to build a basic/shallow CNN auto-encoder for 1D time series data in pytorch/pytorch-lightning.
Currently, my encoding block is:
class encodingBlock(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1d_1 = nn.Conv1d(1, 64, kernel_size=32)
        self.relu = nn.ReLU()
        self.batchnorm = nn.BatchNorm1d(64)
        self.maxpool = nn.MaxPool1d(kernel_size=2, stride=2, return_indices=True)
        self.fc = nn.Linear(64, 4)

    def forward(self, x):
        cnn_out1 = self.conv1d_1(x)
        norm_out1 = self.batchnorm(cnn_out1)
        relu_out1 = self.relu(norm_out1)
        maxpool_out, indices = self.maxpool(relu_out1)
        gap_out = torch.mean(maxpool_out, dim = 2)
        fc_out = self.relu(self.fc(gap_out))
        return fc_out, indices

And my decoding block is:
class decodingBlock(nn.Module):
    def __init__(self):
        super().__init__()

        self.Tconv1d_1 = nn.ConvTranspose1d(64, 1, kernel_size=32, output_padding=1)
        self.relu = nn.ReLU()
        self.batchnorm = nn.BatchNorm1d(1)
        self.maxunpool = nn.MaxUnpool1d(kernel_size=2, stride=2)
        self.upsamp = nn.Upsample(size=59, mode='nearest')
        self.fc = nn.Linear(4, 64)

    def forward(self, x, indices):
        fc_out = self.fc(x)
        relu_out = self.relu(fc_out)
        relu_out = relu_out.unsqueeze(dim = 2)
        upsamp_out = self.upsamp(relu_out)
        maxpool_out = self.maxunpool(upsamp_out, indices)
        cnnT_out = self.Tconv1d_1(maxpool_out)
        norm_out = self.batchnorm(cnnT_out)
        relu_out = self.relu(norm_out)
        return relu_out

However, looking at the outputs:
Input size: torch.Size([1, 1, 150])
Conv1D out size: torch.Size([1, 64, 119])
Maxpool out size: torch.Size([1, 64, 59])
Global average pooling out size: torch.Size([1, 64])
Encoder dense out size: torch.Size([1, 4])
...
Decoder input: torch.Size([1, 4])
Decoder dense out size: torch.Size([1, 64])
Unsqueeze out size: torch.Size([1, 64, 1])
Upsample out size: torch.Size([1, 64, 59])
Decoder maxunpool out size: torch.Size([1, 64, 118])
Transpose Conv out size: torch.Size([1, 1, 149])

The outputs from the MaxUnpool1d and ConvTranspose1d layers are not the expected dimension.
I have two questions that I was hoping to get some help on:

Why are the dimensions wrong?
Is there a better way to "reverse" the global average pooling than the upsampling procedure I have used?

Output:
1.  Regarding input and output shapes:
pytorch's doc has the explicit formula relating input and output sizes.
For convolution:

Similarly for pooling:

For transposed convolution:

And for unpooling:

Make sure your padding and output_padding values add up to the proper output shape.
2. Is there a better way?
Transposed convolution has its faults, as you already noticed. It also tends to produce "checkerboard artifacts".
One solution is to use pixelshuffle: that is, predict for each low-res point twice the number of channels, and then split them into two points with the desired number of features.
Alternatively, you can interpolate using a fixed method from the low resolution to the higher one. Apply regular convolutions to the upsampled vectors.
If you choose this path, you might consider using ResizeRight instead of pytorch's interpolate - it has better handling of edge cases.


Instruction: PyTorch with CUDA and Nvidia card: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable, but torch.cuda.is_available() is True

Input:
Problem:
I occasionally get the following CUDA error when running PyTorch scripts with CUDA on an Nvidia GPU, running on CentOS 7.
If I run:
python3 -c 'import torch; print(torch.cuda.is_available()); torch.randn(1).to("cuda")'

I get the following output:
True
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

PyTorch seems to think the GPU is available, but I can't put anything onto it's memory. When I restart the computer, the error goes away. I can't seem to get the error to come back consistently.

Output:
When I'm outside of Python and run nvidia-smi, it shows a process running on the GPU, despite the fact that I cancelled execution of the PyTorch script:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.27.04    Driver Version: 460.27.04    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:00:06.0 Off |                    0 |
| N/A   29C    P0    33W / 250W |   1215MiB / 32510MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     18805      C   python3                          1211MiB |
+-----------------------------------------------------------------------------+

If I kill the process (with PID=18805), by running kill -9 18805, the process no longer appears in nvidia-smi, and the error does not recur.
Any insights on a better solution, or how to avoid this problem in the first place, are very welcome.


Instruction: How to split dataset into two considering fixed seed to ensure reproducibility in PyTorch?

Input:
I am working on one of my University assignment and there is one sub-task which says. Split the data in two (Train and Validation) while using using a fixed seed to ensure reproducibility. I have wrote some code which is working fine but I want to know whether it is the correct way or not?
torch.manual_seed(0)

mnist_train, mnist_val = torch.utils.data.random_split(mnist_rest, [54000,6000])

I am working on MNIST dataset.

Output:
According to PyTorch's docs:
Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.
However, there are some steps you can take to limit the number of sources of nondeterministic behavior for a specific platform, device, and PyTorch release. First, you can control sources of randomness that can cause multiple executions of your application to behave differently. Second, you can configure PyTorch to avoid using nondeterministic algorithms for some operations, so that multiple calls to those operations, given the same inputs, will produce the same result.
To control randomness, it's recommended you to use followings to reproduce the result:
In PyTorch:
You can use torch.manual_seed() to seed the RNG for all devices (both CPU and CUDA):
import torch
torch.manual_seed(0)

In Python:
For custom operators, you might need to set python seed as well:
import random
random.seed(0)

If you or any of the libraries you are using rely on NumPy, you can seed the global NumPy RNG with:
import numpy as np
np.random.seed(0)


Instruction: PyTorch: module 'torch' has no attribute 'gradient'

Input:
PyTorch seems to have a serious bug leading to the error message
AttributeError: module 'torch' has no attribute [some torch function]

In my case, I try to use torch.gradient link.
I am using Python version 3.8.5 and tried the PyTorch Versions 1.6.0, 1.7.0, 1.7.1 , 1.8, 1.9.0 for CPU. (The newest version has another bug for gradient torch.gradient edge order).
There are several answers suggesting that I should install torch via pip, I should install torchvision, nothing worked. There is also the suggestion, that the wrong torch folder is used as a namespace which does not seem to be the case, since there is an initfile when I run
print(torch.__path__)
So my question is:
How to finally solve this problem?

I tried to install it with the recommended pytorch.org version for conda, with anaconda.org/pytorch/pytorch and with pypi.org/project/torch – nothing worked.

The conda list torch element gives
pytorch                   1.7.1               py3.8_cpu_0  [cpuonly]  pytorch
pytorch-mutex             1.0                         cpu    pytorch
torchaudio                0.7.2                      py38    pytorch
torchvision               0.8.2                  py38_cpu  [cpuonly]  pytorch

Output: The same happened to me. What I did was to create a new conda environment and reinstall PyTorch according to https://pytorch.org/