Title: How to build CUDA custom C++ extension for PyTorch without CUDA?
Tags: python;cuda;pytorch
Question: I was tasked with creating a CI workflow for building a PyTorch CUDA extension for this application. Up until now, the application was deployed by creating the target AWS VM with a CUDA GPU, pushing all the sources there and running setup.py, but instead I want to do the build in our CI system and deploy pre-built binaries to the production environment.
When running setup.py in the CI system, I get the error &quot;No CUDA GPUs are available&quot; - which is true, there are no CUDA GPUs in the CI system. Is there a way to just build the CUDA extension without a CUDA GPU available?
This is the error message:
gcc -pthread -shared -B /usr/local/miniconda/envs/build/compiler_compat -L/usr/local/miniconda/envs/build/lib -Wl,-rpath=/usr/local/miniconda/envs/build/lib -Wl,--no-as-needed -Wl,--sysroot=/ /app/my-app/build/temp.linux-x86_64-3.6/my-extension/my-module.o -L/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.6/my-extension/my-module.cpython-36m-x86_64-linux-gnu.so
building 'my-extension.my-module._cuda_ext' extension
creating /app/my-app/build/temp.linux-x86_64-3.6/my-extension/src
Traceback (most recent call last):
File &quot;setup.py&quot;, line 128, in <module&gt;
'build_ext': BuildExtension
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/__init__.py&quot;, line 153, in setup
return distutils.core.setup(**attrs)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/core.py&quot;, line 148, in setup
dist.run_commands()
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/dist.py&quot;, line 955, in run_commands
self.run_command(cmd)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/dist.py&quot;, line 974, in run_command
cmd_obj.run()
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/command/build_ext.py&quot;, line 79, in run
_build_ext.run(self)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py&quot;, line 339, in run
self.build_extensions()
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py&quot;, line 653, in build_extensions
build_ext.build_extensions(self)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py&quot;, line 448, in build_extensions
self._build_extensions_serial()
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py&quot;, line 473, in _build_extensions_serial
self.build_extension(ext)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/command/build_ext.py&quot;, line 196, in build_extension
_build_ext.build_extension(self, ext)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py&quot;, line 533, in build_extension
depends=ext.depends)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py&quot;, line 468, in unix_wrap_ninja_compile
cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py&quot;, line 377, in unix_cuda_flags
cflags + _get_cuda_arch_flags(cflags) +
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py&quot;, line 1407, in _get_cuda_arch_flags
capability = torch.cuda.get_device_capability()
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py&quot;, line 291, in get_device_capability
prop = get_device_properties(device)
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py&quot;, line 296, in get_device_properties
_lazy_init()  # will define _get_device_properties
File &quot;/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py&quot;, line 172, in _lazy_init
torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
I'm not very familiar with CUDA and only half proficient in Python (I'm here as the &quot;ops&quot; part of &quot;devops&quot;).
Comment: I am guessing that the pytorch build system tries to detect the GPU and guess target architecture settings from that for the build. That isn't anything controlled by CUDA, it is something you probably have to manually override or modify within the build system itself

Here is the accepted answer: It is not a complete solution, as I lack details to completely figure out a solution. But it should help you or your teammates.
So first based on the source code, it is not required to reach torch._C._cuda_init() if you have CUDA arch flags set.
This means the pytorch is trying to figure out the CUDA arch because it is not specified by the user.
Here is a related thread. As you can see, setting the TORCH_CUDA_ARCH_LIST environment to something that fits your environment should work for you.
Comment for this answer: That worked great! just setting `TORCH_CUDA_ARCH_LIST=Turing` completed the compile. I'll have to see how well it works like that on AWS G4 instances, but for now it seems to work.


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 1, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:05/07/21', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><cuda><pytorch>', 'Title': 'How to build CUDA custom C++ extension for PyTorch without CUDA?', 'ViewCount': 677, 'id': 67430901, 'user_AboutMe': "<p>I'm a self taught software developer, system administrator and all-round code-guy.</p>\n\n<p>I've been doing software development, QA, developer support and system and even some graphic design work for as long as I can remember (going back 30 years), both on commercial projects as well as open source and free software, and I enjoy both.</p>\n\n<p>Coding is fun, that's why it is worth doing - I hope it never becomes a chore :-)</p>\n", 'user_AccountId': 21889.0, 'user_DisplayName': 'Guss', 'user_Reputation': 28034.0, 'user_Views': 1549.0, 'user_id': 53538.0}}