Space startup never succeeds or ends, and 'Runtime error' when cloned
Hello everyone, I’ve contacted HuggingFace support, on 3@dressemails, but with no success…My space HF, which had been running fine for several weeks, suddenly, one day, although absolutely no action had been taken on it, failed to initialize when it comes to wake it up from sleeping mode, whatever the execution server and its resources, the launch phase lasts forever (message "Builing on T4 non stop for instance » for exemple), and produces no log.Switching to “dev mode”, which would have been very useful for debugging, is also impossible…(nothing happen when i press the button (of « dev mode), just a refresh behavior of the page but then, nothing happen, not even one log text).Here is the name of the Space : « … /spaces/NoQuest/QP_AN «Because i don’t want to fire the “Factory Rebuild” function…(i don’t want to take the risk to lose any settings that i set at the times… Or loose any datas…or Git, and Git LFS states… and knowing also that a lot of heavy models files are there in the disk),…so, i made a duplicate of the space (which is in public visibility on this path> « spaces/NoQuest/QP_AN_/settings" ), and when i start that clone space,It produced the logs that I provide here, just under my signature.But i’m more interested on using the original application of course (not the cloned version)And Finally, if i make a Factory Rebuild on the clone, the same error happen, wich the same log in the attached file.Anyway, i can’t access the space in SSH also on that clone version of my space.It s especially that fact that made me create this help ask cause if it s not, I would already investigate the problem by myself.Thanks in advance for your help;The famouse log :"Runtime errorApp process crashedContainer logs:===== Application Startup at 2024-11-18 23:20:10 =====FalseThe following directories listed in your path were found to be non-existent: {PosixPath(‘/usr/local/etc/pyenv.d’),PosixPath(‘/usr/lib/pyenv/hooks’), PosixPath(‘/usr/etc/pyenv.d’), PosixPath(‘/etc/pyenv.d’)}The following directories listed in your path were found to be non-existent: {PosixPath(‘Europe/Paris’)}The following directories listed in your path were found to be non-existent: {PosixPath(‘//172.20.0.1’), PosixPath(‘tcp’),PosixPath(‘443’)}The following directories listed in your path were found to be non-existent: {PosixPath(‘NoQuest/QP_AN_’)}The following directories listed in your path were found to be non-existent: {PosixPath(‘//172.20.0.1’), PosixPath(‘tcp’),PosixPath(‘443’)}CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths…DEBUG: Possible options found for libcudart.so: {PosixPath(‘/usr/local/cuda/lib64/libcudart.so’)}CUDA SETUP: PyTorch settings found: CUDA_VERSION=124, Highest Compute Capability: 7.5.CUDA SETUP: To manually override the PyTorch CUDA version pleasesee: github /TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.mdCUDA SETUP: Required library version not found: libbitsandbytes_cuda124.so. Maybe you need to compile it fromsource?CUDA SETUP: Defaulting to libbitsandbytes_cpu.so…================================================ERROR=====================================CUDA SETUP: CUDA detection failed! Possible reasons:You need to manually override the PyTorch CUDA version. Please see:"github /TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.mdCUDA driver not installedCUDA not installedYou have multiple conflicting CUDA librariesRequired library not pre-compiled for this bitsandbytes release!CUDA SETUP: If you compiled from source, try again withmake CUDA_VERSION=DETECTED_CUDA_VERSIONfor example,make CUDA_VERSION=113.CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version viaconda list | grep cuda.================================================================================CUDA SETUP: Something unexpected happened. Please compile from source:git clone github /TimDettmers/bitsandbytes.gitcd bitsandbytesCUDA_VERSION=124python setup.py installCUDA SETUP: Setup Failed!Traceback (most recent call last):File “/home/user/app/server.py”, line 35, infrom modules import chat, loaders, presets, shared, training, ui, utilsFile “/home/user/app/modules/chat.py”, line 17, infrom modules.text_generation import (File “/home/user/app/modules/text_generation.py”, line 22, infrom modules.models import clear_torch_cache, local_rankFile “/home/user/app/modules/models.py”, line 10, infrom accelerate import infer_auto_device_map, init_empty_weightsFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/accelerate/init.py”, line 3, infrom .accelerator import AcceleratorFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/accelerate/accelerator.py”, line 35, infrom .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_stateFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/accelerate/checkpointing.py”, line 24, infrom .utils import (File “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/accelerate/utils/init.py”, line 131, infrom .bnb import has_4bit_bnb_layers, load_and_quantize_modelFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/accelerate/utils/bnb.py”, line 42, inimport bitsandbytes as bnbFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/init.py”, line 6, infrom . import cuda_setup, utils, researchFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/research/init.py”, line 1, infrom . import nnFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py”, line 1,infrom .modules import LinearFP8Mixed, LinearFP8GlobalFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py”, line 8,infrom bitsandbytes.optim import GlobalOptimManagerFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/optim/init.py”, line 6, infrom bitsandbytes.cextension import COMPILED_WITH_CUDAFile “/home/user/.pyenv/versions/3.10.15/lib/python3.10/site-packages/bitsandbytes/cextension.py”, line 20, inraise RuntimeError(‘’’RuntimeError:CUDA Setup failed despite GPU being available. Please run the following command to get more information:python -m bitsandbytesInspect the output of the command and see if you can locate CUDA libraries. You might need to add themto your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytesand open an issue at: github /TimDettmers/bitsandbytes/issues"
2024-11-21T02:45:19Z
At HF Spaces, the situation has improved a lot since a while ago, but there are still quite a few errors.NoQuest/QP_AN_Not found…a lot of heavy models files are there in the diskIf you have model files that are several GB or larger, or other large files in Spaces, the build may fail and Spaces may not start up, which may be related to this.