Instruction: "AssertionError: Torch not compiled with CUDA enabled" in spite upgrading to CUDA version

Input:
I figured out this is a popular question, but still I couldn't find a solution for that.

I'm trying to run a simple repo Here which uses PyTorch. Although I just upgraded my Pytorch to the latest CUDA version from pytorch.org (1.2.0), it still throws the same error. I'm on Windows 10 and use conda with python 3.7.

    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

How to fix the problem?

Here is my conda list:

# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0    anaconda
_pytorch_select           1.1.0                       cpu    anaconda
_tflow_select             2.3.0                       mkl    anaconda
absl-py                   0.7.1                    pypi_0    pypi
alabaster                 0.7.12                   py37_0    anaconda
anaconda                  2019.07                  py37_0    anaconda
anaconda-client           1.7.2                    py37_0    anaconda
anaconda-navigator        1.9.7                    py37_0    anaconda
anaconda-project          0.8.3                      py_0    anaconda
argparse                  1.4.0                    pypi_0    pypi
asn1crypto                0.24.0                   py37_0    anaconda
astor                     0.8.0                    pypi_0    pypi
astroid                   2.2.5                    py37_0    anaconda
astropy                   3.2.1            py37he774522_0    anaconda
atomicwrites              1.3.0                    py37_1    anaconda
attrs                     19.1.0                   py37_1    anaconda
babel                     2.7.0                      py_0    anaconda
backcall                  0.1.0                    py37_0    anaconda
backports                 1.0                        py_2    anaconda
backports-csv             1.0.7                    pypi_0    pypi
backports-functools-lru-cache 1.5                      pypi_0    pypi
backports.functools_lru_cache 1.5                        py_2    anaconda
backports.os              0.1.1                    py37_0    anaconda
backports.shutil_get_terminal_size 1.0.0                    py37_2    anaconda
backports.tempfile        1.0                        py_1    anaconda
backports.weakref         1.0.post1                  py_1    anaconda
beautifulsoup4            4.7.1                    py37_1    anaconda
bitarray                  0.9.3            py37he774522_0    anaconda
bkcharts                  0.2                      py37_0    anaconda
blas                      1.0                         mkl    anaconda
bleach                    3.1.0                    py37_0    anaconda
blosc                     1.16.3               h7bd577a_0    anaconda
bokeh                     1.2.0                    py37_0    anaconda
boto                      2.49.0                   py37_0    anaconda
bottleneck                1.2.1            py37h452e1ab_1    anaconda
bzip2                     1.0.8                he774522_0    anaconda
ca-certificates           2019.5.15                     0    anaconda
certifi                   2019.6.16                py37_0    anaconda
cffi                      1.12.3           py37h7a1dbc1_0    anaconda
chainer                   6.2.0                    pypi_0    pypi
chardet                   3.0.4                    py37_1    anaconda
cheroot                   6.5.5                    pypi_0    pypi
cherrypy                  18.1.2                   pypi_0    pypi
click                     7.0                      py37_0    anaconda
cloudpickle               1.2.1                      py_0    anaconda
clyent                    1.2.2                    py37_1    anaconda
colorama                  0.4.1                    py37_0    anaconda
comtypes                  1.1.7                    py37_0    anaconda
conda                     4.7.11                   py37_0    anaconda
conda-build               3.18.9                   py37_3    anaconda
conda-env                 2.6.0                         1    anaconda
conda-package-handling    1.3.11                   py37_0    anaconda
conda-verify              3.4.2                      py_1    anaconda
console_shortcut          0.1.1                         3    anaconda
constants                 0.6.0                    pypi_0    pypi
contextlib2               0.5.5                    py37_0    anaconda
cpuonly                   1.0                           0    pytorch
cryptography              2.7              py37h7a1dbc1_0    anaconda
cudatoolkit               10.0.130                      0    anaconda
curl                      7.65.2               h2a8f88b_0    anaconda
cycler                    0.10.0                   py37_0    anaconda
cython                    0.29.12          py37ha925a31_0    anaconda
cytoolz                   0.10.0           py37he774522_0    anaconda
dask                      2.1.0                      py_0    anaconda
dask-core                 2.1.0                      py_0    anaconda
decorator                 4.4.0                    py37_1    anaconda
defusedxml                0.6.0                      py_0    anaconda
distributed               2.1.0                      py_0    anaconda
docutils                  0.14                     py37_0    anaconda
entrypoints               0.3                      py37_0    anaconda
et_xmlfile                1.0.1                    py37_0    anaconda
ez-setup                  0.9                      pypi_0    pypi
fastcache                 1.1.0            py37he774522_0    anaconda
fasttext                  0.9.1                    pypi_0    pypi
feedparser                5.2.1                    pypi_0    pypi
ffmpeg                    4.1.3                h6538335_0    conda-forge
filelock                  3.0.12                     py_0    anaconda
first                     2.0.2                    pypi_0    pypi
flask                     1.1.1                      py_0    anaconda
freetype                  2.9.1                ha9979f8_1    anaconda
future                    0.17.1                   py37_0    anaconda
gast                      0.2.2                    py37_0    anaconda
get                       2019.4.13                pypi_0    pypi
get_terminal_size         1.0.0                h38e98db_0    anaconda
gevent                    1.4.0            py37he774522_0    anaconda
glob2                     0.7                        py_0    anaconda
google-pasta              0.1.7                    pypi_0    pypi
graphviz                  2.38.0                        4    anaconda
greenlet                  0.4.15           py37hfa6e2cd_0    anaconda
grpcio                    1.22.0                   pypi_0    pypi
h5py                      2.9.0            py37h5e291fa_0    anaconda
hdf5                      1.10.4               h7ebc959_0    anaconda
heapdict                  1.0.0                    py37_2    anaconda
html5lib                  1.0.1                    py37_0    anaconda
http-client               0.1.22                   pypi_0    pypi
hypothesis                4.34.0                   pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1    anaconda
icu                       58.2                 ha66f8fd_1    anaconda
idna                      2.8                      py37_0    anaconda
imageio                   2.4.1                    pypi_0    pypi
imageio-ffmpeg            0.3.0                    pypi_0    pypi
imagesize                 1.1.0                    py37_0    anaconda
importlib_metadata        0.17                     py37_1    anaconda
imutils                   0.5.2                    pypi_0    pypi
intel-openmp              2019.0                   pypi_0    pypi
ipykernel                 5.1.1            py37h39e3cac_0    anaconda
ipython                   7.6.1            py37h39e3cac_0    anaconda
ipython_genutils          0.2.0                    py37_0    anaconda
ipywidgets                7.5.0                      py_0    anaconda
isort                     4.3.21                   py37_0    anaconda
itsdangerous              1.1.0                    py37_0    anaconda
jaraco-functools          2.0                      pypi_0    pypi
jdcal                     1.4.1                      py_0    anaconda
jedi                      0.13.3                   py37_0    anaconda
jinja2                    2.10.1                   py37_0    anaconda
joblib                    0.13.2                   py37_0    anaconda
jpeg                      9b                   hb83a4c4_2    anaconda
json5                     0.8.4                      py_0    anaconda
jsonschema                3.0.1                    py37_0    anaconda
jupyter                   1.0.0                    py37_7    anaconda
jupyter_client            5.3.1                      py_0    anaconda
jupyter_console           6.0.0                    py37_0    anaconda
jupyter_core              4.5.0                      py_0    anaconda
jupyterlab                1.0.2            py37hf63ae98_0    anaconda
jupyterlab_server         1.0.0                      py_0    anaconda
keras                     2.2.4                         0    anaconda
keras-applications        1.0.8                      py_0    anaconda
keras-base                2.2.4                    py37_0    anaconda
keras-preprocessing       1.1.0                      py_1    anaconda
keyring                   18.0.0                   py37_0    anaconda
kiwisolver                1.1.0            py37ha925a31_0    anaconda
krb5                      1.16.1               hc04afaa_7
lazy-object-proxy         1.4.1            py37he774522_0    anaconda
libarchive                3.3.3                h0643e63_5    anaconda
libcurl                   7.65.2               h2a8f88b_0    anaconda
libiconv                  1.15                 h1df5818_7    anaconda
liblief                   0.9.0                ha925a31_2    anaconda
libmklml                  2019.0.5                      0    anaconda
libpng                    1.6.37               h2a8f88b_0    anaconda
libprotobuf               3.8.0                h7bd577a_0    anaconda
libsodium                 1.0.16               h9d3ae62_0    anaconda
libssh2                   1.8.2                h7a1dbc1_0    anaconda
libtiff                   4.0.10               hb898794_2    anaconda
libxml2                   2.9.9                h464c3ec_0    anaconda
libxslt                   1.1.33               h579f668_0    anaconda
llvmlite                  0.29.0           py37ha925a31_0    anaconda
locket                    0.2.0                    py37_1    anaconda
lxml                      4.3.4            py37h1350720_0    anaconda
lz4-c                     1.8.1.2              h2fa13f4_0    anaconda
lzo                       2.10                 h6df0209_2    anaconda
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
make-dataset              1.0                      pypi_0    pypi
markdown                  3.1.1                    py37_0    anaconda
markupsafe                1.1.1            py37he774522_0    anaconda
matplotlib                3.1.0            py37hc8f65d3_0    anaconda
mccabe                    0.6.1                    py37_1    anaconda
menuinst                  1.4.16           py37he774522_0    anaconda
mistune                   0.8.4            py37he774522_0    anaconda
mkl                       2019.0                   pypi_0    pypi
mkl-service               2.0.2            py37he774522_0    anaconda
mkl_fft                   1.0.12           py37h14836fe_0    anaconda
mkl_random                1.0.2            py37h343c172_0    anaconda
mock                      3.0.5                    py37_0    anaconda
more-itertools            7.0.0                    py37_0    anaconda
moviepy                   1.0.0                    pypi_0    pypi
mpmath                    1.1.0                    py37_0    anaconda
msgpack-python            0.6.1            py37h74a9793_1    anaconda
msys2-conda-epoch         20160418                      1
multipledispatch          0.6.0                    py37_0    anaconda
mysqlclient               1.4.2.post1              pypi_0    pypi
navigator-updater         0.2.1                    py37_0    anaconda
nbconvert                 5.5.0                      py_0    anaconda
nbformat                  4.4.0                    py37_0    anaconda
networkx                  2.3                        py_0    anaconda
ninja                     1.9.0            py37h74a9793_0    anaconda
nltk                      3.4.4                    py37_0    anaconda
nose                      1.3.7                    py37_2    anaconda
notebook                  6.0.0                    py37_0    anaconda
numba                     0.44.1           py37hf9181ef_0    anaconda
numexpr                   2.6.9            py37hdce8814_0    anaconda
numpy                     1.16.4                   pypi_0    pypi
numpy-base                1.16.4           py37hc3f5095_0    anaconda
numpydoc                  0.9.1                      py_0    anaconda
olefile                   0.46                     py37_0    anaconda
opencv-contrib-python     4.1.0.25                 pypi_0    pypi
opencv-python             4.1.0.25                 pypi_0    pypi
openpyxl                  2.6.2                      py_0    anaconda
openssl                   1.1.1c               he774522_1    anaconda
packaging                 19.0                     py37_0    anaconda
pandas                    0.24.2           py37ha925a31_0    anaconda
pandoc                    2.2.3.2                       0    anaconda
pandocfilters             1.4.2                    py37_1    anaconda
parso                     0.5.0                      py_0    anaconda
partd                     1.0.0                      py_0    anaconda
path.py                   12.0.1                     py_0    anaconda
pathlib2                  2.3.4                    py37_0    anaconda
patsy                     0.5.1                    py37_0    anaconda
pattern                   3.6                      pypi_0    pypi
pdfminer-six              20181108                 pypi_0    pypi
pep8                      1.7.1                    py37_0    anaconda
pickleshare               0.7.5                    py37_0    anaconda
pillow                    6.1.0            py37hdc69c19_0    anaconda
pip                       19.1.1                   py37_0    anaconda
pkginfo                   1.5.0.1                  py37_0    anaconda
pluggy                    0.12.0                     py_0    anaconda
ply                       3.11                     py37_0    anaconda
portend                   2.5                      pypi_0    pypi
post                      2019.4.13                pypi_0    pypi
powershell_shortcut       0.0.1                         2    anaconda
proglog                   0.1.9                    pypi_0    pypi
prometheus_client         0.7.1                      py_0    anaconda
prompt_toolkit            2.0.9                    py37_0    anaconda
protobuf                  3.7.1                    pypi_0    pypi
psutil                    5.6.3            py37he774522_0    anaconda
public                    2019.4.13                pypi_0    pypi
py                        1.8.0                    py37_0    anaconda
py-lief                   0.9.0            py37ha925a31_2    anaconda
pybind11                  2.3.0                    pypi_0    pypi
pycodestyle               2.5.0                    py37_0    anaconda
pycosat                   0.6.3            py37hfa6e2cd_0    anaconda
pycparser                 2.19                     py37_0    anaconda
pycrypto                  2.6.1            py37hfa6e2cd_9    anaconda
pycryptodome              3.8.2                    pypi_0    pypi
pycurl                    7.43.0.3         py37h7a1dbc1_0    anaconda
pydot                     1.4.1                    pypi_0    pypi
pyflakes                  2.1.1                    py37_0    anaconda
pygments                  2.4.2                      py_0    anaconda
pylint                    2.3.1                    py37_0    anaconda
pyodbc                    4.0.26           py37ha925a31_0    anaconda
pyopenssl                 19.0.0                   py37_0    anaconda
pyparsing                 2.4.0                      py_0    anaconda
pyqt                      5.9.2            py37h6538335_2    anaconda
pyreadline                2.1                      py37_1    anaconda
pyrsistent                0.14.11          py37he774522_0    anaconda
pysocks                   1.7.0                    py37_0    anaconda
pytables                  3.5.2            py37h1da0976_1    anaconda
pytest                    5.0.1                    py37_0    anaconda
pytest-arraydiff          0.3              py37h39e3cac_0    anaconda
pytest-astropy            0.5.0                    py37_0    anaconda
pytest-doctestplus        0.3.0                    py37_0    anaconda
pytest-openfiles          0.3.2                    py37_0    anaconda
pytest-remotedata         0.3.1                    py37_0    anaconda
python                    3.7.3                h8c8aaf0_1    anaconda
python-dateutil           2.8.0                    py37_0    anaconda
python-docx               0.8.10                   pypi_0    pypi
python-graphviz           0.11.1                   pypi_0    pypi
python-libarchive-c       2.8                     py37_11    anaconda
pytorch                   1.2.0               py3.7_cpu_1  [cpuonly]  pytorch
pytube                    9.5.1                    pypi_0    pypi
pytz                      2019.1                     py_0    anaconda
pywavelets                1.0.3            py37h8c2d366_1    anaconda
pywin32                   223              py37hfa6e2cd_1    anaconda
pywinpty                  0.5.5                 py37_1000    anaconda
pyyaml                    5.1.1            py37he774522_0    anaconda
pyzmq                     18.0.0           py37ha925a31_0    anaconda
qt                        5.9.7            vc14h73c81de_0  [vc14]  anaconda
qtawesome                 0.5.7                    py37_1    anaconda
qtconsole                 4.5.1                      py_0    anaconda
qtpy                      1.8.0                      py_0    anaconda
query-string              2019.4.13                pypi_0    pypi
request                   2019.4.13                pypi_0    pypi
requests                  2.22.0                   py37_0    anaconda
rope                      0.14.0                     py_0    anaconda
ruamel_yaml               0.15.46          py37hfa6e2cd_0    anaconda
scikit-image              0.15.0           py37ha925a31_0    anaconda
scikit-learn              0.21.2           py37h6288b17_0    anaconda
scipy                     1.3.0                    pypi_0    pypi
scipy-stack               0.0.5                    pypi_0    pypi
seaborn                   0.9.0                    py37_0    anaconda
send2trash                1.5.0                    py37_0    anaconda
setuptools                41.1.0                   pypi_0    pypi
simplegeneric             0.8.1                    py37_2    anaconda
singledispatch            3.4.0.3                  py37_0    anaconda
sip                       4.19.8           py37h6538335_0    anaconda
six                       1.12.0                   py37_0    anaconda
snappy                    1.1.7                h777316e_3    anaconda
snowballstemmer           1.9.0                      py_0    anaconda
sortedcollections         1.1.2                    py37_0    anaconda
sortedcontainers          2.1.0                    py37_0    anaconda
soupsieve                 1.8                      py37_0    anaconda
sphinx                    2.1.2                      py_0    anaconda
sphinxcontrib             1.0                      py37_1    anaconda
sphinxcontrib-applehelp   1.0.1                      py_0    anaconda
sphinxcontrib-devhelp     1.0.1                      py_0    anaconda
sphinxcontrib-htmlhelp    1.0.2                      py_0    anaconda
sphinxcontrib-jsmath      1.0.1                      py_0    anaconda
sphinxcontrib-qthelp      1.0.2                      py_0    anaconda
sphinxcontrib-serializinghtml 1.1.3                      py_0    anaconda
sphinxcontrib-websupport  1.1.2                      py_0    anaconda
spyder                    3.3.6                    py37_0    anaconda
spyder-kernels            0.5.1                    py37_0    anaconda
sqlalchemy                1.3.5            py37he774522_0    anaconda
sqlite                    3.29.0               he774522_0    anaconda
statsmodels               0.10.0           py37h8c2d366_0    anaconda
summa                     1.2.0                    pypi_0    pypi
sympy                     1.4                      py37_0    anaconda
tbb                       2019.4               h74a9793_0    anaconda
tblib                     1.4.0                      py_0    anaconda
tempora                   1.14.1                   pypi_0    pypi
tensorboard               1.14.0           py37he3c9ec2_0    anaconda
tensorboardx              1.8                      pypi_0    pypi
tensorflow                1.14.0          mkl_py37h7908ca0_0    anaconda
tensorflow-base           1.14.0          mkl_py37ha978198_0    anaconda
tensorflow-estimator      1.14.0                     py_0    anaconda
tensorflow-mkl            1.14.0               h4fcabd2_0    anaconda
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.8.2                    py37_0    anaconda
testpath                  0.4.2                    py37_0    anaconda
tk                        8.6.8                hfa6e2cd_0    anaconda
toolz                     0.10.0                     py_0    anaconda
torchvision               0.4.0                  py37_cpu  [cpuonly]  pytorch
tornado                   6.0.3            py37he774522_0    anaconda
tqdm                      4.32.1                     py_0    anaconda
traitlets                 4.3.2                    py37_0    anaconda
typing                    3.6.6                    pypi_0    pypi
typing-extensions         3.6.6                    pypi_0    pypi
unicodecsv                0.14.1                   py37_0    anaconda
urllib3                   1.24.2                   py37_0    anaconda
validators                0.13.0                   pypi_0    pypi
vc                        14.1                 h0510ff6_4    anaconda
vs2015_runtime            14.15.26706          h3a45250_4    anaconda
wcwidth                   0.1.7                    py37_0    anaconda
webencodings              0.5.1                    py37_1    anaconda
werkzeug                  0.15.4                     py_0    anaconda
wheel                     0.33.4                   py37_0    anaconda
widgetsnbextension        3.5.0                    py37_0    anaconda
win_inet_pton             1.1.0                    py37_0    anaconda
win_unicode_console       0.5                      py37_0    anaconda
wincertstore              0.2                      py37_0    anaconda
winpty                    0.4.3                         4    anaconda
wrapt                     1.11.2           py37he774522_0    anaconda
xlrd                      1.2.0                    py37_0    anaconda
xlsxwriter                1.1.8                      py_0    anaconda
xlwings                   0.15.8                   py37_0    anaconda
xlwt                      1.3.0                    py37_0    anaconda
xz                        5.2.4                h2fa13f4_4    anaconda
yaml                      0.1.7                hc54c509_2    anaconda
youtube-dl                2019.8.2                 pypi_0    pypi
zc-lockfile               1.4                      pypi_0    pypi
zeromq                    4.3.1                h33f27b4_3    anaconda
zict                      1.0.0                      py_0    anaconda
zipp                      0.5.1                      py_0    anaconda
zlib                      1.2.11               h62dcd97_3    anaconda
zstd                      1.3.7                h508b16e_0    anaconda

Output:
you dont have to install it via anaconda, you could install cuda from their website. after install ends open a new terminal and check your cuda version with:
>>> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:52:33_Pacific_Standard_Time_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

my is V11.5
after, go here and select your os and preferred package manager(pip or anaconda), and the cuda version you installed, and copy the generated install command, I got:
pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio===0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

notice that for me I had python 3.10 installed but my project run over 3.9 so either use virtual environment or run pip of your wanted base interpreter explicitly (for example C:\Software\Python\Python39\python.exe -m pip install .....)
else you will be stuck with Could not find a version that satisfies the requirement torch errors
after, open python console and check for cuda availability
>>> import torch
>>> torch.cuda.is_available()
True


Instruction: pytorch collate_fn reject sample and yield another

Input:
I have built a Dataset, where I'm doing various checks on the images I'm loading. I'm then passing this DataSet to a DataLoader.

In my DataSet class I'm returning the sample as None if a picture fails my checks and i have a custom collate_fn function which removes all Nones from the retrieved batch and returns the remaining valid samples.

However at this point the returned batch can be of varying size. Is there a way to tell the collate_fn to keep sourcing data until the batch size meets a certain length?

class DataSet():
     def __init__(self, example):
          # initialise dataset
          # load csv file and image directory
          self.example = example
     def __getitem__(self,idx):
          # load one sample
          # if image is too dark return None
          # else
          # return one image and its equivalent label

dataset = Dataset(csv_file='../', image_dir='../../')

dataloader = DataLoader(dataset , batch_size=4,
                        shuffle=True, num_workers=1, collate_fn = my_collate )

def my_collate(batch): # batch size 4 [{tensor image, tensor label},{},{},{}] could return something like G = [None, {},{},{}]
    batch = list(filter (lambda x:x is not None, batch)) # this gets rid of nones in batch. For example above it would result to G = [{},{},{}]
    # I want len(G) = 4
    # so how to sample another dataset entry?
    return torch.utils.data.dataloader.default_collate(batch)

Output:
There are 2 hacks that can be used to sort out the problem, choose one way:

By using the original batch sample Fast option:

def my_collate(batch):
    len_batch = len(batch) # original batch length
    batch = list(filter (lambda x:x is not None, batch)) # filter out all the Nones
    if len_batch > len(batch): # if there are samples missing just use existing members, doesn't work if you reject every sample in a batch
        diff = len_batch - len(batch)
        for i in range(diff):
            batch = batch + batch[:diff]
    return torch.utils.data.dataloader.default_collate(batch)

Otherwise just load another sample from dataset at random Better option:

def my_collate(batch):
    len_batch = len(batch) # original batch length
    batch = list(filter (lambda x:x is not None, batch)) # filter out all the Nones
    if len_batch > len(batch): # source all the required samples from the original dataset at random
        diff = len_batch - len(batch)
        for i in range(diff):
            batch.append(dataset[np.random.randint(0, len(dataset))])

    return torch.utils.data.dataloader.default_collate(batch)


Instruction: how to install pytorch in python2.7?

Input:
i am using python2.7 in virtual environment. i tried to install pytorch in python2.7 but i got error belows:

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

  - pytorch-cpu -> python[version='3.5.*|3.6.*']
  - pytorch-cpu -> python[version='>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to.  Your current python version
is (python=2.7).  Note that conda will not change your python version to a different minor version
unless you explicitly specify that.

The following specifications were found to be incompatible with each other:

Package wheel conflicts for:
python=2.7 -> pip -> wheel
pytorch-cpu -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel
Package vc conflicts for:
python=2.7 -> sqlite[version='>=3.27.2,<4.0a0'] -> vc[version='14.*|>=14,<15.0a0|>=14.1,<15.0a0']
python=2.7 -> vc[version='9.*|>=9,<10.0a0']
pytorch-cpu -> numpy[version='>=1.11'] -> vc[version='14|14.*|>=14,<15.0a0']
pytorch-cpu -> vc[version='>=14.1,<15.0a0']
Package cffi conflicts for:
pytorch-cpu -> cffi
pytorch-cpu -> python[version='>=3.6,<3.7.0a0'] -> pip -> requests -> urllib3[version='>=1.21.1,<1.25'] -> cryptography[version='>=1.3.4'] -> cffi[version='>=1.7']
python=2.7 -> pip -> requests -> urllib3[version='>=1.21.1,<1.25'] -> cryptography[version='>=1.3.4'] -> cffi[version='>=1.7']
Package pip conflicts for:
python=2.7 -> pip
pytorch-cpu -> python[version='>=3.6,<3.7.0a0'] -> pip
Package setuptools conflicts for:
python=2.7 -> pip -> setuptools
pytorch-cpu -> python[version='>=3.6,<3.7.0a0'] -> pip -> setuptools
Package msgpack-python conflicts for:
python=2.7 -> pip -> cachecontrol -> msgpack-python
pytorch-cpu -> python[version='>=3.6,<3.7.0a0'] -> pip -> cachecontrol -> msgpack-python

i tried conda install pytorch-cpu -c pytorch and link(https://pytorch.org/get-started/locally/). but it is not worked. so what should i do for install torch in python version2.7? i want to install pytorch cpu version.

plz help:)

Output:
Here's the link to the PyTorch official download page

From here, you can choose the python version (2.7) and CUDA (None) and other relevant details based on your environment and OS.

Other helpful links:

windows
windows
mac
ubuntu
all


Instruction: TypeError: 'int' object is not callable in loss.backward()

Input:
While trying to set up a pytorch model, I am getting the error that the loss object is not callable when trying to do Pytorch autograd. (Relevant code shown below)

optimizer = torch.optim.Adam(model.parameters(), lr=lr,
  betas(0.0,0.9))

def train(epoch, shuffle, wisdom_model, optim, loss):
    print('train')
    accuracy = 0
    batch_num = 0
    wisdom_model.train()
    for batch in data.train_dl:

        optim.zero_grad()

        result = model(batch[0])
        loss = nn.CrossEntropyLoss()(result, batch[1].long())

        loss.backward()

        accuracy += accuracy(result, batch[1])
        print(accuracy)
        pdb.set_trace()
        batch_num += 1

    return accuracy / batch_num

TypeError                                 Traceback (most recent call last)
<ipython-input-28-5b9c9fe3b320> in <module>
----> 1 run(1, False)

<ipython-input-27-d0d67dbf6eb2> in run(num_models, dropout)
     71
     72     for epoch in range(10):
---> 73         train_accuracy = train(epoch, False, model, optimizer, loss)
     74         accuracy.append(validate(epoch, model))
     75

<ipython-input-27-d0d67dbf6eb2> in train(epoch, shuffle, model, optim, loss)
     24         pdb.set_trace()
     25
---> 26         loss.backward()
     27         optim.step()
     28

TypeError: 'int' object is not callable

Output:
The problem is in this line:

loss = nn.CrossEntropyLoss()(result, batch[1].long())

Check out the nn.CrossEntropyLoss.

Should not look like this:

nn.CrossEntropyLoss()()

Should look like this:

nn.CrossEntropyLoss()


Instruction: Cleaner way to use "with torch.no_grad()" conditioned on an expression

Input:
My code looks like:

if no_grad_condition:
  with torch.no_grad():
    out=network(input)
else:
  out=network(input)

Is there a cleaner way to do it, without duplicating the line out=network(input)?

I am looking for something in the spirit of:

  with torch.no_grad(no_grad_condition):
    out=network(input)

Output:
OP here: By writing down the question, I understood where to look for the answer. According to pytorch docs, we can use set_grad_enabled:

  with torch.set_grad_enabled(not no_grad_condition):
    out=network(input)


Instruction: I have a trouble after trying to parallelize data using nn.Dataparallel

Input:
I didn't have any probelm without data parallelization, but after I put JUST A ONE LINE "model = nn.DataParallel(model)" the error message "TypeError: 'list' object is not callable" comes. If I push out that damn line the source works clean. plz help me

I can't do anything but just mad. I search that google and there are some ways to solve that error message but I can't do anything. Because nn.Dataparallel is already used by other coders. Sorry English is not my mothertongue.

    if use_cuda:
            [[[model = nn.DataParallel(model)]]]
            model = model.cuda()
            criterion = criterion.cuda()
            print('cuda is used')

I just put model = nn.DataParallel(model) and error comes right after.

  Traceback (most recent call last):
    File "/home/scrcdeep2/YBJ/espnet_myself/Main.py", line 119, in
      train(loader, model_, criterion_, optimizer_, use_cuda_, pretrained=None)

  File "/home/scrcdeep2/YBJ/espnet_myself/Main.py", line 83, in train
      outputs = model(inputs)

  File "/home/scrcdeep2/YBJ/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call
      result = self.forward(*input, **kwargs)
   File "/home/scrcdeep2/YBJ/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 151, in forward
      replicas = self.replicate(self.module, self.device_ids[:len(inputs)])

  File "/home/scrcdeep2/YBJ/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 156, in replicate
      return replicate(module, device_ids)

  File "/home/scrcdeep2/YBJ/lib/python3.5/site-packages/torch/nn/parallel/replicate.py", line 114, in replicate
      modules = list(network.modules())

  TypeError: 'list' object is not callable

Output:
I met the same problem, because I've set moules = [list].
I changed my code like this:
def __init__(self, embedding_size, activation_function='relu'):
    super().__init__()
    self.act_fn = getattr(F, activation_function)
    self.embedding_size = embedding_size
    self.conv1 = nn.Conv2d(3, 32, 4, stride=2)
    self.conv2 = nn.Conv2d(32, 64, 4, stride=2)
    self.conv3 = nn.Conv2d(64, 128, 4, stride=2)
    self.conv4 = nn.Conv2d(128, 256, 4, stride=2)
    self.fc = nn.Identity() if embedding_size == 1024 else nn.Linear(1024, embedding_size)
    if not args.MultiGPU:
      self.modules = [self.conv1, self.conv2, self.conv3, self.conv4]


Instruction: How to clear GPU memory after PyTorch model training without restarting kernel

Input:
I am training PyTorch deep learning models on a Jupyter-Lab notebook, using CUDA on a Tesla K80 GPU to train. While doing training iterations, the 12 GB of GPU memory are used. I finish training by saving the model checkpoint, but want to continue using the notebook for further analysis (analyze intermediate results, etc.).

However, these 12 GB continue being occupied (as seen from nvtop) after finishing training. I would like to free up this memory so that I can use it for other notebooks.

My solution so far is to restart this notebook's kernel, but that is not solving my issue because I can't continue using the same notebook and its respective output computed so far.

Output:
The answers so far are correct for the Cuda side of things, but there's also an issue on the ipython side of things.

When you have an error in a notebook environment, the ipython shell stores the traceback of the exception so you can access the error state with %debug. The issue is that this requires holding all variables that caused the error to be held in memory, and they aren't reclaimed by methods like gc.collect(). Basically all your variables get stuck and the memory is leaked.

Usually, causing a new exception will free up the state of the old exception. So trying something like 1/0 may help. However things can get weird with Cuda variables and sometimes there's no way to clear your GPU memory without restarting the kernel.

For more detail see these references:

https://github.com/ipython/ipython/pull/11572

How to save traceback / sys.exc_info() values in a variable?


Instruction: PyTorch equivalent of `numpy.unpackbits`?

Input:
I am training a neural net on GPU. It uses a lot of binary input features.

Since moving data to/from GPU is expensive, I am looking for ways to make the initial representation more compact. Now, I encode my features as int8, move them over to GPU and then expand as float32:

# create int8
features = torch.zeros(*dims, dtype=torch.int8)

# fill in some data (set some features to 1.)
…

# move int8 to GPU
features = features.to(device=cuda, non_blocking=True)

# expand int8 as float32
features = features.to(dtype=float32)

Now, I am looking for ways to compress those binary features to bits instead of bytes.

NumPy has functions packbits and unpackbits

>>> a = np.array([[2], [7], [23]], dtype=np.uint8)
>>> b = np.unpackbits(a, axis=1)
>>> b
array([[0, 0, 0, 0, 0, 0, 1, 0],
       [0, 0, 0, 0, 0, 1, 1, 1],
       [0, 0, 0, 1, 0, 1, 1, 1]], dtype=uint8)

Is there any way to unpack bits in PyTorch on GPU?

Output:
There is no similar functions at the time of writing this answer. However, a workaround is using torch.from_numpy as in:

In[2]: import numpy as np
In[3]: a = np.array([[2], [7], [23]], dtype=np.uint8)
In[4]: b = np.unpackbits(a, axis=1)
In[5]: b
Out[5]:
array([[0, 0, 0, 0, 0, 0, 1, 0],
       [0, 0, 0, 0, 0, 1, 1, 1],
       [0, 0, 0, 1, 0, 1, 1, 1]], dtype=uint8)
In[6]: import torch
In[7]: torch.from_numpy(b)
Out[7]:
tensor([[0, 0, 0, 0, 0, 0, 1, 0],
        [0, 0, 0, 0, 0, 1, 1, 1],
        [0, 0, 0, 1, 0, 1, 1, 1]], dtype=torch.uint8)


Instruction: Unable to Install Torch or torch vision in pycharm I am running python 3.6

Input:
Installing torch from PyCharm interpreter but error occurs. Python 3.6

Collecting torch==0.4.1.post2

  Could not find a version that satisfies the requirement torch==0.4.1.post2 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
No matching distribution found for torch==0.4.1.post2

Output:
Try installing lower version of torch, go to:

  File->Setting->Project[project_name]-> Project Interpreter -> + ->
  search for torch, in right lower corner is check box 'Specify version'

check it and try sevral starting from top. From what i see current is 1.2.0


Instruction: ImportError: cannot import name 'parameter_parser' from 'parser' (unknown location)

Input:
I am trying to Import Parameter_parser from Parser. but it is showing the error below:

ImportError: cannot import name 'parameter_parser' from 'parser'

In the line below that I also get:

ModuleNotFoundError: No module named 'load_data'

This is my code:

    import matplotlib
    matplotlib.use('agg')
    import numpy as np
    import time
    import os
    import torch.utils.data
    import torch.nn.functional as F
    import torch.optim as optim
    import torch.optim.lr_scheduler as lr_scheduler
    from torch.utils.data import DataLoader
    from os.path import join as pjoin
    from parser import parameter_parser
    from load_data import split_ids, GraphData, collate_batch
    from models.gcn_modify import GCN_MODIFY
    from models.gcn_origin import GCN_ORIGIN
    from models.gat import GAT
    from models.mgcn import MGCN
    from sklearn import metrics`

Output:
When I try the same thing in my python console I get this:

>>>  from parser import parameter_parser

File "<stdin>", line 1
    from parser import parameter_parser
    ^
IndentationError: unexpected indent
>>> from parser import parameter_parser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name parameter_parser

Is this the same problem for you? This is because you don't have the module installed via pip (pip install PACKAGE_NAME) or whatever you use to install your packages. Another idea is that you have set up a virtual env, installed it there and did not activate it.

At any case although I didn't downvote your answer (I think there are no wrong questions!) I assume the person that did was not able to find the additional information to help you solve your problem. for next time try adding what OS you are using, what package is causing the problem and what solutions have you already tried (did you find other answers on stackoverflow? did you google the problem? did you try importing the package by itself in the console?).


Instruction: Interpret GAN loss

Input: I am currently training the standard DCGAN network on my dataset. After 40 epochs, the loss of both generator and discriminator is 45-50. Can someone please explain the reason and possible solution for this?

Output:
This interpretation may be added to unsolved problems.

You cannot interpret the loss of generator and discriminator. Since when one improves it will be harder for the other. When generator improves it will be harder for the critic. When critic improves it will be harder for the generator.

The values totally depend on your loss function. You may expect that numbers should be "about the same" over time.


Instruction: DataLoader messing up transformed data

Input:
I am testing the MNIST dataset in Pytorch, and after I apply a transformation to the X data, it seems the DataLoader puts all values out of the original order, potentially messing up the training step.

My transformation is to divide all values by 255. One should notice that the transformation itself does not change positions, as shown by the first scatterplots. But after the data is passed to the DataLoader and I retrieve it back, they are out of order. If I make no transformation, everything is fine (not shown). The distribution of the values is the same among before, after1 (divided by 255/before DataLoader) and after2 (divided by 255/after DataLoader) (also not shown), only the order seems to be affected.

import torch
from torchvision import datasets
import torchvision.transforms as transforms
import matplotlib.pyplot as plt

transform = transforms.ToTensor()

train = datasets.MNIST(root = '.', train = True, download = True, transform = transform)
test = datasets.MNIST(root = '.', train = False, download = True, transform = transform)

before = train.data[0]

train.data = train.data.float()/255
after1 = train.data[0]

train_loader = torch.utils.data.DataLoader(train, batch_size = 128)
test_loader = torch.utils.data.DataLoader(test, batch_size = 128)

fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after1.view(-1))), after1.view(-1))
ax[1].set_title('After1')

after2 = next(iter(train_loader))[0][0]

fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after2.view(-1))), after2.view(-1))
ax[1].set_title('After2')

fig, ax = plt.subplots(1, 3)
ax[0].imshow(before, cmap = 'gray')
ax[1].imshow(after1, cmap = 'gray')
ax[2].imshow(after2.view(28, 28), cmap = 'gray')

I know that this might not be the best way to deal with this data (transforms.Normalize should solve it), but I would really like to understand what is happening.

Thank you!

Output:
So... I posted this same question at Pytorch's GitHub page, and they answered the following:

  It's unrelated to data loader. You are messing with an attribute of
  the particular dataset object, however, the actual __getitem__ of that
  object does much more:
  https://github.com/pytorch/vision/blob/6de158c473b83cf43344a0651d7c01128c7850e6/torchvision/datasets/mnist.py#L92

  In particular this line (mode='L') assumes uint8 input. Since you
  replaced it with float, it is wrong.

Then I guess the preferred approach would be to apply a transform when preparing the dataset at the very beginning of my code.


Instruction: PyTorch normalize two sets of gradients during training

Input:
In this GAN tutorial, if you scroll down to the training loop you can see they combine the gradients
 errD = errD_real + errD_fake like this. Where errD_real = criterion(output, label) and errD_fake = criterion(output, label) and criterion = nn.BCELoss(). I want to do the same thing but before doing a backward pass I want to normalize both gradients to the lower Euclidean norm of the two. How would I do that?

I know I can access the gradients of each weight individually on netD by printing out netD.weight.grad, but is there some way to batchnorm them to the lower Euclidean norm of the two?

Here's the part of the training loop I'm talking about:

for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
        ...

Output:
You mentioned two things which are incorrect.

errD = errD_real + errD_fake basically combines two objective function, not gradients.
You need to change the gradients after the backward pass, not before. With the backward pass, the gradients will be computed and then you can modify the gradients before calling optimizer.step() which updates the model parameters.

Answer to your question:

I believe there is no direct API support from PyTorch to achieve your goal. However, you can easily modify the gradients by yourself after the backward pass. To compute the Euclidean norm, you can use the norm function. Then you can simply divide the gradients (inplace).


Instruction: How Pytorch Tensor get the index of elements?

Input:
I have 2 Tensors named x and list and their definitions are below:

x = torch.tensor(3)
list = torch.tensor([1,2,3,4,5])

Now I want to get the index of element x from list. The expected output is an Integer:

2

How can I do in an easy way?

Output:
import torch

x = torch.tensor(3)

list = torch.tensor([1,2,3,4,5])
idx = (list == x).nonzero().flatten()
print (idx.tolist()) # [2]

list = torch.tensor([1,2,3,3,5])
idx = (list == x).nonzero().flatten()
print (idx.tolist()) # [2, 3]


Instruction: How to save GPU memory usage in PyTorch

Input:
In PyTorch I wrote a very simple CNN discriminator and trained it. Now I need to deploy it to make predictions. But the target machine has a small GPU memory and got out of memory error. So I think that I can set requires_grad = False to prevent PyTorch from storing the gradient values. However I didn't find it making any difference.

There are about 5 millions of parameters in my model. But when predicting a single batch of input, it consumes about 1.2GB of memory. I think there should be no need for such large memory.

The question is how to save GPU memory usage when I just want to use my model to make predictions?

Here is a demo, I use discriminator.requires_grad_ to disable/enable autograd of all parameters. But it seems to be no use.

import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as functional

from pynvml.smi import nvidia_smi
nvsmi = nvidia_smi.getInstance()

def getMemoryUsage():
    usage = nvsmi.DeviceQuery("memory.used")["gpu"][0]["fb_memory_usage"]
    return "%d %s" % (usage["used"], usage["unit"])

print("Before GPU Memory: %s" % getMemoryUsage())

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        # trainable layers
        # input: 2x256x256
        self.conv1 = nn.Conv2d(2, 8, 5, padding=2) # 8x256x256
        self.pool1 = nn.MaxPool2d(2) # 8x128x128
        self.conv2 = nn.Conv2d(8, 32, 5, padding=2) # 32x128x128
        self.pool2 = nn.MaxPool2d(2) # 32x64x64
        self.conv3 = nn.Conv2d(32, 96, 5, padding=2) # 96x64x64
        self.pool3 = nn.MaxPool2d(4) # 96x16x16
        self.conv4 = nn.Conv2d(96, 256, 5, padding=2) # 256x16x16
        self.pool4 = nn.MaxPool2d(4) # 256x4x4
        self.num_flat_features = 4096
        self.fc1 = nn.Linear(4096, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 1)
        # loss function
        self.loss = nn.MSELoss()
        # other properties
        self.requires_grad = True
    def forward(self, x):
        y = x
        y = self.conv1(y)
        y = self.pool1(y)
        y = functional.relu(y)
        y = self.conv2(y)
        y = self.pool2(y)
        y = functional.relu(y)
        y = self.conv3(y)
        y = self.pool3(y)
        y = functional.relu(y)
        y = self.conv4(y)
        y = self.pool4(y)
        y = functional.relu(y)
        y = y.view((-1,self.num_flat_features))
        y = self.fc1(y)
        y = functional.relu(y)
        y = self.fc2(y)
        y = functional.relu(y)
        y = self.fc3(y)
        y = torch.sigmoid(y)
        return y
    def predict(self, x, score_th=0.5):
        if len(x.shape) == 3:
            singlebatch = True
            x = x.view([1]+list(x.shape))
        else:
            singlebatch = False
        y = self.forward(x)
        label = (y > float(score_th))
        if singlebatch:
            y = y.view(list(y.shape)[1:])
        return label, y
    def requires_grad_(self, requires_grad=True):
        for parameter in self.parameters():
            parameter.requires_grad_(requires_grad)
        self.requires_grad = requires_grad

x = torch.cuda.FloatTensor(np.zeros([2, 256, 256]))
discriminator = Discriminator()
discriminator.to("cuda:0")

# comment/uncomment this line to make difference
discriminator.requires_grad_(False)

discriminator.predict(x)

print("Requires grad", discriminator.requires_grad)
print("After GPU Memory: %s" % getMemoryUsage())

By comment out the line discriminator.requires_grad_(False), I got output:

Before GPU Memory: 6350MiB
Requires grad True
After GPU Memory: 7547MiB

While by uncomment the line, I got:

Before GPU Memory: 6350MiB
Requires grad False
After GPU Memory: 7543MiB

Output:
You can use pynvml.

This python tool made Nvidia so you can Python query like this:

from pynvml.smi import nvidia_smi
nvsmi = nvidia_smi.getInstance()
nvsmi.DeviceQuery('memory.free, memory.total')

You can always also execute:

torch.cuda.empty_cache()

To empty the cache and you will find even more free memory that way.

Before calling torch.cuda.empty_cache() if you have objects you don't use anymore you can call this:

obj = None

And after that you call

gc.collect()


Instruction: plt.imshow() gives TypeError: Image data of dtype object cannot be converted to float using PyTorch

Input:
I am trying to create a custom Dataset Processor for a set of images. However, when I try to view the images in my dataset, I get hit with the TypeError: Image data of dtype object cannot be converted to float.

I tried to check if I am passing in a PIL Image into the plt.imshow() function and I am.

class DatasetProcessing(Dataset):

    def __init__(self, input_data, output_data, transform=None):
        self.transform = transform
        self.input_data =
        input_data.reshape((-1,64,64)).astype(np.float32)[:,:,:,None]
        self.output_data = output_data

    def __getitem__(self, index):
        return self.transform(self.input_data[index]), self.output_data[index]

    def __len__(self):
        return len(list(self.input_data))

transform = transforms.Compose([transforms.ToPILImage()])

dset_train = DatasetProcessing(X_slices_train, Y_train, transform)

train_loader = torch.utils.data.DataLoader(dset_train, batch_size=4,
                                      shuffle=True, num_workers=4)

plt.figure(figsize = (16, 4))
for num, x in enumerate(dset_train):
    plt.subplot(1,6,num+1)
    plt.axis('off')
    print(x)
    plt.imshow(np.asarray(x))
    plt.title(y_train[num])

I expected to get pictures of my dataset, but instead I get the following error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-8b8caac49d97> in <module>
      4     plt.axis('off')
      5     print(x)
----> 6     plt.imshow(np.asarray(x))
      7     plt.title(y_train[num])

~/anaconda3/lib/python3.7/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs)
   2675         filternorm=filternorm, filterrad=filterrad, imlim=imlim,
   2676         resample=resample, url=url, **({"data": data} if data is not
-> 2677         None else {}), **kwargs)
   2678     sci(__ret)
   2679     return __ret

~/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs)
   1587     def inner(ax, *args, data=None, **kwargs):
   1588         if data is None:
-> 1589             return func(ax, *map(sanitize_sequence, args), **kwargs)
   1590
   1591         bound = new_sig.bind(ax, *args, **kwargs)

~/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs)
    367                 f"%(removal)s.  If any parameter follows {name!r}, they "
    368                 f"should be pass as keyword, not positionally.")
--> 369         return func(*args, **kwargs)
    370
    371     return wrapper

~/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs)
    367                 f"%(removal)s.  If any parameter follows {name!r}, they "
    368                 f"should be pass as keyword, not positionally.")
--> 369         return func(*args, **kwargs)
    370
    371     return wrapper

~/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)
   5658                               resample=resample, **kwargs)
   5659
-> 5660         im.set_data(X)
   5661         im.set_alpha(alpha)
   5662         if im.get_clip_path() is None:

~/anaconda3/lib/python3.7/site-packages/matplotlib/image.py in set_data(self, A)
    676                 not np.can_cast(self._A.dtype, float, "same_kind")):
    677             raise TypeError("Image data of dtype {} cannot be converted to "
--> 678                             "float".format(self._A.dtype))
    679
    680         if not (self._A.ndim == 2

TypeError: Image data of dtype object cannot be converted to float

Output:
Your dset_train yields self.transform(self.input_data[index]), self.output_data[index] if understood correctly self.transform(self.input_data[index]) is an image tensor (data) and self.output_data[index] is a label, but here:

plt.imshow(np.asarray(x))

you are passing unpacked x which is actually (data, label)

So, you need to unpack it first:

plt.figure(figsize = (16, 4))
for num, x in enumerate(dset_train):
    data, label = x
    plt.subplot(1,6,num+1)
    plt.axis('off')
    print(x)
    plt.imshow(np.asarray(data))
    plt.title(y_train[num])

EDIT:

  Why I have to unpack x?

You're inheriting from PyTorch's Dataset, and according to docs:

  All datasets that represent a map from keys to data samples should subclass it. All subclasses should overrite __getitem__(), supporting fetching a data sample for a given key.

In your defined DatasetProcessing class __getitem__() returns a tuple of 2 items: self.transform(self.input_data[index]) and self.output_data[index], the first one is data, the second one is appropriate label. And that's why you need to unpack it like data, y = x, because your DatasetProcessing dataset yields data and a label.

  Is there any documentation/tutorials you can link me to?

I can recommend you this links:

Data Loading and Processing Tutorial
Dataset docs
torch.utils.data docs


Instruction: How a robust background removal is implemented?

Input:
I found that a deep-learning-based method (e.g., 1) is much more robust than a non-deep-learning-based method (e.g., 2, using OpenCV).

https://www.remove.bg
How do I remove the background from this kind of image?

In the OpenCV example, Canny is used to detect the edges. But this step can be very sensitive to the image. The contour detection may end up with wrong contours. It is also difficult to determine which contours should be kept.
How a robust deep-learning method is implemented? Is any good example code? Thanks.

Output:
For that to work you need to use Unet. You can search for that on github.
Unet transofrm is: I->I.
Space of the image will become image (of same or similar size).

You need to have say 10.000 images with bg removed. People, (long hair people), cats, cars, shoes, T-shirts, etc.

So you set different backgrounds on all these images as source and prediction should be images with removed background.

You can also do a segmentation model and when you find the foreground you can remove the bg.


Instruction: How can I disable gradient updates for some modules in autograd backpropagation?

Input:
I'm building a multi-model neural network for reinforcement learning to include an action network, a world model network, and a critic. The idea is train the world model to emulate whatever simulation you are trying to master based on input from the action network and the previous state, to train the critic to maximize the Bellman equation (total reinforcement over time) based on the world model output, and then backpropagate the critic value through the world model to provide gradient targets for training the actions. So - from some state, the action network outputs an action which is fed into the model to generate the next state, and that state feeds into the critic network for evaluation against some goal state.

For all this to work, I must use 3 separate loss functions, one for each network, and they all add something to the gradients in one or more networks but they can be in conflict. For example - to train the world model I use a target from an environmental simulation and for the critic I use a target of the current state reward + discount * next state forecast value. However, to train the a actor I just use the negative critic value as a loss and backpropagate all the way through all three models to calibrate the best action.

I can make this work without any batching by zeroing out gradients incrementally, but that is inefficient and doesn't let me accumulate gradients for any kind of "time-series batching" optimizer update step. Each model has it's own trainable parameters, but the execution graph flows through all three networks. So inside the calibration loop after firing the networks in sequence:
...

        if self.actor.calibrating:
            self.actor.optimizer.zero_grad()
            #Pick loss For maximizing the value of all actions
            loss = -self.critic.value
            #Backpropagate through all three networks to train actor output
            #How do I stop the critic and model networks from incrementing their gradient values?
            loss.backward(retain_graph=True)
            self.actor.optimizer.step()
        if self.model.calibrating:
            self.model.optimizer.zero_grad()
            #Reduce loss for ambiguous actions
            loss = self.model.get_loss() * self.actor.get_confidence()**2
            #How can I block this from backpropagating through action network?
            loss.backward(retain_graph=True)
            self.model.optimizer.step()
        if self.critic.calibrating:
            self.critic.optimizer.zero_grad()
            #Reduce loss for ambiguous actions
            loss = self.critic.get_loss(self.goal) * self.actor.get_confidence()**2
            #How do I stop this from backpropagating through the model and action networks?
            loss.backward(retain_graph=True)
            self.critic.optimizer.step()

...

Finally - my question is in two parts:

How can I temporarily stop loss.backward() at a given layer without detaching it forever?
How can I block loss.backward() from updating some gradients where I'm just flowing through a model to get gradients for another model?

Output:
Got this figured out thanks to a suggestion from a colleague to try the requires_grad setting. (I had assumed that would break the execution graph, but it doesn't)

So - to answer my own two questions:

If you calibrate the chained models in the correct order, you can detach them one at a time so that loss.backward() doesn't run over models that aren't needed. I was thinking that this would break the graph but... this is Pytorch, not Tensorflow 1.x and the graph is regenerated on every forward pass anyway. Silly me for missing this yesterday.
If you set requires_grad to False for a model (or a layer or an individual weight) then loss.backward() will STILL traverse the entire connected graph but it will leave those individual gradients as they were while still setting any gradients earlier in the graph. Exactly what I wanted.

This code works to minimize the execution of unnecessary graph traversals and gradient updates. I still need to refactor it for staggered updates over time so that it can accumulate gradients for several cycles before stepping the optimizers, but this definitely works as intended.

#Step through all models in a chain to create gradient paths from critic back through the world model, to the actor.
    def step(self):
        #Get the current state from the simulation
        state = self.world.state
        #Fire the actor to select a softmax action.
        self.actor(state)
        #run the world simulation on that action.
        self.world.step(self.actor.action)
        #Combine the action and starting state as input to the world model.
        if self.actor.calibrating:
            action_state = torch.cat([self.actor.value, state], dim=0)
        else:
            #Push softmax action closer to 1.0
            action_state = torch.cat([self.actor.hard_value, state], dim=0)
        #Run the model and then the critic on the action_state
        self.critic(self.model(action_state))
        if self.actor.calibrating:
            self.actor.optimizer.zero_grad()
            self.model.requires_grad = False
            self.critic.requires_grad = False
            #Pick loss For maximizing the value of the action choice
            loss = -self.critic.value * self.actor.get_confidence()
            loss.backward(retain_graph=True)
            self.actor.optimizer.step()
        if self.model.calibrating:
            #Don't need to backpropagate through actor again
            self.actor.value.detach_()
            self.model.optimizer.zero_grad()
            self.model.requires_grad = True
            #Reduce loss for ambiguous actions
            loss = self.model.get_loss() * self.actor.get_confidence()**2
            loss.backward(retain_graph=True)
            self.model.optimizer.step()
        if self.critic.calibrating:
            #Don't need to backpropagate through the model or actor again
            self.model.value.detach_()
            self.critic.optimizer.zero_grad()
            self.critic.requires_grad = True
            #Reduce loss for ambiguous actions
            loss = self.critic.get_loss(self.goal) * self.actor.get_confidence()**2
            loss.backward(retain_graph=True)
            self.critic.optimizer.step()


Instruction: any script to test the installation of Pytorch

Input: I have installed the pytorch, and would like to check are there any script to test whether the installation is correct, e.g., whether it can enable CUDA or not, etc?

Output:
Coming to your 1st question,
In your python script....
just add
import torch

if this gives "ModuleNotFoundError: No module named 'torch'",
then your pytorch installation is not complete
And your 2nd question to check if your pytorch is using cuda,use this
torch.cuda.is_available()

this will return True if your pytorch is using cuda


Instruction: Pytorch tensor multiplication with Float tensor giving wrong answer

Input:
I am seeing some strange behavior when i multiply two pytorch tensors.

x = torch.tensor([99397544.0])
y = torch.tensor([0.1])
x * y

This outputs

tensor([9939755.])

However, the answer should be 9939754.4

Output:
In default, the tensor dtype is torch.float32 in pytorch. Change it to torch.float64 will give the right result.

x = torch.tensor([99397544.0], dtype=torch.float64)
y = torch.tensor([0.1], dtype=torch.float64)
x * y
# tensor([9939754.4000])

The mismatched result for torch.float32 caused by rounding error if you do not have enough precision to calculate (represent) it.

What Every Computer Scientist Should Know About Floating-Point Arithmetic


Instruction: embedding layer outputs nan

Input:
I am trying to learn a seq2seq model.
An embedding layer is located in the encoder and it sometimes outputs nan value after some iterations.
I cannot identify the reason.
How can I solve this??
The problem is the first emb_layer in the forward function in the code below.

class TransformerEncoder(nn.Module):
    def __init__(self, vocab_size, hidden_size=1024, num_layers=6, dropout=0.2, input_pad=1, batch_first=False, embedder=None, init_weight=0.1):
        super(TransformerEncoder, self).__init__()
        self.input_pad = input_pad
        self.vocab_size = vocab_size
        self.num_layers = num_layers
        self.embedder = embedder

        if embedder is not None:
            self.emb_layer = embedder
        else:
            self.emb_layer = nn.Embedding(vocab_size, hidden_size, padding_idx=1)

        self.positional_encoder = PositionalEncoder()
        self.transformer_layers = nn.ModuleList()
        for _ in range(num_layers):
            self.transformer_layers.append(
                    TransformerEncoderBlock(num_heads=8, embedding_dim=1024, dropout=dropout))

    def set_mask(self, inputs):
        self.input_mask = (inputs == self.input_pad).unsqueeze(1)

    def forward(self, inputs):
        x = self.emb_layer(inputs)
        x = self.positional_encoder(x)

Output:
It is usually the inputs more than the weights which tend to become nan (either goes too high or too low). Maybe these are incorrect to start out with and worsen after some gradients. You can identify these inputs by running the tensor or np.array thru' a simple condition check like:
print("Inp value too high") if len(bert_embeddings[bert_embeddings>1000]) > 1 else None

A common mistake for a beginner is to use a torch.empty instead of torch.zeros. This invariably leads to Nan over time.
If all your inputs are good, then it is the vanishing or exploding gradients issue. See if the problem worsens after a few iterations. Explore different activations or clipping gradients which usually fix these type of issues. If you are using latest optimizers you usually need not worry about adjusting the learning rate.


Instruction: Loading .npy files as dataset for pytorch

Input: I have preprocessed data in .npy files, let's call it X.npy for raw data and Y.npy for labels. They're organized to match every element from both files (first element from X has first label from Y etc.). How can I load it as dataset using torch.utils.data.DataLoader? I'm very new to pytorch, and any help will be useful.

Output:
You could also use DatasetFolder, which basically is the underlying class of ImageFolder. Using this class you can provide your own file extensions and loader to load the samples.

def npy_loader(path):
    return torch.from_numpy(np.load(path))


Instruction: GroupNorm is considerably slower and consumes higher GPU memory than BatchNorm in Pytorch

Input:
I use GroupNorm in pytorch instead of BatchNorm and keep all the others (network architecture) unchanged. It shows that in Imagenet dataset, using resnet50 architecture, GroupNorm is 40% slower than BatchNorm, and consumes 33% more GPU memory than BatchNorm. I am really confused because GroupNorm shouldn’t need more calculation than BatchNorm. The details are listed below.

For details of Group Normalization, one can see this paper: https://arxiv.org/pdf/1803.08494.pdf

For BatchNorm, one minibatch consumes 12.8 seconds with GPU memory 7.51GB;

For GroupNorm, one minibatch consumes 17.9 seconds with GPU memory 10.02GB.

I use the following code to convert all the BatchNorm layers to GroupNorm layers.

def convert_bn_model_to_gn(module, num_groups=16):
"""
Recursively traverse module and its children to replace all instances of
``torch.nn.modules.batchnorm._BatchNorm`` with :class:`torch.nn.GroupNorm`.
Args:
    module: your network module
    num_groups: num_groups of GN
"""
mod = module
if isinstance(module, nn.modules.batchnorm._BatchNorm):
    mod = nn.GroupNorm(num_groups, module.num_features,
                       eps=module.eps, affine=module.affine)
    # mod = nn.modules.linear.Identity()
    if module.affine:
        mod.weight.data = module.weight.data.clone().detach()
        mod.bias.data = module.bias.data.clone().detach()
for name, child in module.named_children():
    mod.add_module(name, convert_bn_model_to_gn(
        child, num_groups=num_groups))
del module
return mod

Output:
Yes, you are right GN does use more resources compared to BN. I'm guessing this is because it has to calculate the mean and variance for every group of channels, whereas BN only has to calculate once over the whole batch.

But the advantage with GN, is that you can lower your Batch Size up to 2, without reducing any performance, as stated within the paper, so you can make up for the overhead computation.


Instruction: PyTorch having trouble detecting CUDA

Input:
I am running CNN on PyTorch. The torch.cuda.is_available() function returned false and no GPU is detected. However, I can run Keras model with GPU. Here is my system information:

OS: Ubuntu 18.04.3
Python 3.7.3 (Conda)
GPU: GTX1080Ti
Nvidia driver: 430.50

When I check nvidia-smi, the output said that the CUDA version is 10.1. However, the nvcc -V command tells me that it is CUDA 9.1.

I downloaded NVIDIA-Linux-x86_64-430.50.run from the official site and install it with command line. I installed CUDA 10.1 using these following command line recommended by the official site:

wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run
sudo sh cuda_10.1.243_418.87.00_linux.run

I installed PyTorch through pip install. What is wrong? Thanks in advance!

Output:
The default Pytorch 1.2 package depends on CUDA 10.0, but you have CUDA 9.1. The output of nvidia-smi just tells you the maximum CUDA version your GPU supports, nvcc gives the CUDA installed on your system. It seems that your installation of CUDA 10.1 was unsuccessful.

In addition to CUDA 10.0, Pytorch also supports CUDA 9.2 and I've found that the Pytorch package compiled for CUDA 10.0 also works with CUDA 10.1. So you can either upgrade your CUDA installation to 9.2 and install the Pytorch CUDA 9.2 package with

pip3 install torch==1.2.0+cu92 torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

Or get a working installation of CUDA 10.1. There are detailed Linux instructions here. (Note that you may have to remove previous installations of CUDA before installing a new one.)


Instruction: Module Not Found Error when importing Pytorch_Transformers

Input:
After downloading pytorch_transformers through Anaconda and executing the import command through the Jupyter Notebook, I am facing several errors related to missing modules.

I tried searching sacremoses to import the package via Anaconda, but it is only available for Linux machines. Has anyone else faced similar issues? Thanks in advance!

from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM

This is the error:

<ipython-input-5-218d0858d00f> in <module>
----> 1 from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM

~\Anaconda3\lib\site-packages\pytorch_transformers\__init__.py in <module>
      1 __version__ = "1.2.0"
----> 2 from .tokenization_auto import AutoTokenizer
      3 from .tokenization_bert import BertTokenizer, BasicTokenizer, WordpieceTokenizer
      4 from .tokenization_openai import OpenAIGPTTokenizer
      5 from .tokenization_transfo_xl import (TransfoXLTokenizer, TransfoXLCorpus)

~\Anaconda3\lib\site-packages\pytorch_transformers\tokenization_auto.py in <module>
     24 from .tokenization_transfo_xl import TransfoXLTokenizer
     25 from .tokenization_xlnet import XLNetTokenizer
---> 26 from .tokenization_xlm import XLMTokenizer
     27 from .tokenization_roberta import RobertaTokenizer
     28 from .tokenization_distilbert import DistilBertTokenizer

~\Anaconda3\lib\site-packages\pytorch_transformers\tokenization_xlm.py in <module>
     25 from io import open
     26
---> 27 import sacremoses as sm
     28
     29 from .tokenization_utils import PreTrainedTokenizer

ModuleNotFoundError: No module named 'sacremoses'```

Output:
Please try to create a conda environment and install the packages in the created environment using the below steps:
conda create -n env_pytorch -c intel python=3.5
source activate env_pytorch
pip install pytorch-transformers


Instruction: how to identify wrong classification with batches in pytorch

Input:
I have a script like this where batches of images were used

correct = 0
total = 0
incorrect_classification=[]
for (i, [images, labels]) in enumerate(test_loader):
  images = Variable(images.view(-1, n_pixel*n_pixel))
  outputs = net(images)
  _, predicted = torch.min(outputs.data, 1)
  total += labels.size(0)
  correct += (predicted == labels).sum()
print('Accuracy: %d %%' %
      (100 * correct / total))

With a batch size of 10, each enumerate returns 10 x image size tensor. How can I save all the wrong classifications into the array incorrect_classification or wrong img and their probability into an dictionary so I use can plt.imshow to inspect them later?

If batch size is 1 I could use this:

if (predicted==labels).item()==0:
    incorrect_examples.append(images.numpy())

But with a batch size specified (like 100 images per batch) how should I save the wrong classifications?

Thanks in advance for any answers.

Output:
As already said in the comment of @zihaozhihao, images[predicted==labels] should do the work.

In other words, you will get a mask of indexes and then access the images you want with this mask:

correct = 0
total = 0
incorrect_examples=[]
for (i, [images, labels]) in enumerate(test_loader):
    images = Variable(images.view(-1, n_pixel*n_pixel))
    outputs = net(images)
    _, predicted = torch.min(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()
    print('Accuracy: %d %%' % (100 * correct / total))

    # if (predicted==labels).item()==0:
    #     incorrect_examples.append(images.numpy())

    idxs_mask = (predicted == labels).view(-1)
    incorrect_examples.append(images[idxs_mask].numpy())

The view(-1) will flatten the mask which will be used to mask the batch channel of the images tensor.

At the end of the loop (out of it), the itens in the list incorrect_examples will have shape [batch_size, n_pixel, n_pixel], and for convenience, you can group all of them in one tensor by concatenating them:

incorrect_images = torch.cat(incorrect_examples)
# incorrect_images.size() -> (n_incorrect_images, n_pixel, n_pixel)


Instruction: numpy argmax in array with multiple brackets

Input:
I have an issue in apply argmax to an array which has multiple brackets.
In real life I am getting this as a result of a pytorch tensor.
Here I can put an example:

a = np.array([[1.0, 1.1],[2.1,2.0]])
np.argmax(a,axis=1)

array([1, 0])

It is correct. But:

a = np.array([[[1.0, 1.1]],[[2.1,2.0]]])
np.argmax(a,axis=1)

array([[0, 0],
       [0, 0]])

It does not give me what I expect.
Consider that in reality I have this level of inner brackets:

a = np.array([[[[1.0, 1.1]]],[[[2.1,2.0]]]])

Output:
Use .squeeze() and a negative index.

a = np.array([[[[1.0, 1.1]]], [[[2.1, 2.0]]]])
np.argmax(a, axis = -1).squeeze()

array([1, 0], dtype=int32)


Instruction: Difference of torch.matmul and python built-in @ operator to do matrix multiplication

Input:
Can I always replace torch.matmul with python's built-in @ operator to do the matrix multiplication? Please assume that I know the difference between torch.matmul, torch.mm and many others. I just want to make sure how many of them can be safely replaced by @ operator without sacrificing speed or some native support from torch.

If it does no harm, I would like to extensively use them in the future.

Output: yes you can always use @ in place of .matmul. You can also do the same in numpy. Also see PyTorch mapping operators to functions


Instruction: Is there a handy way to dump the running_stats for a pytorch model?

Input:
I'm writing a C version of the pytorch model to run it on my special hardware.
Everything looks ok so far, except the running_mean and running_var in every batchnorm layer.

We have a python code to dump all named_parameters, but nothing to do for the running_stats, although we need to use it in the forwarding computation.

So is there a way to dump it with sort of built-in features?
I searched pytorch doc, no help on my task.
Otherwise I might need to write a regexp code to recognize and dump them.

Thanks a lot.
/Patrick

for name, param in model.named_parameters():
    # here can dump weight and bias, but not running_stats
    names.append(name)
    shapes.append(list(param.data.numpy().shape))
    values.append(param.data.numpy().flatten().tolist())

Output:
running_mean and others are registered_buffers in PyTorch. You can save (as you say dump) them with torch.nn.Module's state_dict:

torch.save(model.state_dict(), PATH)

You can iterate over named buffers and save each of them however you like similarly to parameters:

for name, buffer in model.named_buffers():
    # do your thing with them


Instruction: what does nn.Linear() do in pytorch's last, and why is it necessary?

Input:
I am working with some code that trains an lstm to generate sequences. After training the model, the lstm() method is called:

x = some_input
lstm_output, (h_n, c_n) = lstm(x, hc)
funcc = nn.Linear(in_features=lstm_num_hidden,
                  output_features=vocab_size,
                  bias=True)
func_output = func(lstm_output)

I've looked at the documentation for nn.Linear() but I still don't understand what this transformation is doing and why it is necessary. If the lstm has already been trained, then the output it gives should already have a pre-established dimensionality. This output (lstm_output) would be the generated sequence, or in my case an array of vectors. Am I missing something here?

Output:
Here, the Linear layer is transforming the hidden state representations (lstm_output) produced by the LSTM into a vector of size vocab_size. Your understanding is perhaps wrong. The Linear layer should be trained along with the LSTM.

And I guess you are trying to generate a sequence of tokens (words), so the Linear layer should be followed by a Softmax operation to predict a probability distribution over the vocabulary.


Instruction: Pytorch:Apply cross entropy loss with custom weight map

Input:
I am solving multi-class segmentation problem using u-net architecture in pytorch.
As specified in U-NET paper, I am trying to implement custom weight maps to counter class imbalances.

Below is the opertion which I want to apply -

Also, I reduced the batch_size=1 so that I can remove that dimension while passing it to precompute_to_masks function.
I tried the below approach-

def precompute_for_image(masks):
    masks = masks.cpu()
    cls = masks.unique()
    res = torch.stack([torch.where(masks==cls_val, torch.tensor(1), torch.tensor(0)) for cls_val in cls])
    return res

def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):

        ###################
        # train the model #
        ###################
        model.train()
        for batch_idx, (data, target) in enumerate(final_train_loader):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            optimizer.zero_grad()
            output = model(data)
            temp_target = precompute_for_image(target)
            w = weight_map(temp_target)
            loss = criterion(output,target)
            loss = w*loss
            loss.backward()
            optimizer.step()

            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

    return model

where weight_map is the function to calculate weight mask which I got from here
The issue, I am facing is I am getting memory error when I apply the following method. I am using 61gb RAM and Tesla V100 GPU.
I really think I am applying it in incorrect way.
How to do it?
I am omitting the non-essential details from the training loop.
Below is my weight_map function:

from skimage.segmentation import find_boundaries

w0 = 10
sigma = 5

def make_weight_map(masks):
    """
    Generate the weight maps as specified in the UNet paper
    for a set of binary masks.

    Parameters
    ----------
    masks: array-like
        A 3D array of shape (n_masks, image_height, image_width),
        where each slice of the matrix along the 0th axis represents one binary mask.

    Returns
    -------
    array-like
        A 2D array of shape (image_height, image_width)

    """
    nrows, ncols = masks.shape[1:]
    masks = (masks > 0).astype(int)
    distMap = np.zeros((nrows * ncols, masks.shape[0]))
    X1, Y1 = np.meshgrid(np.arange(nrows), np.arange(ncols))
    X1, Y1 = np.c_[X1.ravel(), Y1.ravel()].T
    for i, mask in enumerate(masks):
        # find the boundary of each mask,
        # compute the distance of each pixel from this boundary
        bounds = find_boundaries(mask, mode='inner')
        X2, Y2 = np.nonzero(bounds)
        xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
        ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
        distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
    ix = np.arange(distMap.shape[0])
    if distMap.shape[1] == 1:
        d1 = distMap.ravel()
        border_loss_map = w0 * np.exp((-1 * (d1) ** 2) / (2 * (sigma ** 2)))
    else:
        if distMap.shape[1] == 2:
            d1_ix, d2_ix = np.argpartition(distMap, 1, axis=1)[:, :2].T
        else:
            d1_ix, d2_ix = np.argpartition(distMap, 2, axis=1)[:, :2].T
        d1 = distMap[ix, d1_ix]
        d2 = distMap[ix, d2_ix]
        border_loss_map = w0 * np.exp((-1 * (d1 + d2) ** 2) / (2 * (sigma ** 2)))
    xBLoss = np.zeros((nrows, ncols))
    xBLoss[X1, Y1] = border_loss_map
    # class weight map
    loss = np.zeros((nrows, ncols))
    w_1 = 1 - masks.sum() / loss.size
    w_0 = 1 - w_1
    loss[masks.sum(0) == 1] = w_1
    loss[masks.sum(0) == 0] = w_0
    ZZ = xBLoss + loss
    return ZZ

Traceback of the error-

MemoryError                               Traceback (most recent call last)
<ipython-input-30-f0a595b8de7e> in <module>
      1 # train the model
      2 model_scratch = train(20, final_train_loader, unet, optimizer,
----> 3                       criterion, train_on_gpu, 'model_scratch.pt')

<ipython-input-29-b481b4f3120e> in train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path)
     24             loss = criterion(output,target)
     25             target.requires_grad = False
---> 26             w = make_weight_map(target)
     27             loss = W*loss
     28             loss.backward()

<ipython-input-5-e75a6281476f> in make_weight_map(masks)
     33         X2, Y2 = np.nonzero(bounds)
     34         xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
---> 35         ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
     36         distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
     37     ix = np.arange(distMap.shape[0])

MemoryError:

Output:
Your final_train_loader provides you with an input image data and the expected pixel-wise labeling target. I assume (following pytorch's conventions) that data is of shape B-3-H-W and of dtype=torch.float.
More importantly, target is of shape B-H-W and of dtype=torch.long.

On the other hand make_weight_map expects its input to be C-H-W (with C = number of classes, NOT batch size), of type numpy array.

Try providing make_weight_map the input mask as it expects it and see if you get similar errors.
I also recommend that you visualize the resulting weight map - to make sure your function does what you expect it to do.


Instruction: How to do sequence classification with pytorch nn.Transformer?

Input:
I am doing a sequence classification task using nn.TransformerEncoder(). Whose pipeline is similar to nn.LSTM().

I have tried several temporal features fusion methods:

Selecting the final outputs as the representation of the whole sequence.
Using an affine transformation to fuse these features.
Classifying the sequence frame by frame, and then select the max values to be the category of the whole sequence.

But, all these 3 methods got a terrible accuracy, only 25% for 4 categories classification. While using nn.LSTM with the last hidden state, I can achieve 83% accuracy easily. I tried plenty of hyperparameters of nn.TransformerEncoder(), but without any improvement for the accuracy.

I have no idea about how to adjust this model now. Could you give me some practical advice? Thanks.

For LSTM: the forward() is:

    def forward(self, x_in, x_lengths, apply_softmax=False):

        # Embed
        x_in = self.embeddings(x_in)

        # Feed into RNN
        out, h_n = self.LSTM(x_in) #shape of out: T*N*D

        # Gather the last relevant hidden state
        out = out[-1,:,:] # N*D

        # FC layers
        z = self.dropout(out)
        z = self.fc1(z)
        z = self.dropout(z)
        y_pred = self.fc2(z)

        if apply_softmax:
            y_pred = F.softmax(y_pred, dim=1)
        return y_pred

For transformer:

    def forward(self, x_in, x_lengths, apply_softmax=False):

        # Embed
        x_in = self.embeddings(x_in)

        # Feed into RNN
        out = self.transformer(x_in)#shape of out T*N*D

        # Gather the last relevant hidden state
        out = out[-1,:,:] # N*D

        # FC layers
        z = self.dropout(out)
        z = self.fc1(z)
        z = self.dropout(z)
        y_pred = self.fc2(z)

        if apply_softmax:
            y_pred = F.softmax(y_pred, dim=1)
        return y_pred

Output:
The accuracy you mentioned indicates that something is wrong. Since you are comparing LSTM with TransformerEncoder, I want to point to some crucial differences.

Positional embeddings: This is very important since the Transformer does not have recurrence concept and so it doesn't capture sequence information. So, make sure you add positional information along with the input embeddings.
Model architecture: d_model, n_head, num_encoder_layers are important. Go with the default size as used in Vaswani et al., 2017. (d_model=512, n_head=8, num_encoder_layers=6)
Optimization: In many scenarios, it has been found that the Transformer needs to be trained with smaller learning rate, large batch size, WarmUpScheduling.

Last but not least, for a sanity check, just make sure the parameters of the model is updating. You can also check the training accuracy to make sure the accuracy keeps increasing as the training proceeds.

Although it is difficult to say what is exactly wrong in your code but I hope that the above points will help!


Instruction: Splitting a directory with images into sub folders using Pytorch or Python

Input:
I have a directory with two sub directories in it. Each sub-dir has bunch of images. These sub-dir also specify the two classes of images.

I want to have 3 directories (train, validation, test) and within each of these 3 sub dir, I want to 2 sub directory of each class respectively with images.

I want to split the no.of images into train, val, test directories by random sampling. SO some 60% images go to train, 20, val and 20 in test.

Initial Structure:

Main_folder
 - Good
 - Bad

What I want:

Main_folder:
 - Train
  - Good
  - Bad
 - Val
  - Good
  - Bad
 - Test
  - Good
  - Bad

I want to split each sub directories into two directories within it with random assignment of images (

Output:
There is a pytorch way to do that (and I would advise using single library for such easy task).
Creating dataset
There is a pre-made one inside torchvision, namely ImageFolder.
Simply use it like this:
import torchvision

dataset = torchvision.datasets.ImageFolder("./my_data")

This will create a dataset, where Good folder has 0 label and Bad has label of 1 for each image in those folder respectively.
Dividing into train, validation, test
No need for sklearn for such an easy job + torch has most of the numpy functionality as well and I would rather stick with one library instead of 3 (though train_test_split could be used twice consecutively, similar to what @ESZ proposed).
IMO simpler would be:
def get_subset(indices, start, end):
    return indices[start : start + end]

TRAIN_PCT, VALIDATION_PCT = 0.6, 0.2  # rest will go for test
train_count = int(len(dataset) * TRAIN_PCT)
validation_count = int(len(dataset) * VALIDATION_PCT)

indices = torch.randperm(len(dataset))

train_indices = get_subset(indices, 0, train_count)
validation_indices = get_subset(indices, train_count, validation_count)
test_indices = get_subset(indices, train_count + validation_count, len(dataset))

This will create indices for SubsetRandomSampler and torch.utils.data.DataLoader.
So, similarly to @ESZ once again:
dataloaders = {
    "train": torch.utils.data.DataLoader(
        dataset, sampler=SubsetRandomSampler(train_indices)
    ),
    "validation": torch.utils.data.DataLoader(
        dataset, sampler=SubsetRandomSampler(validation_indices)
    ),
    "test": torch.utils.data.DataLoader(
        dataset, sampler=SubsetRandomSampler(test_indices)
    ),
}

You can specify batch_size and other argument to DataLoader, see documentation if you need more info.


Instruction: Increasing the size of images displayed in Pytorch

Input:
I want to display few images and their respective labels using Pytorch dataloader.

However the image displayed is very tiny grid.

How do I increase the width of each image so it's bigger.

Here's the code I used:

mean_nums = [0.485, 0.456, 0.406]
std_nums = [0.229, 0.224, 0.225]

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array(mean_nums)
    std = np.array(std_nums)
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated

# Get a batch of training data
inputs, classes = next(iter(dataloaders['trainLoader']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs,nrow=2)

imshow(out, title=[image_datasets['train'].classes[x] for x in classes])

Output:
Try to insert plt.figure(figsize=[width, height]) before plt.imshow And choose the width, height  that will satisfy you.

So, for example, the imshow function may be:

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array(mean_nums)
    std = np.array(std_nums)
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.figure(figsize=[20, 20])
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated


Instruction: Pytorch :Why my dataset variance do not get the correct result?

Input:
Here is the function I wrote:

def channel_var(image_dataset):
    res = image_dataset[0]
    for image in image_dataset[1:]:
        res += image
    return tuple(map(lambda x: x/len(image_dataset),
     (torch.var(res[0]),
      torch.var(res[1]),
      torch.var(res[2]))))

then I tested it with a Normal distribution :

m = normal.Normal(0, 3)
m.sample((1, 3, 32, 32))

And I get this wrong result :

channel_var(list_test)

>>(tensor(0.0338), tensor(0.0352), tensor(0.0365))

Thank you

Output:
Your function is wrong. And that's because you are computing the average image and then computing the channel variance in the average image. I don't think you want that. You can just find the variance in each channel by using

torch.var(img, dim=[0,2,3])

assuming dim=1 is the channel dimension and img is a torch tensor. If img is not a torch tensor, you can concatenate list of imgs to make a tensor.

You can do this as torch.var(torch.cat(img, dim=0), dim=[0,2,3]) cat operation concatenates list to a tensor.


Instruction: Conda install of pytorch fails

Input:
I created an environment with conda and I want to install pytorch in it, but it doesn't work. After I get inside my environment with source activate env_name I tried this: conda install pytorch torchvision -c pytorch (I also tried it like this: conda install -c pytorch pytorch torchvision) but I am getting this error:

Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: ......
Error: Could not find some dependencies for pytorch: mkl >=2018, cudatoolkit >=9.0,<9.1, blas * mkl, cudatoolkit >=10.0,<10.1, cudatoolkit >=9.2,<9.3, blas * openblas, cudnn 7.0.*, cudatoolkit 9.*

Did you mean one of these?

    pytorch, pytorch-gpu, pytorch-cpu

Did you mean one of these?

    cudatoolkit

You can search for this package on anaconda.org with

    anaconda search -t conda cudatoolkit 9.*

 (and similarly for the other packages)

Here are my installed packages:

backports                 1.0                      py34_0
backports.shutil-get-terminal-size 1.0.0                     <pip>
decorator                 4.0.11                   py34_0
get_terminal_size         1.0.0                    py34_0
ipython                   4.2.0                    py34_0
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py34_0
libgfortran               1.0                           0
numpy                     1.9.2                    py34_0
openssl                   1.0.2l                        0
path.py                   10.0                     py34_0
pexpect                   4.2.1                    py34_0
pickleshare               0.7.4                    py34_0
pip                       9.0.1                    py34_1
ptyprocess                0.5.1                    py34_0
python                    3.4.5                         0
readline                  6.2                           2
scipy                     0.16.0               np19py34_0
setuptools                27.2.0                   py34_0
simplegeneric             0.8.1                    py34_1
six                       1.10.0                   py34_0
sqlite                    3.13.0                        0
tk                        8.5.18                        0
traitlets                 4.3.1                    py34_0
wheel                     0.29.0                   py34_0
xz                        5.2.3                         0
zlib                      1.2.11                        0

What should I do? Thank you!

Output:
Pytorch's vision package (aka torchvision) was developed post-Python 3.4, and so only has versions supporting Python 2.7, 3.5-7. Please create a new environment with a later Python version. Note it is always better to include the packages you care about in the creation of the environment, e.g.,

conda create -n env_name -c pytorch torchvision

and Conda will figure the rest out. If you need to have a specific version of Python, you can include that as well (e.g., python=3.6).