How to optimize pip imports for Dockerfile layers caching
I have a Dockerfile for ML/DL stack that needs a lot of requirements that could be logically split into python standard libraries and python ml libraries at least:

Python libraries (requirements.txt):

Cython
python-dateutil==2.8.0
setuptools&gt;=41.0.0
progressbar2
argparse
smart_open
backoff
boto3
botocore
google
protobuf
tornado==5.1.1


Python ml libraries (requirements.lib.txt):

numpy==1.15.1
intel-numpy
matplotlib
pandas
scipy==1.2.1
scikit-learn==0.21.3
torch
tensorflow==1.14.0
keras==2.1.1


Now when I build my Docker image it turns out that the whole image size is ~5-7GB.
The other point is that I have, in terms of size in MB, small layers (&lt; 100MB), big layers (~100MB-500MB) and huge layers (>500MB up to 1-2GB).

Of course ml python libraries cached to layers do not help since torch itself is about 800MB, tensorflow is ~500MB, intel mkl is about 300MB, etc.

Currently to prevent packages version override I do like

COPY $ROOT_APPLICATION/src/requirements.txt /tmp/requirements.txt
RUN cat /tmp/requirements.txt | xargs -n 1 -L 1 pip3 install

COPY $ROOT_APPLICATION/src/requirements.lib.txt /tmp/requirements.lib.txt
RUN cat /tmp/requirements.lib.txt | xargs -n 1 -L 1 pip3 install


where I do a copy to /tmp before RUN for caching reasons.
To keep everything compact I start from the good python slim buster:

FROM python:3.7.4-slim-buster


while the missing part of the Dockerfile is the apt-get of required libraries like:

########################################  BASE SYSTEM
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends \
    software-properties-common \
    build-essential \
    pkg-config \
    libjemalloc2 \
    libgmp3-dev \
    libicu-dev \
    python3.7-icu \
    swig \
    curl \
    unzip \
    cron \
    jq

######################################## AUDIO
RUN apt-get update &amp;&amp; apt-get install -y \
    libyaml-0-2 \
    libfftw3-3 \
    libtag1v5 \
    libsamplerate0 \
    libasound-dev \
    portaudio19-dev \
    libportaudio2 \
    libportaudiocpp0 \
    ffmpeg \
    espeak


How to optimize layers caching to 1) keep the docker images as small as possible 2) optimize layer's size for bandwidth when using docker push (then avoiding to push to registry huge layers over the network)?


You're going to want pip install --no-cache-dir, so it doesn't keep copies of the downloads around.
You don't want to keep the toolchain (compiler etc.) installed, but you need them to build the image. So what you do is, you use multi-stage builds: you use one image to build everything, and then a second image that just copies the built packages over and omits all the build tools and artifacts. You can find a guide to multi-stage builds for Python here (it's three parts, this is part 1): https://pythonspeed.com/articles/smaller-python-docker-images/

