Deploy your model with Docker

Download this notebook

With the finished program, we can now look at how to deploy it permanently. As we have used several Python packages, which in turn have numerous dependencies and may rely on specific versions, we are looking for a way to run our program reliably on (almost) any server independently of other applications. Software for container virtualization, such as Docker, is ideal for this.

What is Docker?

Docker is a software that combines applications in so-called containers. In addition to the application itself, these containers also contain the necessary libraries (or packages), configurations and other files that are required to run the application. Several containers can then be operated on one machine in isolation from each other and only communicate via explicitly defined channels. Compared to virtual machines, they are extremely lean, with the possibility to run several containers on one system without any problems. Even though Docker was originally developed for Linux, it can now also be installed on Windows and macOS. You can find installation instructions here.

Each container is created from a so-called image, which cannot be changed and serves as a template for the containers, so to speak. In a Dockerfile, a few commands are used at the beginning to describe how this image is structured. A new layer is created for each of these commands, so that the resulting image consists of several layers and an exact history can be traced. Custom images are usually based on existing images, which can be found in registries and can be downloaded directly from there. The best known for this is Docker Hub.

Python requirements#

In order to be able to build our Docker image easily later, we need to know which dependencies our application has. We have developed in Python 3.7 and installed the latest version of the required packages. Since we already know that our application works with this version, we will keep it for the deployment. As you should already know, the requirements for a Python project are usually recorded in a requirements.txt file. Even independently of Docker, it always makes sense to keep such a file up to date in order to make it as easy as possible for others to re-execute it after some time.

If we have not already done so, we now create a requirements.txt file. If the version of individual packages is no longer known, it can be queried within Python using the __version__ attribute directly on the imported package. Alternatively, with the command

pip freeze > requirements.txt

all packages installed in the current Python environment are written to a file together with their version. The call seems to be simple, but the generated list should be checked again manually! Among other things, packages are listed here that may have been used in the meantime during development but are no longer used in the final project. In addition, not only manually installed packages are included, but also all dependencies installed with them. However, these can vary from operating system to operating system and thus lead to problems. Our finished requirements.txt contains the following entries:

Flask==2.0.2
Flask-Cors==3.0.10
gunicorn==20.1.0
numpy==1.21.4
pandas==1.3.4
scikit-learn==1.0.1
tensorflow-cpu==2.7.0

We did not include the photonai package, as we only needed it in the first chapter, but decided to use the TensorFlow solution for the final application. Instead, only gunicorn appears in the list, although we did not need this package during development. However, we want to use it to run the web server in our Docker container and have to install it accordingly beforehand. We have also replaced the tensorflow package with tensorflow-cpu to reduce the size of the container and because our free Heroku package does not include a GPU anyway. If you have another server with a GPU available, it often makes sense to use this as well.

Dockerfile#

The Dockerfile is a text file with the same name (Dockerfile) without any other extension. This file serves Docker as a kind of recipe that can be used to build an image. As previously mentioned, a new layer is created for each command so that a history can be traced later. Docker can recognize changes in files and only then rebuilds a layer. However, as soon as a layer has been replaced, all subsequent layers must also be recreated. This means that the order of the commands in the Dockerfile is relevant. Our Dockerfile has the following structure:

FROM python:3.7
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ .
COPY models/ /models
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "main:app"]

First, we specify which image we want to use as the basis for our own image. Fortunately, there are official Python images where we can also specify a version. As this image is available in the official Docker Hub (see here), we do not need to explicitly specify a registry, just the name and our desired version. If the image is not yet available locally, Docker will automatically download it from there later.

Alternative Registry: Due to a rate limit with Docker, it may be useful to use an alternative registry, e.g. Harbor (see below).

Now we change our WORKDIR to /app. This step is comparable to calling cd in the command line. We simply change our work location within the container. We then copy our previously created requirements.txt file into this folder within the container. The RUN command can be used to execute any command within the container. We use it to install the packages with pip. As we are in a separate environment anyway, we do not need to use an additional virtual environment for the installation.

Due to the structure of several layers we only copy our code and the required models into the container afterwards. If we have to change something in our code or replace the models later, the installed packages can be retained and do not have to be reinstalled. We do not copy our models into the /app folder, but into a separate folder. The simple reason for this is that it preserves the relative paths that we have specified in our program for importing the models.

In the last line, we use the CMD command to specify what is to be executed when the container is started. This command is therefore not used to create the image. In our case, we start our Gunicorn web server with our app here. We use 5000 as the port, we must remember this information (or simply look it up again here later). The port is initially only released within the container and is therefore not accessible from outside.

Image and container#

We can now build our image with our finished Dockerfile. Docker must be installed for this. If this is the case, you can execute the following command in the command line:

docker image build -t deploy-tutorial .

If you are on a system with an ARM processor (e.g. Apple’s M series), some packages may not be available for this architecture and errors may occur. In this case, you can tell Docker to build the container for Intel/AMD architectures. This makes the local execution a little slower, but the image is then perfectly stored on a server with the corresponding architecture.

docker image build --platform linux/amd64 -t deploy-tutorial .

Note the . at the end of the command for the current directory. The Dockerfile should be in the same directory and all other files must of course be findable from there according to the specifications. You can also use -t to assign a name so that you can find the image again later. Docker will then create your image. This step can take several minutes the first time, but the time is significantly reduced the second time due to the layers. With the command docker image ls you can view all local images. The image you have just created should now also appear there.

If this is the case, you can start a container from it. The command for this is

docker run -d -p 8000:5000 --name deploy-tutorial-container deploy-tutorial.

The -d flag ensures that your container is detached, i.e. runs in the background and the console window does not have to remain open. As already mentioned, we have so far only released the port within the container. With -p 8000:5000 we now forward it to the outside. The port on the host machine is specified first and then the port within the container. Both do not have to match, but the port inside the container must correspond to our specification when starting the web server in the Dockerfile. Optionally, we can assign a name for our container with --name, otherwise Docker assigns a random name. Finally, we specify our previously created image that is to be used to start the container. We can now call our API again in the browser and use it as usual. Note that we have changed the port to 8000 for the example.

If you encounter problems during execution, you can simply omit the -d flag. This will display the error reports directly in your console window, so you can get to the bottom of the cause.

Over time, many old containers can quickly accumulate on your system. The command docker ps -a can be used to display all running and stopped containers. You can stop and delete them with docker kill CONTAINER and docker rm CONTAINER if they are no longer needed.

Multiple services: Docker Compose#

It often makes sense to separate individual services of an app, for example because different programming languages or versions are used. It can also be prepared so that individual services are used on different hardware, e.g. a neural network often requires GPU support, whereas a web interface can manage without it.

If you need several services during your project that communicate with each other (e.g. an API that executes the neural network and delivers results and a frontend, e.g. a dashboard), then you can use several folders in your project that contain a Dockerfile and the associated files (e.g. the folders api and frontend), then you can use a file docker-compose.yaml at the top level. In this case, such a file could look like this:

version: '3'
services:
  api:
    container_name: deploy-tutorial-api
    build:
      context: ./api
      dockerfile: Dockerfile
      platforms:
        - "linux/amd64"
    environment:
      APP_NAME: "DEPLOYMENT-TUTORIAL"
    ports:
      - '8000:80'
    networks:
      - deployment-network
    healthcheck:
          test: curl --fail http://localhost:80/ || exit 1
          interval: 10s
          timeout: 10s
          retries: 3
          start_period: 8s
   
  frontend:
    container_name: deploy-tutorial-frontend
    build:
      context: ./frontend
      dockerfile: Dockerfile
      platforms:
        - "linux/amd64"
    environment:
      API_URL: "http://api:80/"
    ports:
      - '8002:8080'
    depends_on:
      api:
        condition: service_healthy
    networks:
      - deployment-network
       
 
networks:
  deployment-network:

With the command

docker compose up

a container is now built (use the -d flag again to let it run in the background). The two services communicate via an internal network (deployment-network). When executed, both services are built and executed one after the other, whereby a health check is performed to ensure that the API is working before the frontend is started. The frontend would now be accessible via http://localhost:8002/.

By the way, you can use the following command to rebuild a container if something has been changed in the code:

docker compose up -d --build

You can stop the container started in this way with the following command:

docker compose stop.

Saving the container in GitLab#

GitLab offers a container registry in which we can save our container in order to be able to deploy it on the respective deployment platforms.

To display the container registry, you can go to the menu item Packages & Registries → Container Registry in the GitLab repository. If you do not see this, you must first activate the container registry in the repository settings.

To push the container to GitLab, you must first log in there via the console. This can be done using a Personal Access Token or a Deployment Token, which we have explained in our introduction in the section about Git.

The command for the login in the console is now

docker login <registry url> -u <username> -p <token>

The URL for the container registry in WWU’s zivgitlab is zivgitlab.wwu.io.

You should now build your container locally and give it a name in the following form

<registry URL>/<namespace>/<project>/<image>

In our example, the command to build the image would be

docker image build -t zivgitlab.wwu.io/reach-euregio/incubaitor/deploy-tutorial .

To upload the image to GitLab, you can use the push command again:

docker push zivgitlab.wwu.io/reach-euregio/incubaitor/deploy-tutorial

Save the container in the harbor#

The University of Münster offers a container registry in which we can store our container in order to be able to roll it out on the respective deployment platforms. This is the Harbor https://harbor.uni-muenster.de/. For instructions, see also the following guide Docker Registry.

It may be useful to store additional images there, as Docker has introduced a rate limit. For example, the image python:3.7 is used above. It may be useful to set up your own project project on the harbor and upload the locally used image there.

To log in to the harbor, you can use the command

docker login harbor.uni-muenster.de

can be used. The username is then nutzerkennung@uni-muenster.de and the password can be obtained via the personal settings in Harbor under CLI Secret.

To upload the image to the project, you can execute the following commands, for example:

docker pull python:3.7.

docker tag python:3.7 harbor.uni-muenster.de/<projekt>/python:3.7

docker push harbor.uni-muenster.de/<project>/python:3.7

Other local images can then also be uploaded.

Instead of

FROM python:3.7

you can now use

FROM harbor.uni-muenster.de/<projekt>/python:3.7

in the Dockerfile. The project must be publicly accessible for this. Alternatively, you can create your own access in the harbor (robot account). You can also use this to push a registry.