Josh Schertz

Dockerize the pySecMaster

In an attempt to simplify the startup procedure for the pySecMaster, I've setup Docker for containerizing the system. The beautiful part of containerization is that I can now plan for the exact environment the scripts and database will be run in. Not only does this simplify the scripts I have to prepare, but it means that users can quickly get up and running.

Container Architecture

The pySecMaster structure is currently straightforward, with the Postgres database for storing all the values, and the python files for downloading, parsing and cleaning the data to be put into the database. I following this structure when establishing the generic Docker structure.

The first container stores the Postgres database engine, where I used the Postgres 10-alpine Docker image. The alpine images are much smaller than other distributions (~5MB), and is based on Alpine Linux. Following Docker best practices, I specified the database store the actual data in a dedicated volume container. This allows the Postgres engine container to be updated without worrying about the actual database values being impacted. Additionally, this container needs to be accessible from the localhost, thus the container's port 5432 needs to be exposed.

All of the Python files are run from a dedicated application container. I used the Python 3 container, where all of the pySecMaster files are copied into it via a Dockerfile. The Dockerfile also installs all dependent libraries required, and establishes the normal entrypoint and default commands. This container is dependent on the Postgres container, thus it must wait until Postgres is ready before starting up.

The entire Docker process is orchestrated by a Docker Compose file, which specifies the containers to create and what parameters they require. The file is listed below.

version: '3'

services:
    postgres:
        container_name: postgres_pysecmaster
        image: postgres:10-alpine
        volumes:
            - ./database:/docker-entrypoint-initdb.d/
            - postgres_pysecmaster_data:/var/lib/postgresql
        restart: always
        ports:
            - "127.0.0.1:5432:5432"

    app:
        container_name: pysecmaster
        build: .
        environment:
            - "PYTHONUNBUFFERED=1"  # Prevent python prints from using buffer
        volumes:
            - ./load_tables:/load_tables/
        depends_on:
            - postgres

volumes:
    postgres_pysecmaster_data:

Running the System

Getting the system up and running by:

  1. Installing Docker Community Edition (CE)
  2. Downloading the pySecMaster repository to your computer
  3. Navigate to `pySecMaster/pySecMaster`
  4. Open the `utilities/user_dir.py` file:
    1. Change the default name (josh) to your system user name (line 30)
    2. Add your Quandl API Key to both the root and your user section (lines 28 and 50)
    3. [optional] Change the `main_password` (lines 16 and 38) and `pysecmaster_password` (lines 23 and 45) in both the root and your user sections; Also change the passwords within the `pySecMaster/pySecMaster/database/init.sql` file on lines 2 and 12 to the same respective passwords you used in the `user_dir.py` file; You can also change or remove the special `remote_users` password and users created between lines 16 and 33 within the `init.sql` file.
  5. Run `docker-compose up`, and the system will start building itself using the variables specified here. To view the arguments you can provide, either run `pySecMaster/pySecMaster/pySecMaster.py -h` or view the arguments here.

If a script breaks or you need to change a parameter, you can rebuild the containers (required) by running:

docker-compose up --build

This will copy any file changes you make to the pySecMaster code into the Python container.

The Docker Compose script specifies that the Postgres container should always be restarted. Meaning that if the computer is rebooted or the container crashes, Docker will restart it.

By default, the pySecMaster will download the daily prices from Quandl, and only run the cross validator for the prior 30 days. You can change this default behavior by modifying the commands given in the `pySecMaster/pySecMaster/Dockerfile`.

Let me know if you have any questions. There is a lot of information I haven't mentioned here.