Docker Compose - adding volume using Amazon S3 - django

Question

I'm using docker-compose v2 to build my containers (Django and Nginx).
I'm wondering how to store the static and media files. At the beginning I stored them as a volume on the machine, but the machine crashed and I lost the data (or at least, I didn't know how to recover it).
I thought it's better to store it on Amazon S3, but there are no guides for that (maybe it means something :) ).
This is my docker-compose file:
I tried to add the the needed fields (name, key, secret,...) but no success so far.
Is it the right way?
Thanks!
version: '2'
services:
web:
build:
context: ./web/
dockerfile: Dockerfile
expose:
- "8000"
volumes:
- ./web:/code
- static-data:/www/static
- media-data:/www/media
env_file: devEnv
nginx:
build: ./nginx/
ports:
- "80:80"
volumes:
- static-data:/www/static
- media-data:/www/media
volumes_from:
- web
links:
- web:web
volumes:
static-data:
driver: local
media-data:
driver: s3

Answer 1

Here a example of how upload files to S3(for backup) from a container, but would be made in the host OS too, since you have a volume of container mounted on Host OS.
In this script I download media from S3 to a local container/server. After it, I use pynotify to watch the dir static/media, for modifications. If any change occur it upload the file to S3 using the command subprocess.Popen(upload_command.split(" ")).
I think that you can adapt this script for your problem too.
Before you test this script, you should set your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on the environment variables of OS.
For more details S4cmd documentation.
#!-*- coding:utf-8 -*-
import pyinotify
import os
import subprocess
from single_process import single_process
# You most have set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
# in enviromnet variables
PROJECT_DIR = os.getcwd()
MEDIA_DIR = os.path.join(PROJECT_DIR, "static/media")
AWS_BUCKET_NAME = os.environ.get("AWS_BUCKET_NAME", '')
S4CMD_DOWNLOAD_MEDIA = "s4cmd get --sync-check --recursive s3://%s/static/media/ static/" % (AWS_BUCKET_NAME)
UPLOAD_FILE_TO_S3="s4cmd sync --sync-check %(absolute_file_dir)s s3://"+ AWS_BUCKET_NAME +"/%(relative_file_dir)s"
# Download all media from S3
subprocess.Popen(S4CMD_DOWNLOAD_MEDIA.split(" ")).wait()
class ModificationsHandler(pyinotify.ProcessEvent):
def process_IN_CLOSE_WRITE(self, event):
try:
dir = event.path
file_name = event.name
absolute_file_dir=os.path.join(dir, file_name)
relative_dir=dir.replace(PROJECT_DIR, "")
relative_file_dir=os.path.join(relative_dir, file_name)
if relative_file_dir.startswith("/"):
relative_file_dir = relative_file_dir[1:]
print("\nSeding file %s to S3" % absolute_file_dir)
param = {}
param.update(absolute_file_dir=absolute_file_dir)
param.update(relative_file_dir=relative_file_dir)
upload_command = UPLOAD_FILE_TO_S3 % param
print(upload_command)
subprocess.Popen(upload_command.split(" "))
except Exception as e:
# log excptions
print("Some problem:", e.message)
#single_process
def main():
handler = ModificationsHandler()
wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm, handler)
print("\nListening changes in: " + MEDIA_DIR)
if MEDIA_DIR:
wdd = wm.add_watch(MEDIA_DIR, pyinotify.IN_CLOSE_WRITE, auto_add=True, rec=True)
notifier.loop()
if __name__ == "__main__":
main()