I have an aws code pipeline which currently successfully deploys code to my EC2 instances.
I have a Docker image that has the necessary setup to run my code, Dockerfile provided below. When I run docker run -t it just loads up an interactive shell on my docker but then hangs on any command (eg: ls)
Any advice?
FROM continuumio/anaconda2
RUN apt-get install git
ENV PYTHONPATH /app/phdcode/panaxeaA1
# setting up venv
RUN conda create --name panaxea -y
RUN /bin/bash -c "source activate panaxea"
# Installing necessary packages
RUN conda install -c guyer pysparse
RUN conda install -c conda-forge pympler
RUN pip install pysparse
RUN git clone https://github.com/usnistgov/fipy.git
RUN cd fipy && python setup.py install
RUN cd ~
WORKDIR /app
COPY . /app
RUN cd panaxeaA1/models/alpha04c/launchers
RUN echo "launching..."
CMD python launcher_260818_aws.py
docker run -t simply starts a docker container with a pseuodo-tty connection to the container's stdin. However, just running this command does not establish an interactive shell to the container. You will need this to be able to have run commands within your container.
You need to also append the -i command line flag along with the shell you wish to use. For example, docker run -it IMAGE_NAME bash will launch a container from the image you provide using bash as your interactive shell. You can then run Bash commands as you normally would.
If you are looking for a simple way to run containers on EC2 instances in AWS, I highly recommend AWS EC2 Container Service (ECS) as an option. It is a very simple service for running containers that abstracts and manages much of the server level work involved in running containers.
Related
I'm trying build an image for use on EC2 instances in an AWS Batch job. I need to use Ubuntu 18.04 because the goal is to run some Fortran software that I can only get to compile on Ubuntu 18.04. I have the Fortran software and some python scripts running well on a manually started Ubuntu 18.04 EC2 instance.
Now, I'm trying to build an image with Docker (that I'll eventually apply to 100s or 1000s of EC2 instances)... but I have to get CloudWatchAgent (CWA) installed and started, and I can't get CWA to start in the Docker build. CWA starts and runs fine in my manual EC2 development instance (Ubuntu 18.04). I initially had problems with CWA in my manual instance because CWA uses systemctl, and so I had to manually install systemd, and that worked after a reboot. But, I'm not able to replicate this in my Docker build, but always get the error:
System has not been booted with systemd as init system (PID 1). Can't operate.
unknown init system
The command '/bin/sh -c sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:amazon-cloudwatch-agent.json' returned a non-zero code: 1
I tried starting with an ubuntu 18.04 image that is supposed to have systemd already installed, and tried rebooting my EC2 instance, same error. Here's the source: https://hub.docker.com/r/jrei/systemd-ubuntu
I looked for other ideas, e.g.: Docker System has not been booted with systemd as init system
... but couldn't figure out how to make it work in a Docker build.
So,
am I using the Ubuntu 18.04 image (that has systemd) in my build wrong- how to use in a Docker build?
is there another way to start CloudWatchAgent in Ubuntu 18.04 that gets around the systemd problem?
would it work/is there a way to restart the operating system inside the Docker container, during the docker build stage?
am I stuck and will have to try recompile everything on a different Ubuntu or AMI like Amazon Linux?
Or is there something else I'm missing?
Here's my Docker file:
#version with systemd already installed
FROM jrei/systemd-ubuntu#sha256:1b65424e0ec4f6772576b55c49e1470ba506504d1033e9da5795785b1d6a4d88 as ubuntu-base
RUN apt-get update && apt-get install -y \
sudo \
wget \
python3-pip
RUN sudo apt-get -y install libgfortran3
RUN sudo pip3 install boto3
RUN wget https://s3.us-east-2.amazonaws.com/amazoncloudwatch-agent-us-east-2/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
RUN sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
COPY . .
RUN cp amazon-cloudwatch-agent.json /opt/aws/amazon-cloudwatch-agent/etc/
ENV ECS_AVAILABLE_LOGGING_DRIVERS = awslogs
RUN sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:amazon-cloudwatch-agent.json
RUN mkdir -p cpseqlogs
CMD python3 cpsequence.py
Thanks for any suggestions, ideas, or tips (I'm fairly new to Docker, but not totally green on linux).
I'm having trouble getting a docker container to update the images (like .png's) on my local system.
The docker container has script in the dockerfile that copies the images into a folder in the container. Then, that folder gets copied into a new directory that is a set up as a shared volume. This process is split between the dockerfile for this container, and a "command" entry in the docker-compose.yaml.
Everything seems to run fine; following the output of the copy command looks right. I don't get any errors, and once the command stops running the container stops.
I've tried destroying the container and image completely and recreating it, but I still see the old images in the application. I'm guessing that it's not overwriting the existing images, but I don't know why.
Dockerfile:
FROM ubuntu:18.04
# Install packages
RUN apt-get update
RUN apt-get install -y apt-utils
RUN apt-get install -y curl unzip
# Intall AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
# Load AWS Access Keys
COPY ./config /root/.aws/
COPY ./credentials /root/.aws/
RUN chmod 600 /root/.aws/config /root/.aws/credentials
# Download media
RUN mkdir -p /var/media
RUN aws s3 cp --recursive s3://images/ /var/media
Snippet from the docker-compose.yaml:
version: '3.1'
services:
client:
[[other stuff here]]
media:
image: [[our own image stored with AWS]]
command: 'cp -R -v -f /var/media/images/ /var/media-access/'
volumes:
- ./src/media:/var/media-access
[[other stuff here]]
Any advice is appreciated!
I have a Dockerfile as follow:
FROM centos
RUN mkdir work
RUN yum install -y python3 java-1.8.0-openjdk java-1.8.0-openjdk-devel tar git wget zip
RUN pip install pandas
RUN pip install boto3
RUN pip install pynt
WORKDIR ./work
CMD ["bash"]
where i am installing some basic dependencies.
Now when I run
docker run imagename
it does nothing but when I run
docker run -it imageName
I lands into the bash shell. But I want to get into the bash shell as soon as I trigger the run command without any extra parameters.
I am using this docker container in AWS codebuild and there I can't specify any parameters like -it but I want to execute my code in the docker container itself.
Is it possible to modify CMD/ENTRYPOINT in such a way that when running the docker image I land right inside the container?
I checked your container, it will not even build due to missing pip. So I modified it a bit so that it at least builds:
FROM centos
RUN mkdir glue
RUN yum install -y python3 java-1.8.0-openjdk java-1.8.0-openjdk-devel tar git wget zip python3-pip
RUN pip3 install pandas
RUN pip3 install boto3
RUN pip3 install pynt
WORKDIR ./glue
Build it using, e.g.:
docker build . -t glue
Then you can run command in it using for example the following syntax:
docker run --rm glue bash -c "mkdir a; ls -a; pwd"
I use --rm as I don't want to keep the container.
Hope this helps.
We cannot login to the docker container directly.
If you want to run any specific commands when the container start in detach mode than either you can give it in CMD and ENTRYPOINT command of the Dockerfile.
If you want to get into the shell directly, you can run
docker -it run imageName
or
docker run imageName bash -c "ls -ltr;pwd"
and it will return the output.
If you have triggered the run command without -it param then you can get into the container using:
docker exec -it imageName
and you will land up into the shell.
Now, if you are using AWS codebuild custom images and concerned about how the commands can be submitted to the container than you have to put your commands into the build_spec.yaml file and put your commands either in pre_build, build or post_build parameter and those commands will be submitted to the docker container.
-build_spec.yml
version: 0.2
phases:
pre_build:
commands:
- pip install boto3 #or any prebuild configuration
build:
commands:
- spark-submit job.py
post_build:
commands:
- rm -rf /tmp/*
More about build_spec here
I'm trying to use Elastic Beanstalk to deploy my Django server.
My problem is that part of my deployment process is to "npm install" from my package.json, and then executing webpack (npx webpack ..... --output main.js)
How can I do that while maintaining an easy deployment process (eb deploy) and without committing main.js to the repository?
To do it, you'll probably need ebextensions to configure your Elastic Beanstalk environment. Documentation is here.
I recently deploy my Symfony app on ElasticBeanstalk which needed Yarn to execute webpack.
To do it, I created a .config file in which I write the commands to install Yarn and another .config file to run Yarn at each deployment. All .config files are in .ebextensions directory at the root of the project.
commands:
01_install_node:
command: |
sudo curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -
sudo yum -y install nodejs
02_install_yarn:
command: |
sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo
sudo yum -y install yarn
You can use the container_commands key to execute commands that affect
your application source code. Container commands run after the
application and web server have been set up and the application
version archive has been extracted.
container_commands:
02_run_yarn:
command: |
yarn install
yarn run encore production
I have a django app that I will need to deploy on Amazon's EC2 Container Service. In the meantime, in order to test the deployment, I am trying to deploy it in a docker container locally first, but even when running a simple demo django application, I am unable to see the page at localhost:8000.
Here is my setup.
Create a docker machine:
$ docker-machine create --driver virtualbox testmachine
After this I set up my environment:
$ eval "$(docker-machine env testmachine)"
I set up a Dockerfile for my test container:
FROM ubuntu
RUN echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -sc) main universe" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install python-pip -y
RUN pip install django
RUN mkdir django_test
RUN cd django_test && \
django-admin.py startproject django_test .
Then I call
$ docker build -t dockertest .
... builds successfully
$ docker run -d -i -t -p 8000:8000 dockertest
cbef144ac068eb61b0c3e032448cc207c8f0384a9a67a710df6d9beb26d2ab32
$ docker attach cbef144ac068eb61b0c3e032448cc207c8f0384a9a67a710df6d9beb26d2ab32
root#cbef144ac068:/# cd django_test
root#cbef144ac068::/django_test# python manage.py runserver 0.0.0.0:8000
This successfully starts the server at 0.0.0.0:8000/ of the container.
However, when I try to go to localhost:8000 in my browser, I get a "This webpage is not available." What am I missing?
Turns out I was looking at the wrong IP.
To figure out the correct IP, I ran:
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
testmachine * virtualbox Running tcp://192.168.99.100:2376
I then loaded 192.168.99.100:8000 in my browser, and it worked like a charm.