I have a QuestDB container that's running an older version at the moment, it's started and stopped by --name so that the data is persisted:
docker run --name old_questdb \
-p 9000:9000 -p 9009:9009 questdb/questdb:5.0.5.4-linux-amd64
Is there any way to mount the volume or get the persisted data from this named container into a new instance with the latest version? I would like to run 6.0.4
You can copy the contents of a container to a local directory and then mount it back to a new one:
# copy the contents of the old_questdb container to the current dir
docker cp old_questdb:/root/.questdb $(pwd)
# run 6.0.4 and mount to the current dir
docker run --name new_questdb \
-v "$(pwd)/.questdb:/root/.questdb/" \
-p 9000:9000 -p 9009:9009 questdb/questdb:6.0.4
Related
I am trying to run docker image inside ec2 instance using gitlab CI/CD.
Trying to expose 5000 port for the application.
But i am aware of the face this job will work for the first time, but for the susequent runs the job will fail, as docker does not allow to run image on the same port, so i am trying to implement a fail safe mechanism where where running it checks for the process, if it exist, it will stop and remove container and then run the image on port 5000.
Here i am facing the problem that if this job runs for the first time docker stop needs at least one argument in the current command.
is there a way to run this command in a if condition basis, like if process exist then only run otherwise dont.
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script: ssh -o StrictHostKeyChecking=no -i $SSH_KEY ec2-user#ecxxxxx-xxxx.ap-southeast-1.compute.amazonaws.com "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
docker ps -aq | xargs docker stop | xargs docker rm &&
docker run -d -p 5000:5000 $IMAGE_NAME:$IMAGE_TAG"
error on pipeline
"docker stop" requires at least 1 argument.
See 'docker stop --help'.
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop one or more running containers
"docker rm" requires at least 1 argument.
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
The problem is with xargs docker stop | xargs docker rm command. is there a way to solve this kind of problem
Edit :- This doesn't exactly answer my question because what if a junior engineer is assigned this task to setup a pipeline who doesn't know the name of image, this solution requires us to know the name of the image, in this case this won't work.
Here what I understood is you are not stopping image but you are stopping container and removing it and then creating new container with the expose port 5000.
So give a variable constant name to container which will be same whenever it creates. The "|| true" helps you to stop the container only if it exists if not it won't stop any container
variables:
CONTAINER_NAME: <your-container-name> #please give a name for container to be created for this image
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script: ssh -o StrictHostKeyChecking=no -i $SSH_KEY ec2-user#ecxxxxx-xxxx.ap-southeast-1.compute.amazonaws.com "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS" &&
docker stop $CONTAINER_NAME; docker rm $CONTAINER_NAME || true &&
docker run -d -p 5000:5000 --name $CONTAINER_NAME $IMAGE_NAME:$IMAGE_TAG"
Primarily wanted to migrate postgress volume to another host, following this instruction https://docs.docker.com/storage/volumes/ on macos made a backup of volume using
docker run --rm --volumes-from a1df4bee1dc1 -v $(pwd):/backup debian:jessie tar cvf /backup/backup_db_floo.tar /var/lib/postgresql/data
And after deleting and recreating new container/volume and restoring via
docker run --rm --volumes-from docker-db-1 -v $(pwd):/backup debian:jessie bash -c "cd /var/lib/postgresql/data && tar xvf /backup/backup_db_floo.tar --strip 1"
no my data is there. Now realise that it was bad idea and i have to use pg_dumpall - but is it possible to recover somehow database from that postgres volume backup?
I want to copy a file from AWS S3 to a local directory through a docker container.
This copying command is easy without docker, I can see the file downloaded in the current directory.
But the problem is with docker that I don’t even know how to access the file.
Here is my Dockerfile:
FROM ubuntu
WORKDIR "/Users/ezzeldin/s3docker-test"
RUN apt-get update
RUN apt-get install -y awscli
ENV AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
CMD [ "aws", "s3", "cp", "s3://ezz-test/s3-test.py", "." ]
The current working folder that I should see the file downloaded to is s3docker-test/. This is what I'm doing after building the Dockerfile to mount a volume myvol to the local directory
docker run -d --name devtest3 -v $PWD:/var/lib/docker/volumes/myvol/_data ubuntu
So after running the image I get this:
download: s3://ezz-test/s3-test.py to ./s3-test.py
which shows that the file s3-test.py is already downloaded, but when I run ls in the interactive terminal I can't see it. So how can I access that file?
Looks like you are overriding containers folder with your empty folder, when you run -v $PWD:/var/lib/docker/volumes/myvol/_data.
Try to simply copy the files from container to host fs by running:
docker cp \
<containerId>:/Users/ezzeldin/s3docker-test/s3-test.py \
/host/path/target/s3-test.py
You could perform this command even on downed container. But first you will have to run it without folder override:
docker run -d --name devtest3 ubuntu
I am trying to use ECS for deployment with travis.
At one point everything was working but now it stopped.
I am following this tutorial https://testdriven.io/part-five-ec2-container-service/
There are 2 tasks that keep stopping and starting.
These are the messages I see in tasks:
STOPPED (CannotStartContainerError: API error (500): oci ru)
STOPPED (Essential container in task exited)
These are the messages I see in the logs:
FATAL: could not write to file "pg_wal/xlogtemp.28": No space left on device
container_linux.go:262: starting container process caused "exec: \"./entrypoint.sh\": permission denied"
Why is ECS stopping and starting so many new tasks? This was not happening before.
This is my docker_deploy.sh from my main microservice which I am calling via travis.
#!/bin/sh
if [ -z "$TRAVIS_PULL_REQUEST" ] || [ "$TRAVIS_PULL_REQUEST" == "false" ];
then
if [ "$TRAVIS_BRANCH" == "staging" ];
then
JQ="jq --raw-output --exit-status"
configure_aws_cli() {
aws --version
aws configure set default.region us-east-1
aws configure set default.output json
echo "AWS Configured!"
}
make_task_def() {
task_template=$(cat ecs_taskdefinition.json)
task_def=$(printf "$task_template" $AWS_ACCOUNT_ID $AWS_ACCOUNT_ID)
echo "$task_def"
}
register_definition() {
if revision=$(aws ecs register-task-definition --cli-input-json "$task_def" --family $family | $JQ '.taskDefinition.taskDefinitionArn');
then
echo "Revision: $revision"
else
echo "Failed to register task definition"
return 1
fi
}
deploy_cluster() {
family="testdriven-staging"
cluster="ezasdf-staging"
service="ezasdf-staging"
make_task_def
register_definition
if [[ $(aws ecs update-service --cluster $cluster --service $service --task-definition $revision | $JQ '.service.taskDefinition') != $revision ]];
then
echo "Error updating service."
return 1
fi
}
configure_aws_cli
deploy_cluster
fi
fi
This is my Dockerfile from my users microservice:
FROM python:3.6.2
# install environment dependencies
RUN apt-get update -yqq \
&& apt-get install -yqq --no-install-recommends \
netcat \
&& apt-get -q clean
# set working directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
# add requirements (to leverage Docker cache)
ADD ./requirements.txt /usr/src/app/requirements.txt
# install requirements
RUN pip install -r requirements.txt
# add entrypoint.sh
ADD ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
ADD . /usr/src/app
# run server
CMD ["./entrypoint.sh"]
entrypoint.sh:
#!/bin/sh
echo "Waiting for postgres..."
while ! nc -z users-db 5432;
do
sleep 0.1
done
echo "PostgreSQL started"
python manage.py recreate_db
python manage.py seed_db
gunicorn -b 0.0.0.0:5000 manage:app
I tried deleting my cluster and deregistering my tasks and restarting but ECS still continuously stops and starts new tasks now.
When it was working fine: the difference was that instead of the CMD ["./entrypoint.sh"] in my Dockerfile, I had
RUN python manage.py recreate_db
RUN python manage.py seed_db
CMD gunicorn -b 0.0.0.0:5000 manage:app
travis is passing.
The errors are right there.
You don't have enough space on your host; and the entrypoint.sh file is being denied.
Ensure your host has enough disk space (Shell in and df -h to check and expand the volume or just bring up a new instance with more space) and for the entrypoint.sh ensure that when building your image it is executable chmod +x and also is readable by the user the container is running as.
Test your containers locally first; the second error should have been caught in development instantly.
I realize this answer isn't 100% relevant to the question asked, but some googling brought me here due to the title and I figure my solution might help someone later down the line.
I also had this issue, but the reason why my containers kept restarting wasn't a lack of space or other resources, it was because I had enabled dynamic host port mapping and forgotten to update my security group as needed. What happened then is that the health checks my load balancer sent to my containers inevitably failed and ECS restarted the containers (whoops).
Dynamic Port Mapping in AWS Documentation:
https://aws.amazon.com/premiumsupport/knowledge-center/dynamic-port-mapping-ecs/
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PortMapping.html Contents --> hostPort
tl;dr - Make sure your load balancer can health check ports 32768 - 65535.
If it's too many tasks running and they have consumed the space then you will need to shell in to the host and do the following. Don't use -f on the docker rm as that will remove the running ECS agent container
docker rm $(docker ps -aq)
Do docker ps -a
Which results in all the stopped containers which are excited, these also consumes disk space.use below command to remove those zoombie
docker rm $(docker ps -a | grep Exited | awk '{print $1}')
And also remove older images or unused images these takes more DiskStation size than containers
docker rmi -f image_name
I'm trying to put my new ASP Website (MCV5) on the AWS server. For this I used EC2 and I created a virtual machine on linux. I also build my docker and now I'm trying to run it with:
sudo docker run -t -d -p 80:5004 myapp
But each time I just get a random number like this :
940e7abfe315d32cc8f5cfeb0c1f13750377fe091aa43b5b7ba4
When I try to know if my docker is running with:
sudo docker ps
no information is showed...
For information, when I put sudo docker images, I get my application created.
My Dockerfile contain:
FROM microsoft/aspnet
COPY . /app
WORKDIR /app
RUN ["dnu", "restore"]
EXPOSE 5004
ENTRYPOINT ["dnx", "-p", "project.json", "kestrel"]
-d means to run detached. The number you see is the container id.
If you want to get information about the container, you can use that container id
docker inspect <container id>
docker logs <container id>
If you'd like the container to run and print logs to the terminal, remove the -d