So I have a docker container with .txt and .csv files in it. I need to copy these to host. But I only need to copy files .txt files. The command sudo docker cp -a env1_1:/path/*.txt . does not seem to work. Is copying files of a specific types possible using docker cp? I am unable to find any alternatives for now.
Any suggestions?
Thanks.
Indeed, docker cp does not support glob patterns, and as an aside, container paths are absolute:
The docker cp command assumes container paths are relative to the container’s / (root) directory. This means supplying the initial forward slash is optional […]
Local machine paths can be an absolute or relative value. The command interprets a local machine’s relative paths as relative to the current working directory where docker cp is run.
However, one may devise a workaround, relying on docker exec, and a manually crafted shell command relying on the tar command on both sides (assuming it is available in the image):
sudo docker exec env1_1 sh -c 'cd /path && tar cf - *.txt' | tar xvf -
or if need be:
sudo docker exec env1_1 sh -c 'cd /path && tar cf - *.txt' | ( cd /dest/path && tar xvf - )
Here, the special filename - denotes STDOUT (or respectively STDIN).
Usual disclaimer: the final tar command will overwrite without further notice the selected files in the current folder (or /dest/path in the second example).
Related
This is my 3rd day of tear-your-hair-out since the weekend and I just cannot get ENTRYPOINT to work via gitlab runner 13.3.1, this for something that previously worked with a simple ENTRYPOINT ["/bin/bash"] but that was using local docker desktop and using docker run followed by docker exec commands which worked like a synch. Essentially, at the end of it all I previously got a WAR file built.
Currently I build my container in gitlab runner 13.3.1 and push to s3 bucket and then use the IMAGE:localhost:500/my-recently-builtcontainer and then try and do whatever it is I want with the container but I cannot even get ENTRYPOINT to work, in it's exec form or in shell form - atleast in the shell form I get to see something. In the exec form it just gave "OCI runtime create failed" opaque errors so I shifted to the shell form just to see where I could get to.
I keep getting
sh: 1: sh: echo HOME=/home/nonroot-user params=#$ pwd=/ whoami=nonroot-user script=sh ENTRYPOINT reached which_sh=/bin/sh which_bash=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin; ls -alrth /bin/bash; ls -alrth /bin/sh; /usr/local/bin/entrypoint.sh ;: not found
In my Dockerfile I distinctly have
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN bash -c "ls -larth /usr/local/bin/entrypoint.sh"
ENTRYPOINT "echo HOME=${HOME} params=#$ pwd=`pwd` whoami=`whoami` script=${0} ENTRYPOINT reached which_sh=`which sh` which_bash=`which bash` PATH=${PATH}; ls -alrth `which bash`; ls -alrth `which sh`; /usr/local/bin/lse-entrypoint.sh ;"
The output after I build the container in gitlab is - and I made sure anyone has rights to see this file and use it - just so that I can proceed with my work
-rwxrwxrwx 1 root root 512 Apr 11 17:40 /usr/local/bin/entrypoint.sh
So, I know it is there and all the chmod flags indicate anybody can look at it - so I am so perplexed why it is saying NOT FOUND
/usr/local/bin/entrypoint.sh ;: not found
entrypoint.sh is ...
#!/bin/sh
export PATH=$PATH:/usr/local/bin/
clear
echo Script is $0
echo numOfArgs is $#
echo paramtrsPassd is $#
echo whoami is `whoami`
bash --version
echo "About to exec ....."
exec "$#"
It does not even reach inside this entrypoint.sh file.
given a config file that has new line delimitted a set of folders (cannot use complete list of dirs (TOO LARGE)) in Google Cloud Storage as follows:
gs://databucket/path/to/dir/441738
gs://databucket/path/to/dir/441739
gs://databucket/path/to/dir/441740
how can one use gsutil inside a bash script to recursively rsync the files, whilst deleting files present in the destination folder that don't exist on the bucket?
I have tried using the following in a bash script
cat ${1} | gsutil -m rsync -r -d ${2}
after which I receive an error code 126
whereby ${1} references the aforementioned config file and ${2} references the destination folder to which each folder in the config file list is to be rsynced.
This works with gsutil cp however rsync more efficiently/effectively suits my needs.
cat ${1} | gsutil -m cp -R -I ${2}
How might one accomplish this?
Thanks
As you know, rsync does not support function uses stdin like -I flag...
So you have to use a different method than cp.
If you want synchronize multiple folders in a single command, Write batch script that has rsync command each line like below.
gsutil -m rsync -r -d gs://databucket/path/to/dir/441738 *destination_folder1*
gsutil -m rsync -r -d gs://databucket/path/to/dir/441739 *destination_folder2*
gsutil -m rsync -r -d gs://databucket/path/to/dir/441740 *destination_folder3*
And run a script file you wrote.
This method is a bit bothersome, but it can work same result you want.
I follow, verbatim, the instructions given, and I get an error almost like some utterly unrelated program is being invoked. For the record, it seemed to be working yesterday.
I am running this on Amazon linux 2:
sudo yum update
sudo yum -y install ruby wget
cd /home/ec2-user
wget https://aws-codedeploy-${AWS::Region}.s3.${AWS::Region}.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto
sudo service codedeploy-agent status
and here is what happens:
[root#ip-10-204-84-134 bin]# sudo ./install auto
./install: missing destination file operand after ‘auto’
Try './install --help' for more information.
[root#ip-10-204-84-134 bin]# sudo ./install --help
Usage: ./install [OPTION]... [-T] SOURCE DEST
or: ./install [OPTION]... SOURCE... DIRECTORY
or: ./install [OPTION]... -t DIRECTORY SOURCE...
or: ./install [OPTION]... -d DIRECTORY...
This install program copies files (often just compiled) into destination
locations you choose. If you want to download and install a ready-to-use
package on a GNU/Linux system, you should instead be using a package manager
like yum(1) or apt-get(1).
In the first three forms, copy SOURCE to DEST or multiple SOURCE(s) to
the existing DIRECTORY, while setting permission modes and owner/group.
In the 4th form, create all components of the given DIRECTORY(ies).
Mandatory arguments to long options are mandatory for short options too.
--backup[=CONTROL] make a backup of each existing destination file
-b like --backup but does not accept an argument
-c (ignored)
-C, --compare compare each pair of source and destination files, and
in some cases, do not modify the destination at all
-d, --directory treat all arguments as directory names; create all
components of the specified directories
-D create all leading components of DEST except the last,
then copy SOURCE to DEST
-g, --group=GROUP set group ownership, instead of process' current group
-m, --mode=MODE set permission mode (as in chmod), instead of rwxr-xr-x
-o, --owner=OWNER set ownership (super-user only)
-p, --preserve-timestamps apply access/modification times of SOURCE files
to corresponding destination files
-s, --strip strip symbol tables
--strip-program=PROGRAM program used to strip binaries
-S, --suffix=SUFFIX override the usual backup suffix
-t, --target-directory=DIRECTORY copy all SOURCE arguments into DIRECTORY
-T, --no-target-directory treat DEST as a normal file
-v, --verbose print the name of each directory as it is created
-P, --preserve-context preserve SELinux security context (-P deprecated)
-Z set SELinux security context of destination
file to default type
--context[=CTX] like -Z, or if CTX is specified then set the
SELinux or SMACK security context to CTX
--help display this help and exit
--version output version information and exit
The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.
The version control method may be selected via the --backup option or through
the VERSION_CONTROL environment variable. Here are the values:
none, off never make backups (even if --backup is given)
numbered, t make numbered backups
existing, nil numbered if numbered backups exist, simple otherwise
simple, never always make simple backups
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Report install translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'install invocation'
Ok, The documentation is totally broken, but upon just looking in that bucket I found an RPM and ran it:
[root#ip-10-204-84-134 bin]# sudo yum install -y https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/latest/codedeploy-agent.noarch.rpm^C
[root#ip-10-204-84-134 bin]# sudo service codedeploy-agent status
The AWS CodeDeploy agent is running as PID 4572
[root#ip-10-204-84-134 bin]#
I want to copy a file from AWS S3 to a local directory through a docker container.
This copying command is easy without docker, I can see the file downloaded in the current directory.
But the problem is with docker that I don’t even know how to access the file.
Here is my Dockerfile:
FROM ubuntu
WORKDIR "/Users/ezzeldin/s3docker-test"
RUN apt-get update
RUN apt-get install -y awscli
ENV AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
CMD [ "aws", "s3", "cp", "s3://ezz-test/s3-test.py", "." ]
The current working folder that I should see the file downloaded to is s3docker-test/. This is what I'm doing after building the Dockerfile to mount a volume myvol to the local directory
docker run -d --name devtest3 -v $PWD:/var/lib/docker/volumes/myvol/_data ubuntu
So after running the image I get this:
download: s3://ezz-test/s3-test.py to ./s3-test.py
which shows that the file s3-test.py is already downloaded, but when I run ls in the interactive terminal I can't see it. So how can I access that file?
Looks like you are overriding containers folder with your empty folder, when you run -v $PWD:/var/lib/docker/volumes/myvol/_data.
Try to simply copy the files from container to host fs by running:
docker cp \
<containerId>:/Users/ezzeldin/s3docker-test/s3-test.py \
/host/path/target/s3-test.py
You could perform this command even on downed container. But first you will have to run it without folder override:
docker run -d --name devtest3 ubuntu
I have a dockerfile for running a c++ application. Part of the docker file has the following command :
RUN tar -xvf boost_1_56_0.tar.bz2 && \
cd boost_1_56_0 && \
./bootstrap.sh && \
./b2 install
The tar file is part of docker image.
The problem is that each time I build the dockerfile the entire package gets installed which takes an awful amount of time. How can I prevent it ?
If nothing has changed up to and including a command in a docker file, then Docker will used the cached data from a previous build. So if you have something like this:
ADD ./myfiles /path/in/container # changes each time
RUN tar -xvf boost # etc
Then boost will be rebuilt every time. But if reorganise your Dockerfile like this:
RUN tar -xvf boost # etc
ADD ./myfiles /path/in/container # changes each time
Then the binary build of boost from your last docker build will be reused from the cache. More generally, put things earlier in the Docker file the less likely they are to change.