Using gustil, I want to delete all objects in a bucket. But I do not want to delete the bucket. The job takes about 2 minutes and I would like to speed it up with the -m option for multi threading.
However, when I try gsutil -m rm -a gs://dls-qa/** I get 'CommandException: Incorrect option(s) specified'. Am I not allowed to use -m with the double asterisk ** which is required to keep the bucket?
https://cloud.google.com/storage/docs/gsutil/commands/rm
I tried the same command and it worked in my case :
gsutil -m rm -a "gs://mazlum_test/**"
I added " chars in my bucket pattern because I am using omyzsh
Maybe you have to upgrade your gcloud sdk version.
Related
To move a medium sized blob (>250mb) from one place to another in GCS (gsutil cp my_blob my_new_blob),
gsutil wants me to compose it :
so I am doing gsutil compose my_blob my_blob to compose it and overcome this error, but I then get another error:
where it would just retry again and again and I would finally get a
503 - We encountered an internal error - Try again
error.
Why is this happenning ? Is there a limit to the size of the file to be composed and why this limit would be only 250mb ?
Trid it on my end using this docs Cloud Storage cp options.
$ gsutil -o "GSUtil:max_upload_compression_buffer_size=8G" -m cp -J filetest filtest_new
Copying file://filetest...
/ [1/1 files][300.0 MiB/300.0 MiB] 100% Done
Operation completed over 1 objects/300.0 MiB.
Tried to simplify it, same Chaotic comments with slight changes
gsutil -m cp filetest filtest_new
XXXXX#cloudshell:~ (XXXXX)$ gsutil -m cp filetest filtest_new2
Copying file://filetest...
/ [1/1 files][300.0 MiB/300.0 MiB] 100% Done
Operation completed over 1 objects/300.0 MiB.
I'm trying to download an entire bucket to my local machine.
I'm aware that the command to do this is:
gsutil -m cp -r \ "gs://bucket_name/folder_name/" \ .
However, I'd like to specify exactly where this gets downloaded on my machine due to storage limitations.
Can anyone share any advice regarding this?
Thanks in advance,
Tommy
You can place your download files where you want to put by adding a destination url value to the last parameter of the gsutil cp command you are using, for example:
gsutil -m cp -r "gs://bucket_name" "D:\destination_folder"
I have a problem downloading entire folder in GCP. How should I download the whole bucket? I run this code in GCP Shell Environment:
gsutil -m cp -R gs://my-uniquename-bucket ./C:\Users\Myname\Desktop\Bucket
and I get an error message: "CommandException: Destination URL must name a directory, bucket, or bucket subdirectory for the multiple source form of the cp command. CommandException: 7 files/objects could not be transferred."
Could someone please point out the mistake in the code line?
To download an entire bucket You must install google cloud SDK
then run this command
gsutil -m cp -R gs://project-bucket-name path/to/local
where path/to/local is your path of local storage of your machine
The error lies within the destination URL as specified by the error message.
I run this code in GCP Shell Environment
Remember that you are running the command from the Cloud Shell and not in a local terminal or Windows Command Line. Thus, it is throwing that error because it cannot find the path you specified. If you inspect the Cloud Shell's file system/structure, it resembles more that of a Unix environment in which you can specify the destination like such instead: ~/bucketfiles/. Even a simple gsutil -m cp -R gs://bucket-name.appspot.com ./ will work since Cloud Shell can identify the ./ directory which is the current directory.
A workaround to this issue is to perform the command on your Windows Command Line. You would have to install Google Cloud SDK beforehand.
Alternatively, this can also be done in Cloud Shell, albeit with an extra step:
Download the bucket objects by running gsutil -m cp -R gs://bucket-name ~/ which will download it into the home directory in Cloud Shell
Transfer the files downloaded in the ~/ (home) directory from Cloud Shell to the local machine either through the User Interface or by running gcloud alpha cloud-shell scp
Your destination path is invalid:
./C:\Users\Myname\Desktop\Bucket
Change to:
/Users/Myname/Desktop/Bucket
C: is a reserved device name. You cannot specify reserved device names in a relative path. ./C: is not valid.
There is not a one-button solution for downloading a full bucket to your local machine through the Cloud Shell.
The best option for an environment like yours (only using the Cloud Shell interface, without gcloud installed on your local system), is to follow a series of steps:
Downloading the whole bucket on the Cloud Shell environment
Zip the contents of the bucket
Upload the zipped file
Download the file through the browser
Clean up:
Delete the local files (local in the context of the Cloud Shell)
Delete the zipped bucket file
Unzip the bucket locally
This has the advantage of only having to download a single file on your local machine.
This might seem a lot of steps for a non-developer, but it's actually pretty simple:
First, run this on the Cloud Shell:
mkdir /tmp/bucket-contents/
gsutil -m cp -R gs://my-uniquename-bucket /tmp/bucket-contents/
pushd /tmp/bucket-contents/
zip -r /tmp/zipped-bucket.zip .
popd
gsutil cp /tmp/zipped-bucket.zip gs://my-uniquename-bucket/zipped-bucket.zip
Then, download the zipped file through this link: https://storage.cloud.google.com/my-uniquename-bucket/zipped-bucket.zip
Finally, clean up:
rm -rf /tmp/bucket-contents
rm /tmp/zipped-bucket.zip
gsutil rm gs://my-uniquename-bucket/zipped-bucket.zip
After these steps, you'll have a zipped-bucket.zip file in your local system that you can unzip with the tool of your choice.
Note that this might not work if you have too much data in your bucket and the Cloud Shell environment can't store all the data, but you could repeat the same steps on folders instead of buckets to have a manageable size.
We are using automation scripts to upload thousands of files from MAPR HDFS to GCP storage. Sometimes the files in the main bucket appear with tmp~!# suffix it causes failures in our pipeline.
Example:
gs://some_path/.pre-processing/file_name.gz.tmp~!#
We are using rsync -m and in certain cases cp -I
some_file | gsutil -m cp -I '{GCP_DESTINATION}'
gsutil -m rsync {MAPR_SOURCE} '{GCP_DESTINATION}'
It's possible that copy attempt failed and retried later from a different machine, eventually, we have both the file and another one with the tmp~!# suffix
I'd want to get rid of these files without actively looking for them.
we have gsutil 4.33, appreciate any lead. Thx
I am trying to copy files out to a network directory during a build, and I keep getting a " No such file or directory" error message.
Copying to local drive works fine:
cp -Rf c:/Jenkins/deployments/TW_ISSUE_A/src c:/Jenkins/deployments/TW_ISSUE_A/target
The following all throw the same message:
cp -Rf c:/Jenkins/deployments/TW_ISSUE_A/src H:/some_dir
cp -Rf c:/Jenkins/deployments/TW_ISSUE_A/src H:\some_dir
cp -Rf c:/Jenkins/deployments/TW_ISSUE_A/src //Hubbell/MISGenl/some_dir
cd c:/Jenkins/deployments/TW_ISSUE_A/src
rsync -avuzb //Hubbell/MISGenl/Projects/Tronweb/TronwebBuilds/test/ora/sql/
cp -Rf c:/Jenkins/deployments/TW_ISSUE_A/src /cygdrive/h/some_dir
I've even created a shell script to call from Jenkins, but I continue to receive that message.
#!/bin/bash
url="http://as-test02:8080/job/TW_ISSUE_A_BUILD/lastSuccessfulBuild/artifact/bui
ld-TW_ISSUE_A_BUILD.tar";
remote_stage_dir="/cygdrive/h/some_dir"
#fetch the artifacts
(cd "$remote_stage_dir" && wget "$url" && tar xvf build-TW_ISSUE_A_BUILD.tar dat
java ora && rm -rf *.tar && cp -r ./ora/* ../INTEGRATION)
Is there any way to copy files out to a mapped drive on the build machine?
Thank you!!
I would guess the mapped drive isn't available in the services context, or that the user executing Jenkins doesn't have access to it. What user is Jenkins running as?
Edit:
I think your problem has two aspects:
The user running the Jenkins service isn't allowed to connect to the network.
h: isn't known to the user.
If you haven't modified it, the service is most likely running under the LocalSystem account. You can modify this by running services.msc (or navigate to services via the Windows control panel) and locating the jenkins service. This should resolve the first problem.
The second problem can be resolved by using UNC paths (as you tried above) instead of network drives.
The Jenkins wiki has an article about problems like this: My software builds on my computer but not on Jenkins