I'm trying to rename my file toto.html by removing its extension using gsutils. My folder/files are organized this way:
toto ----|
|
|--file1.html
|
|--file2.txt
|
|-- ...
|
toto.html
So I used the command :
gsutil mv gs://my_bucket/toto.html gs://my_bucket/toto
the problem is by doing so, the file is moved inside the folder that has the same name as the directory. ( which is normal )
How can I rename my file without having this issue?
thanks in advance.
Let me explain briefly about how directories work on Google Cloud Storage, as detailed in the documentation:
Cloud Storage operates with a flat namespace, which means that folders don't actually exist within Cloud Storage. If you create an object named folder1/file.txt in the bucket your-bucket, the path to the object is your-bucket/folder1/file.txt, but there is no folder named folder1; instead, the string folder1 is part of the object's name.
So when you perform the move command gsutil mv gs://my_bucket/toto.html gs://my_bucket/toto, gsutil interprets it like you would like to move the file inside the folder named toto as there is no way to differentiate between the file and the folder.
What you could do to achieve the task is move the file inside a folder first to change its name and then get it back to root:
gsutil mv gs://my_bucket/toto.html gs://my_bucket/toto/toto
gsutil mv gs://my_bucket/toto/toto gs://my_bucket/
Related
I have a gcp bucket say gs://my-folder that contains folders that either start with newVersion or don't. Each of these folders contain files. For example I have gs://my-folder/newVersion=1, gs://my-folder/newVersion=2, gs://my-folder/newVersion=3. I want to copy whatever is in these folders into gs://my-folder/1, gs://my-folder/2 and gs://my-folder/3 respectively. Is there a way of using gsutil cp command to do this? It would have much been easier if the these were different folders.
I don't want to copy folders individually. I want a script maybe that could identify all the newVersion= folders and copy them to the corresponding folders.
If it doesn't have to be a copy then just:
Renaming Groups Of Objects
You can use the gsutil mv command to rename all objects with a given prefix to have a new prefix. For example, the following command renames all objects under gs://my_bucket/oldprefix to be under gs://my_bucket/newprefix, otherwise preserving the naming structure:
gsutil mv gs://my_bucket/oldprefix gs://my_bucket/newprefix
Note that when using mv to rename groups of objects with a common prefix, you cannot specify the source URL using wildcards; you must spell out the complete name.
If you do a rename as specified above and you want to preserve ACLs, you should use the -p option (see OPTIONS).
If you have a large number of files to move you might want to use the gsutil -m option, to perform a multi-threaded/multi-processing move:
gsutil -m mv gs://my_bucket/oldprefix gs://my_bucket/newprefix
ref: https://cloud.google.com/storage/docs/gsutil/commands/mv
You can simply use the gsutil cp command if you want to copy a folder and its content.
gsutil cp gs://my-folder/newVersion=1/* gs://my-folder/1
To copy the entire directory tree, you can use -r command.
gsutil cp -r gs://my-folder/newVersion=1/* gs://my-folder/1
However, note that this will only work if the folder to copy contains files. Since it expects actual objects/files to be copied and not empty directories.
On the other hand, using gsutil mv command will allow you to perform a copy from source to destination followed by removing the source for each object. If you don't want your original file/object to be removed, you can use the cp instead of mv.
So there was a bug in our recent code, and instead of creating bunch of numbered folders in a root folder, it created bunch of folders right next to root folder
To put it simply, what we wanted was:
customers
|-18
|-158
|-405
|-1238
|-1447
...
|-4797
Unfortunately, what we got was:
customers
customers18
customers158
customers405
customers1238
customers1447
..
customers4797
Now, there are about 1000 folders (with their internal sub-folder structure) that we need to delete. I tried to look up regex and other filtering method, but it seems like they don't work on rm command.
What is the command line I need to delete all the "customers[numbers]" folders but NOT the "customers" folder?
Try the following command. It should work -
ls | grep -P "customers[0-9]+" | xargs -d"\n" rm -R
I have folder A and folder B
Folder A contains approx 100 files all text, js, php, bash etc. They are stored in the root of the folder and sub folders and further sub folders within folder A.
Folder B is a copy of Folder A, but some of the files have been updated.
Is there any way I can compare A to B and create a tar.gz file containing only the files that have changed in Folder B
I would need to keep the folder structure intact when the tar.gz is created.
Currently I use WinMerge to check for differences, but I'm happy to look at any windows or Linux application/commands that will help with this.
Thanks
This line excludes files that are only in one or the other, but creates the tar.gz file that you want.
diff -rq folderA folderB | grep -v "^Only in" | sed "s/^.*and folderB/folderB/g" | sed "s/ differ//g" | tar czf output.tar.gz -T -
Broken down it goes:
dif -rq folderA folderB
Do a recursive diff between these folders, be quiet about it - only output the file names.
| grep -v "^Only in"
Exclude output lines that indicate one file is only in one of the folders. I'm assuming from your description this isn't an issue for you, but the two folders I was playing with were a bit dirty.
| sed "s/^.*and folderB/folderB/g"
Discard the first bit of the output up until it says " and " and then the name of the second folder. This actually takes away the second folder name as well, but then replaces it back in
| sed "s/ differ//g"
Discard the end bit of the diff output.
| tar czf output.tar.gz -T -
Tell tar to do the thing. c == create a tar file z means compress it (zip) f means the filename is coming shortly. output.tar.gz is your output file -T means "get the filenames from the file I'm about to tell you" the final - means "use stdin instead"
I suggest you build this up yourself in the individual steps so you can see how it is constructed, and what the output of each step is like.
I'm trying to implement cp -r command so that when user types in cp -r dir dir1, dir gets copied and pasted inside dir1. Below is what i have so far and it does copy the files and directories inside the directory, but it doesn't copy directory itself. For example, when there is a file1 and a directory 'a'inside dir then it will only copy and paste file1 and 'a' inside of dir1, but not the directory dir itself.
Any suggestions?
You start your algorithm one step too deep into the directory you are copying: dir_entry = readdir(dir) reads all the entrys inside your source directory and therefore the source directory itself is not copied.
The function is fine, you just need one extra step before you call it.
Instead of calling dirCopy("a", "b") you need to start by executing
mkdir("b/a", convertMode("a"));
and then
dirCopy("a", "b/a");
So you are going to need code that extracts the last filename part from your path so you can then append it to newPath. If you need assistance for that you can look at this question: Get a file name from a path
I want to replace folder A in P4 by another folder A.
The two folders have different files and sub folders.
I know, we can do it by deleting old folder A then adding new folder A.
But, can I do it with only one step in a pending changelist ?
As following result in that pending cl:
If this file is in old folder, but not in new folder, then it is marked by "delete".
If this file is in new folder, but not in old folder, then it is marked by "add".
If this file is in new folder and also in old folder, then it is marked by "modify".
Thank you
Are both these folders under source control?
That is, are you trying to make //depot/folder/A contain what //depot/other/A_prime contains?
If so, consider using 'p4 copy':
p4 copy //depot/other/A_prime/... //depot/folder/A/...
If the other folder A is just something you have on your hard disk, then consider using 'reconcile':
p4 edit //depot/folder/A/...
rm -r /path/to/depot/folder/A/*
cp -r /path/to/other/folder/A/* /path/to/depot/folder/A
p4 reconcile -aed //depot/folder/A/...
I kind of like the 'p4 copy' approach, myself, so I'd be tempted to check that other folder into Perforce (in a different location in the repository, naturally), so that I could then run 'p4 copy'.