Semi-complex restructure of directories in linux - regex

I have a bunch of directories that I need to restructure. They're in a format as such:
./1993-02-22 - The Moon - Tallahassee, FL/**files**
./1993-02-23 - The Moon - Tallahassee, FL/**files**
./1993-02-24 - The Moon - Tallahassee, FL/**files**
./1993-02-25 - The Moon - Tallahassee, FL/**files**
./1993-03-01 - The Test - Null, FL/**files**
I want to extract the dates from the beginning of each folder. For example, in regex: ([0-9]{4})\-([0-9]{1,2})\-([0-9]{1,2}) and reformat the directories to ./year/month/day.
So, it should output:
./1993/02/22/**files**
./1993/02/23/**files**
./1993/02/24/**files**
./1993/02/25/**files**
./1993/03/01/**files**
How can I go about doing that from a command line?

I understood the question in a different way than Kent. I thought that you wanted to create a new tree of directories from each original one and move all files that it contained. You could try following perl script if that was what you were looking for:
perl -MFile::Path=make_path -MFile::Copy=move -e '
for ( grep { -d } #ARGV ) {
#date = m/\A(\d{4})-(\d{2})-(\d{2})/;
next unless #date;
$outdir = join q{/}, #date;
make_path( $outdir );
move( $_, $outdir );
}
' *
It reads every file from current directory (* passed as argument) and does two step filter. The first one is the grep for no-directories files and the second one is an undefined #date for those that don't begin with it. Then it joins date's components into a path, creates it if doesn't exist and move the old one with all its files to the new one.
A test:
Here the result of ls -lR to show initial state:
.:
total 24
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 1993-02-22 - The Moon - Tallahassee, FL
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 1993-02-23 - The Moon - Tallahassee, FL
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 1993-02-24 - The Moon - Tallahassee, FL
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:57 1993-02-25 - The Moon - Tallahassee, FL
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:57 1993-03-01 - The Test - Null, FL
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:47 dummy_dir
-rw-r--r-- 1 dcg dcg 0 sep 7 00:47 dummy_file
./1993-02-22 - The Moon - Tallahassee, FL:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file1
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file2
./1993-02-23 - The Moon - Tallahassee, FL:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file3
./1993-02-24 - The Moon - Tallahassee, FL:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file4
./1993-02-25 - The Moon - Tallahassee, FL:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file5
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file6
./1993-03-01 - The Test - Null, FL:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file7
./dummy_dir:
total 0
And after running the previous script note that the base directory only keeps dummy files and the root of the tree created (1993). Running the same ls -lR yields:
.:
total 8
drwxr-xr-x 4 dcg dcg 4096 sep 7 00:59 1993
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:47 dummy_dir
-rw-r--r-- 1 dcg dcg 0 sep 7 00:47 dummy_file
./1993:
total 8
drwxr-xr-x 6 dcg dcg 4096 sep 7 00:59 02
drwxr-xr-x 3 dcg dcg 4096 sep 7 00:59 03
./1993/02:
total 16
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 22
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 23
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:56 24
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:57 25
./1993/02/22:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file1
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file2
./1993/02/23:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file3
./1993/02/24:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:56 file4
./1993/02/25:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file5
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file6
./1993/03:
total 4
drwxr-xr-x 2 dcg dcg 4096 sep 7 00:57 01
./1993/03/01:
total 0
-rw-r--r-- 1 dcg dcg 0 sep 7 00:57 file7
./dummy_dir:
total 0

Suppose your files are stored in "old" folder, then you may write a shell script (avoid using "for" loop that has difficulties with file names containing spaces):
mkdir -p new
ls -d -1 old/*/* | while read oldfile; do
newfile=`echo "$oldfile" | sed -r 's#^old/([0-9]{4})\-([0-9]{1,2})\-([0-9]{1,2})(.*)$#new/\1/\2/\3/\4#'`
newdir=` echo $newfile | sed 's#/[^/]*$##'`
echo "Creating \"$newdir\""
mkdir -p "$newdir"
echo "Moving files from \"$oldfile\" to \"$newfile\""
cp -r "$oldfile" "$newfile"
done
output of script:
Creating "new/1993/02/22/ - The Moon - Tallahassee, FL"
Moving files from "old/1993-02-22 - The Moon - Tallahassee, FL/test" to "new/1993/02/22/ - The Moon - Tallahassee, FL/test"
Creating "new/1993/02/23/ - The Moon - Tallahassee, FL"
Moving files from "old/1993-02-23 - The Moon - Tallahassee, FL/test" to "new/1993/02/23/ - The Moon - Tallahassee, FL/test"
Creating "new/1993/02/24/ - The Moon - Tallahassee, FL"
Moving files from "old/1993-02-24 - The Moon - Tallahassee, FL/test" to "new/1993/02/24/ - The Moon - Tallahassee, FL/test"
Creating "new/1993/02/25/ - The Moon - Tallahassee, FL"
Moving files from "old/1993-02-25 - The Moon - Tallahassee, FL/test" to "new/1993/02/25/ - The Moon - Tallahassee, FL/test"
Creating "new/1993/03/01/ - The Tes - Null, FL"
Moving files from "old/1993-03-01 - The Tes - Null, FL/test2" to "new/1993/03/01/ - The Tes - Null, FL/test2"
and you'll find your new tree in ... the "new" folder indeed:
$ tree old new
old
├── 1993-02-22 - The Moon - Tallahassee, FL
│   └── test
├── 1993-02-23 - The Moon - Tallahassee, FL
│   └── test
├── 1993-02-24 - The Moon - Tallahassee, FL
│   └── test
├── 1993-02-25 - The Moon - Tallahassee, FL
│   └── test
└── 1993-03-01 - The Tes - Null, FL
└── test2
new
└── 1993
├── 02
│   ├── 22
│   │   └── - The Moon - Tallahassee, FL
│   │   └── test
│   ├── 23
│   │   └── - The Moon - Tallahassee, FL
│   │   └── test
│   ├── 24
│   │   └── - The Moon - Tallahassee, FL
│   │   └── test
│   └── 25
│   └── - The Moon - Tallahassee, FL
│   └── test
└── 03
└── 01
└── - The Tes - Null, FL
└── test2

like this?
kent$ awk '{gsub(/-/,"/",$1);sub(/^[^/]*\//,"/",$NF);print $1$NF}' file
./1993/02/22/**files**
./1993/02/23/**files**
./1993/02/24/**files**
./1993/02/25/**files**
./1993/03/01/**files**

Here is a sed+awk after so much of comments :
awk 'BEGIN{FS="-"}{print $1"/"$2"/"$3}' file | awk 'BEGIN{FS="**files**"}{print $1"/"FS}' | sed -e 's/ \//\//g'
./1993/02/22/**files**
./1993/02/23/**files**
./1993/02/24/**files**
./1993/02/25/**files**
./1993/03/01/**files**

Related

When attemping to copy a dir from repo to container in workflow, it's as if COPY command did nothing

My repo:
/
dbt-action/
action.yml
Dockerfile
entrypoint.sh
dbt/
profiles.yml
My workflow step:
- name: Run DBT
uses: ./dbt-action
My Dockerfile:
FROM ghcr.io/dbt-labs/dbt-redshift:1.3.latest
COPY dbt .dbt
COPY entrypoint.sh /entrypoint.sh
My entrypoint:
!/bin/bash
pwd
ls -la
Outputs the following:
drwxr-xr-x 6 1001 123 4096 Jan 7 13:06 .
drwxr-xr-x 6 root root 4096 Jan 7 13:06 ..
drwxr-xr-x 8 1001 123 4096 Jan 7 13:06 .git
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 .github
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 blah
-rw-r--r-- 1 1001 123 1744 Jan 7 13:06 README.md
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 dbt-action
Expected output:
Same as above but with additional directory .dbt coming from COPY dbt .dbt in my Dockerfile.
Why don't I see dir .dbt when I ls -la in my entrypoint?
Seems like you are executing your ‘Docker build’ from the wrong working directory, since the ‘dbt-action’ folder is present, but not it contents. Can you double check the PWD before you build?

How to link an external library in Qt Creator when LIBS += -L"..." doesn't seem to work?

I've been trying to use the Intel Open Image Denoise library in my code but still have no luck as of now. Apparently it seems that I am not linking the library properly.
Here's what I've done so far: after compiling OIDN I moved the oidn directory to the directory where my path tracer code is and did the following in my code (C++11) which I'm working on in Qt Creator.
#include "oidn/include/OpenImageDenoise/oidn.hpp"
.
.
.
oidn::DeviceRef device = oidn::newDevice(); // this is where things break
device.commit();
Note that doing #include "oidn/include/OpenImageDenoise/oidn.hpp" seems to work fine because autocomplete works for oidn and shows its members. However I get the following errors which seems to be due to not linking oidn libraries.
I tried linking the libraries via LIBS += -L"oidn/build/" or LIBS += -L"oidn/build/libOpenImageDenoise.so" (as shown in the image) but I get the same errors. I also appended -loidn but got an error saying "cannot find -loidn". I am now pretty confused on what I'm supposed to do. I'm not even sure if the errors are due to not properly linking libraries because I see people generally link libraries in C++ the same way I am trying to do it here. I would really appreciate if someone can help me figure out what I need to do to get OIDN work.
All the previous answers/comments were correct. The weird thing about my case was the symlinks of the libraries in the build directory were not valid and were not pointing to the right files. The output of ls -lh /home/Warrior/Desktop/pathtracer/oidn/build was as follow:
total 160K
drwxrwxr-x 4 amir amir 4.0K Feb 4 00:09 apps
-rw-rw-r-- 1 amir amir 19K Feb 4 00:09 CMakeCache.txt
drwxrwxr-x 7 amir amir 4.0K Feb 4 00:09 CMakeFiles
-rw-rw-r-- 1 amir amir 6.5K Feb 4 00:09 cmake_install.cmake
drwxrwxr-x 3 amir amir 4.0K Feb 4 00:09 common
-rw-r--r-- 1 amir amir 3.6K Feb 4 00:09 CPackConfig.cmake
-rw-r--r-- 1 amir amir 4.0K Feb 4 00:09 CPackSourceConfig.cmake
-rw-rw-r-- 1 amir amir 103K Feb 4 00:09 Makefile
drwxrwxr-x 3 amir amir 4.0K Feb 4 00:08 mkl-dnn
-rw-r--r-- 1 amir amir 1.6K Feb 4 00:09 OpenImageDenoiseConfigVersion.cmake
But when I would go to the directory I would see the following:
What I did was instead of cloning the oidn repo somewhere else and building it, I cloned the oidn repo in /home/Warrior/Desktop/pathtracer/ and built it there. After that I could see the correct file names/symlinks via ls -lh as follow. Now my program compiles and things work fine. I just had to use LIBS += -L"oidn/build/" -lOpenImageDenoise when compiling my code:
total 51M
drwxrwxr-x 4 amir amir 4.0K Feb 9 10:14 apps
-rw-rw-r-- 1 amir amir 19K Feb 9 10:14 CMakeCache.txt
drwxrwxr-x 7 amir amir 4.0K Feb 9 10:19 CMakeFiles
-rw-rw-r-- 1 amir amir 5.6K Feb 9 10:14 cmake_install.cmake
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 color_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 color_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb 9 10:17 color_ispc.h
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 color_ispc_sse4.h
drwxrwxr-x 3 amir amir 4.0K Feb 9 10:14 common
-rw-r--r-- 1 amir amir 3.3K Feb 9 10:14 CPackConfig.cmake
-rw-r--r-- 1 amir amir 3.7K Feb 9 10:14 CPackSourceConfig.cmake
-rw-rw-r-- 1 amir amir 2.6K Feb 9 10:17 input_reorder_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.6K Feb 9 10:17 input_reorder_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb 9 10:17 input_reorder_ispc.h
-rw-rw-r-- 1 amir amir 2.6K Feb 9 10:17 input_reorder_ispc_sse4.h
-rw-rw-r-- 1 amir amir 41K Feb 9 10:14 libcommon.a
-rw-rw-r-- 1 amir amir 16M Feb 9 10:17 libdnnl.a
lrwxrwxrwx 1 amir amir 24 Feb 9 10:19 libOpenImageDenoise.so -> libOpenImageDenoise.so.0
lrwxrwxrwx 1 amir amir 28 Feb 9 10:19 libOpenImageDenoise.so.0 -> libOpenImageDenoise.so.1.3.0
-rwxrwxr-x 1 amir amir 34M Feb 9 10:19 libOpenImageDenoise.so.1.3.0
-rw-rw-r-- 1 amir amir 37K Feb 9 10:19 libutils.a
drwxrwxr-x 2 amir amir 4.0K Feb 9 10:17 local__core
-rw-rw-r-- 1 amir amir 104K Feb 9 10:14 Makefile
drwxrwxr-x 3 amir amir 4.0K Feb 9 10:14 mkl-dnn
-rwxrwxr-x 1 amir amir 181K Feb 9 10:19 oidnBenchmark
-rwxrwxr-x 1 amir amir 59K Feb 9 10:19 oidnDenoise
-rwxrwxr-x 1 amir amir 882K Feb 9 10:19 oidnTest
-rw-r--r-- 1 amir amir 1.6K Feb 9 10:14 OpenImageDenoiseConfigVersion.cmake
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 output_copy_ispc_avx2.h
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 output_copy_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 1.9K Feb 9 10:17 output_copy_ispc.h
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 output_copy_ispc_sse4.h
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 output_reorder_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 output_reorder_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb 9 10:17 output_reorder_ispc.h
-rw-rw-r-- 1 amir amir 2.5K Feb 9 10:17 output_reorder_ispc_sse4.h
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 upsample_ispc_avx2.h
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 upsample_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 1.8K Feb 9 10:17 upsample_ispc.h
-rw-rw-r-- 1 amir amir 1.5K Feb 9 10:17 upsample_ispc_sse4.h
drwxrwxr-x 2 amir amir 4.0K Feb 9 10:17 weights

Bash, need to change word order in multiple directories

I have a considerable classical FLAC collection where each album is a directory. I've realized that I have used a sub-optimal structure and need to rename all the directories.
My current naming convention is:
COMPOSER (CONDUCTOR) - NAME OF PIECE
E.g.
"Bach (Celibidache) - Mass in F minor"
I want to change the naming to
COMPOSER - NAME OF PIECE (CONDUCTOR)
I.e.
"Bach - Mass in F minor (Celibidache)"
There are some possible exceptions, the (CONDUCTOR) may be (CONDUCTOR, SOLOIST) and some directories do not have the (CONDUCTOR) part and should be left as is. The NAME OF PIECE can contain all legal letters and symbols.
All albums are located in the same parent directory, so no sub-directories.
What is the easy way to do this?
use perl rename (some distributions have this as rename - Ubuntu and related, some as prename - Fedora and Redhat AFAIK). Check first.
prename -n -- '-d && s/(\(.*\)) - (.*)/- \2 \1/' *
-n don't rename just print the results - remove after you are ok with the results.
-- end of the options, start of the perlexpr and files
-d check that the file is a directory
s/.../.../ - substitution
Example:
[test01#localhost composers]$ ls -la
total 12
drwxrwxr-x 3 test01 test01 4096 Feb 14 12:37 .
drwxrwxr-x. 7 test01 test01 4096 Feb 14 12:23 ..
drwxrwxr-x 2 test01 test01 4096 Feb 14 12:37 'Bach (Celibidache) - Mass in F minor'
-rw-rw-r-- 1 test01 test01 0 Feb 14 12:27 'Bach (Celibidache) - Mass in F minor.flac'
[test01#localhost composers]$ prename -n -- '-d && s/(\(.*\)) - (.*)/- \2 \1/' *
Bach (Celibidache) - Mass in F minor -> Bach - Mass in F minor (Celibidache)
[test01#localhost composers]$ prename -- '-d && s/(\(.*\)) - (.*)/- \2 \1/' *
[test01#localhost composers]$ ls -la
total 12
drwxrwxr-x 3 test01 test01 4096 Feb 14 12:38 .
drwxrwxr-x. 7 test01 test01 4096 Feb 14 12:23 ..
-rw-rw-r-- 1 test01 test01 0 Feb 14 12:27 'Bach (Celibidache) - Mass in F minor.flac'
drwxrwxr-x 2 test01 test01 4096 Feb 14 12:37 'Bach - Mass in F minor (Celibidache)'
Note that without -d both the file and the directory would have been renamed.

Can I remove the "sudo" necessity in a django project on OSX

I am doing a django project with Sublime Text 2 on OSX-Lion. I have installed virtualenv, so my project is in "/Users/myname/Virtualenvs/"
I need to use a 'sudo' for each command (syncdb, runserver), can I change these rules whithout moving my project and is it normal ?
These are parameters after a "sudo python manage.py startproject"
Chmod infos :
drwxr-xr-x 17 root staff 578 10 jul 23:24 Platform
-rw-r--r-- 1 root staff 155648 10 jul 23:24 database.sqlite3
-rw-r--r-- 1 root staff 251 10 jul 23:09 manage.py
and in Platform :
drwxrwxr-x# 9 Nicolas staff 306 10 jul 23:13 Templates
-rw-r--r--# 1 Nicolas staff 0 4 jul 16:53 __init__.py
-rw-r--r--# 1 Nicolas staff 144 4 jul 16:54 __init__.pyc
-rw-rw-r--# 1 Nicolas staff 123 10 jul 23:13 admin.py
-rw-r--r-- 1 root staff 321 10 jul 23:24 admin.pyc
-rw-rw-r--# 1 Nicolas staff 1706 10 jul 23:13 models.py
-rw-r--r-- 1 root staff 2603 10 jul 23:22 models.pyc
-rw-r--r--# 1 Nicolas staff 5309 10 jul 23:06 settings.py
-rw-r--r-- 1 root staff 3058 10 jul 23:22 settings.pyc
-rw-r--r--# 1 Nicolas staff 639 10 jul 23:06 urls.py
-rw-r--r-- 1 root staff 784 10 jul 23:24 urls.pyc
-rw-rw-r--# 1 Nicolas staff 1895 10 jul 23:06 views.py
-rw-r--r-- 1 root staff 2745 10 jul 23:24 views.pyc
-rw-r--r--# 1 Nicolas staff 1138 10 jul 23:05 wsgi.py
-rw-r--r-- 1 root staff 1047 10 jul 23:24 wsgi.pyc
You'll need to change ownership of all the root-owned files to yourself, but otherwise there shouldn't be any need for sudo here.

run demo django app on google app engine (DJANGO_SETTINGS_MODULE is undefined.)

I am trying to run the django 1.3 test app named Polls in googleappenginelauncher version 1.6.6 and am getting the following error. Can anyone tell me how to do this, please?
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
INFO 2012-06-05 19:08:30,766 dev_appserver.py:2904] "GET /polls/1/ HTTP/1.1" 500 -
I have placed my app.yaml in the app directory, not the project directory, and that may be a problem, but here is my current app.yaml. Also, I have declared my views.py as my main script, which was also a guess.
application: ctst
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /_ah/queue/deferred
script: djangoappengine.deferred.handler.application
login: admin
- url: /.*
script: views.app
libraries:
- name: django
version: "1.3"
My mysite and my polls directories are as follows.
server:mysite brian$ ls -l
total 352
-rw-r--r-- 1 brian staff 0 Jun 4 10:22 __init__.py
-rw-r--r-- 1 brian staff 155 Jun 4 13:56 __init__.pyc
-rw-r--r-- 1 brian staff 503 Jun 4 10:22 manage.py
drwxr-xr-x 20 brian staff 680 Jun 5 14:54 polls
-rw-r--r-- 1 brian staff 5273 Jun 5 15:00 settings.py
-rw-r--r-- 1 brian staff 2955 Jun 5 15:00 settings.pyc
drwxr-xr-x 2 brian staff 68 Jun 4 14:04 sqlite3.db
-rw-r--r-- 1 brian staff 147456 Jun 5 11:42 sqlite3.dp
-rw-r--r-- 1 brian staff 1048 Jun 5 10:16 urls.py
-rw-r--r-- 1 brian staff 464 Jun 5 10:18 urls.pyc
server:mysite brian$ ls -l polls/
total 104
-rw-r--r-- 1 brian staff 0 Jun 4 14:23 __init__.py
-rw-r--r-- 1 brian staff 151 Jun 4 14:33 __init__.pyc
-rw-r--r-- 1 brian staff 505 Jun 4 16:33 admin.py
-rw-r--r-- 1 brian staff 1123 Jun 4 16:33 admin.pyc
-rw-r--r-- 1 brian staff 263 Jun 5 14:54 app.yaml
-rw-r--r-- 1 brian staff 524 Jun 4 14:50 models.py
-rw-r--r-- 1 brian staff 1484 Jun 4 14:50 models.pyc
-rw-r--r-- 1 brian staff 383 Jun 4 14:23 tests.py
-rw-r--r-- 1 brian staff 708 Jun 5 11:26 urls.py
-rw-r--r-- 1 brian staff 902 Jun 5 11:42 urls.pyc
-rw-r--r-- 1 brian staff 1154 Jun 5 11:41 views.py
-rw-r--r-- 1 brian staff 1231 Jun 5 11:42 views.pyc
server:mysite brian$
Try adding
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
somewhere you know it's called for every request. Note that os.environ is threadlocal with python27, so setting on initial import is probably not enough.