So I setup some code to watch a config file for edits, which worked until I used VIM to edit the file, then I had to also watch the directory for renames and creations. Then I discovered that didn't catch renames higher in the path hierarchy. Then I looked into symlinks...gaaahhhh!
First setup a made up example showing one (of many) tricky symlink scenarios:
mkdir config1
touch config1/config
ln -s config1 machine1
mkdir config2
touch config2/config
ln -s config2 machine2
ln -s machine1 active
Now, given a filename like active/config that I want to watch, I can see how to get an inotify watch descriptor for:
config1/ -> watch active/ follow symlinks (watches inode for config1)
active/ -> watch active/ dont follow symlinks (watches inode for active symlink
active/config -> watch active/config (watches inode for config1/config)
How do I add a watch on the machine1 symlink? Do I need to find some way to manually walk each symlink adding watches for each along the way? How?
The purpose is to allow:
mkdir config3
touch config3/config
ln -s -f -n config3 machine1
And have inotify warn that active/config has been redirected. At the moment it looks like I'll have to add a watch for:
- target file inode
- every directory inode leading to the file (to detect moves/renames of directories)
- every symlink inode involved in reaching any of the above
There must be an easier way to just watch one file? Have I strayed from the path or is this really the way?
My answer is a straight "yes, you are doing it right".
After carefully reading the inotify syscall manpage, I cannot see any way of short of watching every step of a (possibly-symlinked) path-to-a-file in order to detect any and all changes to the full path.
This just seems to be the way inotify works: it only looks at specific files or folders, and it does not do recursion on its own. That, plus having to explicitly follow symlinks, seems to match your 3-step plan to the letter.
Selected quotes from the manpage:
The following further bits can be specified in mask when calling inotify_add_watch(2):
IN_DONT_FOLLOW (since Linux 2.6.15)
Don't dereference pathname if it is a symbolic link.
[...]
Limitations and caveats
Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for large directory trees. [...]
This FAQ also lends support for your strategy re symlinks:
Q: What about the IN_ONLYDIR and IN_DONT_FOLLOW flags?
IN_ONLYDIR ensures that the event occur only on a directory. If you create such watch on a file it will not emit events. IN_DONT_FOLLOW forbids following symbolic links (these ones will be monitored themselves and not the files they point to).
Related
When running scripts in bash, I have to write ./ in the beginning:
$ ./manage.py syncdb
If I don't, I get an error message:
$ manage.py syncdb
-bash: manage.py: command not found
What is the reason for this? I thought . is an alias for current folder, and therefore these two calls should be equivalent.
I also don't understand why I don't need ./ when running applications, such as:
user:/home/user$ cd /usr/bin
user:/usr/bin$ git
(which runs without ./)
Because on Unix, usually, the current directory is not in $PATH.
When you type a command the shell looks up a list of directories, as specified by the PATH variable. The current directory is not in that list.
The reason for not having the current directory on that list is security.
Let's say you're root and go into another user's directory and type sl instead of ls. If the current directory is in PATH, the shell will try to execute the sl program in that directory (since there is no other sl program). That sl program might be malicious.
It works with ./ because POSIX specifies that a command name that contain a / will be used as a filename directly, suppressing a search in $PATH. You could have used full path for the exact same effect, but ./ is shorter and easier to write.
EDIT
That sl part was just an example. The directories in PATH are searched sequentially and when a match is made that program is executed. So, depending on how PATH looks, typing a normal command may or may not be enough to run the program in the current directory.
When bash interprets the command line, it looks for commands in locations described in the environment variable $PATH. To see it type:
echo $PATH
You will have some paths separated by colons. As you will see the current path . is usually not in $PATH. So Bash cannot find your command if it is in the current directory. You can change it by having:
PATH=$PATH:.
This line adds the current directory in $PATH so you can do:
manage.py syncdb
It is not recommended as it has security issue, plus you can have weird behaviours, as . varies upon the directory you are in :)
Avoid:
PATH=.:$PATH
As you can “mask” some standard command and open the door to security breach :)
Just my two cents.
Your script, when in your home directory will not be found when the shell looks at the $PATH environment variable to find your script.
The ./ says 'look in the current directory for my script rather than looking at all the directories specified in $PATH'.
When you include the '.' you are essentially giving the "full path" to the executable bash script, so your shell does not need to check your PATH variable. Without the '.' your shell will look in your PATH variable (which you can see by running echo $PATH to see if the command you typed lives in any of the folders on your PATH. If it doesn't (as is the case with manage.py) it says it can't find the file. It is considered bad practice to include the current directory on your PATH, which is explained reasonably well here: http://www.faqs.org/faqs/unix-faq/faq/part2/section-13.html
On *nix, unlike Windows, the current directory is usually not in your $PATH variable. So the current directory is not searched when executing commands. You don't need ./ for running applications because these applications are in your $PATH; most likely they are in /bin or /usr/bin.
This question already has some awesome answers, but I wanted to add that, if your executable is on the PATH, and you get very different outputs when you run
./executable
to the ones you get if you run
executable
(let's say you run into error messages with the one and not the other), then the problem could be that you have two different versions of the executable on your machine: one on the path, and the other not.
Check this by running
which executable
and
whereis executable
It fixed my issues...I had three versions of the executable, only one of which was compiled correctly for the environment.
Rationale for the / POSIX PATH rule
The rule was mentioned at: Why do you need ./ (dot-slash) before executable or script name to run it in bash? but I would like to explain why I think that is a good design in more detail.
First, an explicit full version of the rule is:
if the path contains / (e.g. ./someprog, /bin/someprog, ./bin/someprog): CWD is used and PATH isn't
if the path does not contain / (e.g. someprog): PATH is used and CWD isn't
Now, suppose that running:
someprog
would search:
relative to CWD first
relative to PATH after
Then, if you wanted to run /bin/someprog from your distro, and you did:
someprog
it would sometimes work, but others it would fail, because you might be in a directory that contains another unrelated someprog program.
Therefore, you would soon learn that this is not reliable, and you would end up always using absolute paths when you want to use PATH, therefore defeating the purpose of PATH.
This is also why having relative paths in your PATH is a really bad idea. I'm looking at you, node_modules/bin.
Conversely, suppose that running:
./someprog
Would search:
relative to PATH first
relative to CWD after
Then, if you just downloaded a script someprog from a git repository and wanted to run it from CWD, you would never be sure that this is the actual program that would run, because maybe your distro has a:
/bin/someprog
which is in you PATH from some package you installed after drinking too much after Christmas last year.
Therefore, once again, you would be forced to always run local scripts relative to CWD with full paths to know what you are running:
"$(pwd)/someprog"
which would be extremely annoying as well.
Another rule that you might be tempted to come up with would be:
relative paths use only PATH, absolute paths only CWD
but once again this forces users to always use absolute paths for non-PATH scripts with "$(pwd)/someprog".
The / path search rule offers a simple to remember solution to the about problem:
slash: don't use PATH
no slash: only use PATH
which makes it super easy to always know what you are running, by relying on the fact that files in the current directory can be expressed either as ./somefile or somefile, and so it gives special meaning to one of them.
Sometimes, is slightly annoying that you cannot search for some/prog relative to PATH, but I don't see a saner solution to this.
When the script is not in the Path its required to do so. For more info read http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html
All has great answer on the question, and yes this is only applicable when running it on the current directory not unless you include the absolute path. See my samples below.
Also, the (dot-slash) made sense to me when I've the command on the child folder tmp2 (/tmp/tmp2) and it uses (double dot-slash).
SAMPLE:
[fifiip-172-31-17-12 tmp]$ ./StackO.sh
Hello Stack Overflow
[fifi#ip-172-31-17-12 tmp]$ /tmp/StackO.sh
Hello Stack Overflow
[fifi#ip-172-31-17-12 tmp]$ mkdir tmp2
[fifi#ip-172-31-17-12 tmp]$ cd tmp2/
[fifi#ip-172-31-17-12 tmp2]$ ../StackO.sh
Hello Stack Overflow
Right now, I'm trying to build a tool from source and use it to build a C++ project. I'm able to extract the tar file (gcc-arm-none-eabi). But, when I try to add it to path (using $GITHUB_PATH, not add-path), the path doesn't apply on my next action and I can't build the file. The error states that it can't find the gcc-arm-none-eabi toolset, which means that it didn't go to path.
Here's the script for the entrypoint of the first function (make is ran in the next action to allow for path to apply)
echo "Downloading ARM Toolchain"
# The one from apt isn't updated so I have to build from source
curl -L https://developer.arm.com/-/media/Files/downloads/gnu-rm/10-2020q4/gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2 -o gcc-arm-none-eabi.tar.bz2
tar -xjf gcc-arm-none-eabi.tar.bz2
echo "/github/workspace/gcc-arm-none-eabi-10-2020-q4-major/bin" >> $GITHUB_PATH
I can't even debug by seeing what's in the path because running echo $(PATH) just says that PATH cannot be found. What should I do?
I can't even debug by seeing what's in the path because running echo $(PATH) just says that PATH cannot be found. What should I do?
First, PATH is not a command so if you want to print its value, it would be something like echo "${PATH}" or echo "$PATH"
Then, if you want to add a value to an existing environment variable, it would be something like
export PATH="${PATH}:/github/workspace/gcc-arm-none-eabi-10-2020-q4-major/bin"
EDIT: seems not a valid way to add something to the path using Github Actions, meanwhile it seems correct in the question. To get more details: https://docs.github.com/en/free-pro-team#latest/actions/reference/workflow-commands-for-github-actions#adding-a-system-path . Thanks to Benjamin W. for pointing this out in the comments.
Finally I think it would be a better fit if you use a docker image that already contains that kind of dependancies (you could easily write your own Dockerfile if this image doesn't already exists). Github action is designed to use docker (or OCI containers) image that contains the dependancies you need to perform your build actions. You should take a look here: https://docs.github.com/en/free-pro-team#latest/actions/creating-actions/dockerfile-support-for-github-actions
I am starting out with F# and trying to get it to work with Sublime Text 3 with a package, https://github.com/fsharp/sublime-fsharp-package. After installing the package using Package Control, I see F# appear as an available language to use in Sublime Text's bottom bar, and syntax highlighting appears to work more or less, from what I can tell, but the build system for F# fails to appear as it should.
So, trying to fix things, I run "build.sh install" and get an error, "Cannot open assembly '.paket/paket.bootstrapper.exe': No such file or directory." I am sort of stuck. Many thanks for any help.
From the comments you've made, you appear to be a little unfamiliar with the Unix underpinnings of OS X. I'll explain those first, then I'll suggest something for you to try that may fix your problem.
Technically, files or directories whose name starts with . are not "reserved for the system" as you put it; they're hidden. Now, it's true that Finder won't allow you to create files or directories whose name starts with ., because Apple didn't want to have to field all the tech-support calls from people who didn't know about the hidden-files feature: "I named my file ... more important stuff for work and now it's gone! Help!!!" But if you're in the Terminal app, then you can easily create files or directories with . as their first letter: mkdir .foo should work. You won't see it when you do ls, but ls -a (a for "all") will show you all files, including hidden files. And you can also do cd .foo and create files inside the hidden .foo directory -- and while the .foo folder won't show up in Finder, it will be perfectly accessible in the Terminal, and to any F# programs you might write.
So when you say that you cloned https://github.com/fsprojects/Paket but it failed to include the .github and .paket directories, I think you just don't know how to see them. You can't see them in the Finder (well, you can if you jump through a couple of hoops but I don't think it's worth the effort), but you can see them with ls -a. Just open your terminal, run cd /Users/Username/Paket, and then run ls -a and I think you'll see that the .paket and .github directories were indeed created by your git clone command.
So what you should probably try is this:
Go to https://github.com/fsprojects/Paket/releases/latest
Download the paket.bootstrapper.exe and paket.exe files. Put them in /Users/Username/Downloads (or wherever the default OS X Downloads directory is if it's different -- just as long as it's somewhere where you can find them easily).
Open the Terminal app.
Go to the directory where you've unpacked the Sublime Text 3 package. I.e., in the Terminal app, run cd /Users/Username/Library/Application\ Support/Sublime\ Text\ 3/Packages/sublime-fsharp-package-master.
Run ls -a and see if there's a .paket directory.
If it does not exist, run mkdir .paket.
Now do cd .paket so you're in the hidden .paket directory under sublime-fsharp-package-master.
Now do ls and see if there's a paket.bootstrapper.exe file.
If it doesn't exist, then copy in the .exe files you downloaded earlier:
cp /Users/Username/Downloads/paket.bootstrapper.exe .
cp /Users/Username/Downloads/paket.exe .
Important: Now do cd .. to go back up to the /Users/Username/Library/Application\ Support/Sublime\ Text\ 3/Packages/sublime-fsharp-package-master/ directory.
Now instead of running /Users/Username/Library/Application\ Support/Sublime\ Text\ 3/Packages/sublime-fsharp-package-master/build.sh install, try running it as ./build.sh install. (And also try ./build.sh Install, since I'm pretty sure the capital I is necessary).
(BTW, If you're not familiar with the syntax that I used in steps 9, 10 and 11, where I used a single . or two dots .. in commands, those are a long-standing Unix idiom: . means "the current directory", and .. means "the parent directory".)
I just looked at the build.sh script that you've been running, and it seems to assume that you've done a cd into the package's base directory (the sublime-fsharp-package-master directory) before running the script. So that could explain why it was failing: you were running it from a different directory, rather than doing a cd first. Hence why I marked step 10 as important: I think that was the root cause of the problem.
I'm the author of a utilty that makes compressing projects using zip a bit easier, especially when you have to compress regularly, such as for updating projects submitted to an application store (like Chrome's Web Store).
I'm attempting to make quite a few improvements, but have run into an issue, described below.
A Quick Overview
My utility's command format is similar to command OPTIONS DEST DIR1 {DIR2 DIR3 DIR4...}. It works by running zip -r DEST.zip DIR1; a fairly simple process. The benefit to my utility, however, is the ability to use a predetermined file (think .gitignore) to ignore specific files/directories, or files/directories which match a pattern.
It's pretty simple -- if the "ignorefile" exists in a target directory (DIR1, DIR2, DIR3, etc), my utility will add exclusions to the zip -r DEST.zip DIR1 command using the pattern -x some_file or -x some_dir/*.
The Issue
I am running into an issue with directory exclusion, however, and I can't quite figure out why (this is probably be because I am still quite the sh novice). I'll run through some examples:
Let's say that I want to ignore two things in my project directory: .git/* and .gitignore. Running command foo.zip project_dir builds the following command:
zip -r foo.zip project -x project/.git/\* -x project/.gitignore
Woohoo! Success! Well... not quite.
In this example, .gitignore is not added to the compressed output file, foo.zip. The directory, .git/*, and all of it's subdirectories (and files) are added to the compressed output file.
Manually running the command:
zip -r foo.zip project_dir -x project/.git/\* -x project/.gitignore
Works as expected, of course, so naturally I am pretty puzzled as to why my identical, but dynamically-built command, does not work.
Attempted Resolutions
I have attempted a few different methods of resolving this to no avail:
Removing -x project/.git/\* from the command, and instead adding each subdirectory and file within that directory, such as -x project/.git/config -x project/.git/HEAD, etc (including children of subdirectories)
Removing the backslash before the asterisk, so that the resulting exclusion option within the command is -x project/.git/*
Bashing my head on the keyboard in angst (I'm really surprised this didn't work, it usually does)
Some notes
My utility uses /bin/sh; I would prefer to keep it that way for maximum compatibility.
I am aware of the git archive feature -- my use of .git/* and .gitignore in the above example is simply as an example; my utility is not dependent on git nor is used exclusively for projects which are git repositories.
I suspected the problem would be in the evaluation of the generated command, since you said the same command when executed directly did right.
So as the comment section says, I think you already found the correct solution. This happens because if you run that variable directly, some things like globs can be expanded directly, instead of passed to the command. And arguments may be messed up, depending on the situation.
Yes, in that case:
eval $COMMAND
is the way to go.
I want to copy all files from server A to server B that have the same parent directory-name in different levels of filesystem hierarchy, e.g:
/var/lib/data/sub1/sub2/commonname/filetobecopied.foo
/var/lib/data/sub1/sub3/commonname/filetobecopied.foo
/var/lib/data/sub2/sub4/commonname/anotherfiletobecopied.foo
/var/lib/data/sub3/sub4/differentname/fileNOTtobecopied.foo
I want to copy the first three files that all have the commonname in path to server B. I already spent a lot of time in finding the correct include/exclude patterns for rsync but I dont get it. The following command does not work:
rsync -a --include='**/commonname/*.foo' --exclude='*' root#1.2.3.4:/var/lib/data /var/lib/data
I either match too much or to few of the files. How can I sync only the files with the commonname in its path?
I guess you're looking for this:
rsync -a -m --include='**/commonname/*.foo' --include='*/' --exclude='*' root#1.2.3.4:/var/lib/data /var/lib/data
There are 2 differences with your command:
The most important one is --include='*/'. Without this, as you specified --exclude='*', rsync will never enter the subdirectories, since everything is excluded. With --include='*/', the subdirectories are not excluded anymore, so rsync can happily recurse.
The least important one is -m: this prunes the empty directories. Without this, you'd also get the (empty) subdirectory /var/lib/data/sub3/sub4/differentname/ copied.