There is a directory where a buddy adds new builds of a product.
The listing looks like this
$ ls path-to-dir/
01
02
03
04
$
where the numbers listed are not files but names of directories containing the builds.
I have to manually go and check every time whether there is a new build or not. I am looking for a way to automate this, so that the program can send an email to some people (including me) whenever path-to-dir/ is updated.
Do we have an already existing utility or a Perl library that does this?
inotify.h does something similar, but it is not supported on my kernel (2.6.9).
I think there can be an easy way in Perl.
Do you think this will work?
Keep running a loop in Perl that does a ls path-to-dir/ after, say, every 5 minutes and stores the results in an array. If it finds that the new results are different from the old results, it sends out an email using Mail or Email.
If you're going for perl, I'm sure the excellent File::ChangeNotify module will be extremely helpful to you. It can use inotify, if available, but also all sorts of other file-watching mechanisms provided by different platforms. Also, as a fallback, it has its own watching implementation, which works on every platform, but is less efficient than the specialized ones.
Checking for different ls output would send a message even when something is deleted or renamed in the directory. You could instead look for files with an mtime newer than the last message sent.
Here's an example in bash, you can run it every 5 minutes:
now=`date +%Y%m%d%H%M.%S`
if [ ! -f "/path/to/cache/file" ] || [ -n "`find /path/to/build/dir -type f -newer /path/to/cache/file`" ]
then
touch /path/to/cache/file -t "$now"
sendmail -t <<< "
To: aaa#bbb.ccc
To: xxx#yyy.zzz
Subject: New files found
Dear friend,
I have found a couple of new files.
"
fi
Can't it be a simple shell script?
while :;do
n = 'ls -al path-to-dir | wc -l'
if n -gt old_n
# your Mail code here; set old_n=n also
fi
sleep 5
done
Yes, a loop in Perl as described would do the trick.
You could keep a track of when the directory was last modified; if it hasn't changed, there isn't a new build. If it has changed, an old build might have been deleted or a new one added. You probably don't want to send alerts when old builds are removed; it is crucial that the email is sent when new builds are added.
However, I think that msw has the right idea; the build should notify when it has completed the copy out to the new directory. It should be a script that can be changed to notify the correct list of people - rather than a hard-wired list of names in the makefile or whatever other build control system you use.
you could use dnotify it is the predecessor of inotify and should be available on your kernel. It is still supported by newer kernels.
Related
On the top right corner of the SSH-in-Browser terminal for a Compute Engine, there is a download button. What I'd like to do is trigger this functionality directly from the VM terminal.
Why?
Every once in a while, I'm requested to make a copy of a database for any reason. So far, this requires me to 1) Make the dumps, 2) Zip the dumps, 3) Download the zip and 4) Send it to whoever requested it. I decided to automate this since its becoming more frequent.
So far, 1 and 2 are done, and I would usually leave it at that, but I'm wondering if I can just trigger the "downloading function" from the script that handles the dumps and zipping, just to save me a few clicks and writing down the path. For example:
#!/bin/bash
here=$(pwd)
mkdir /home/me/back
mkdir /home/me/back/data
/home/me/saveTables $1
/home/me/saveRecords $1
cd /home/me/back
zip -r $here/$1.zip ./* > /dev/null
sudo rm -dr /home/me/back
# and then download it with something like
# 'gcpdk' is made up just for example purposes
gcpdk --download $here/$1.sql
Is this possible from the SSH-in-Browser at all? So far I'm guessing that the download function is not accessible from within the VM terminal, but I have not found documentation about the downloads themselves yet, so I can't tell. Or perhaps this is something I should do with a different tool?
By the way, I can't copy files to Cloud Storage (Lack the permissions) so at least for now, that is not an option.
Any help, link or confirmation of whether or not this is doable is welcome.
And just to be extra sure: This is not something critical. Its just me being lazy. Cheers!
I have been testing an older AWS Tools install using AWSToolsAndSDKForNet_sdk-3.3.398.0_ps-3.3.390.0_tk-1.14.4.1.msi and a newer install using AWSToolsAndSDKForNet_sdk-3.5.2.0_ps-4.1.0.0_tk-1.14.5.0.msi. The code that I am using to test with is
Set-AWSCredential -AccessKey:$ACCESSKEY -SecretKey:$SECRETKEY -StoreAs:default
$items = Get-S3Object -BucketName:$BUCKETNAME -Region:'eu-west-1' -Key:'revit/2020'
Write-Host "$($items.Length) items"
$count = 1
foreach ($item in $items) {
Write-Host "$count $($item.key)"
$count ++
}
I am seeing VERY different behavior, and can't figure out why. With 3.3 the code works as intended, I end up with a list of files in my bucket and key. Performance is pretty decent, it takes a moment but I have about 5000 files in may "subfolders".
When I run this with 4.1 it takes 3-5 times as long and returns nothing.
It seems that Help is a bit different too. A first run of get-help Get-S3Object -detailed will take as long as 10 minutes to run, with CPU, Memory and Disk access often at 99% utilization. A second run is quite quick. 3.3 Does nothing of the sort.
So, is this current build of AWS Tools for Powershell just not ready for prime time? My searches for AWS Tools 4.1 performance have turned up nothing.
For what it is worth, I am using the MSI installer because I need the install to actually work consistently, and the NuGet approach has been very problematic on a number of production workstations. But if there is another option I would love to look at it. The main issue is I need ultimately to do the install and immediately load the modules and work with AWS. I don't have that working with the MSI based install yet, but that's for a different thread.
It looks like they changed the results from Get-S3Object. You will need to add -Select S3Objects.Key to get the results you're looking for (or just -select *). Here's the excerpt from the change notes:
Most cmdlets have a new parameter: -Select. Select can be used to change the value returned by the cmdlet. For example the service API used by Get-S3Object returns a ListObjectsResponse object but the cmdlet is configured to return only the S3Objects field. Now you can specify -Select * to receive the full API response. You can also specify the path to a nested result property like -Select S3Objects.Key. In certain situations it may be useful to return a cmdlet parameter, this can be achieved with -Select ^ParameterName.
Found by going to the Change Notes and doing a CTRL+F for Get-S3Object. Hope this resolves it for you!
I am used to dumping objects from a queue manager without system objects.
But now I have to use dmpmqcfg and I cant find a flag to remove those system objects from output.
MQ version is 7.5.0.1
Command I use now : dmpmqcfg -m SMQ -t all -x object
Possible flags : c:\> dmpmqcfg -h
Usage: dmpmqcfg [-m QMgrName] [-n ObjName] [-t ObjType]
[-x ExportType] [-o Format] [-a] [-z]
[-s MsgSeqNo] [-q RplQName] [-r RmtQMgrName]
[-c default|-c DEFINE CHANNEL..]
-m Queue manager name.
-n Object name or a generic object name.
-t Object type:
all, authinfo, channel, clntconn, comminfo, listener,
namelist, process, queue, qmgr, service, topic.
-x Export type:
all, object, authrec, chlauth, sub.
-o Format: mqsc, 1line, setmqaut, grtmqmaut.
-a Dump all attributes.
-z Suppress warnings.
-s Reset channel message sequence number
-q Reply to queue name, default SYSTEM.DEFAULT.MODEL.QUEUE
-r Remote queue manager name (queued mode)
-c Client connection:
default, DEFINE CHANNEL(chlname) CHLTYPE(CLNTCONN) ...
The saveqmgr command was a SupportPac that was maintained outside of the MQ Development lab and with close interaction with the MQ User community through MQSeries.net and the Vienna MQ List Server. Though it was for many years the only way to get a full backup of the queue manger's configuration files, it was never a supported component of the product.
The dmpmqcfg command is a fully supported method of backing up the full configuration of the queue manager, maintained out of the MQ Dev lab. It's requirements were developed in close communication with the customers participating in the Early Access Program (effectively MQ's Beta program). Though there is some overlap with saveqmgr it was not intended to be a direct replacement for that program. Specifically, since it is primarily intended to create a complete backup it lacks the ability to omit the SYSTEM.* objects.
You can simulate the same thing by using the -o 1line option and filtering out the SYSTEM.* objects.
Windows:
dmpmqcfg -m [qmgr] -o 1line | findstr /V "('SYSTEM"
UNIX/Linux:
dmpmqcfg -m [qmgr] -o 1line | grep -v "('SYSTEM"
Note that this filters out any line containing ('SYSTEM, even if that string is in a description or other field. If you wanted to be completely sure you got only objects named SYSTEM.* you would need to be more explicit and use multiple filters like so:
dmpmqcfg -m [qmgr] -o 1line | grep -v " CHANNEL('SYSTEM" | grep -v " QLOCAL('SYSTEM" | grep -v " QALIAS('SYSTEM"...
I leave it as an exercise to the reader to add all the possible object types to filter onto the end of that command pipeline.
Note that you do not want to filter out AUTHREC definitions that contain PROFILE('SYSTEM because these are needed to control access to model queues, the command queue, etc.
It is unfortunate that the MQ Dev team does not work as closely with the MQ Community as the SupportPac authors, however there is good reason for this. Note that SupportPacs are not always kept up to date and their maintenance and bug-fix are performed on an as-available basis. While in general the SupportPac authors are very good at keeping up they have no enforced deadline.
Contrast this with the MQ Dev Lab's process for creating a new component like dmpmqcfg. Their use of the Early Access Program (EAP) formalizes the requirements and tracks them to ensure that the component is released on time with the rest of the product, translated into many languages, documented in the manual, etc. The issue isn't so much that the dev lab isn't listening to our requirements as much as it is getting companies to join the EAP, and then once there to dedicate time to testing and providing feedback in a timely manner.
However, as Jon stated, submitting a requirement on a released component is possible through the Request For Enhancement (RFE) process. What Jon's answer left out is that in order to be considered and prioritized, the RFE must either have a really compelling business case, or else receive some votes and/or comments. There are some strategies to help get the RFE passed:
Don't split votes between RFEs. Before submitting one, search (browse really because the RFE search functionality sucks) for similar ones. If you find one that is close, comment on it to clarify your requirement rather than submitting a new one, and vote it up.
Discuss your proposal with the communities at the Listserv and/or MQSeries.net. If someone has an RFE that you missed, you'll hear about it here. You will also get an idea of how much support you might get for an RFE, or if the thing you want is an anti-pattern (looks GREAT until you do it, at which point it becomes dangerous in unanticipated ways - for example using DNS names in CHLAUTH records).
If you do submit an RFE post a note to the Listserv and/or to MQSeries.net to let the community know that you have submitted it and ask them to comment and vote it up.
If you find an existing RFE or submit your own, update your question or add a comment to it to track progress. Stack Overflow is supposed to evolve over time and if the problem you face is solved, capturing that here will help others. And some of us here on SO will vote for it if we find the link in your follow-up.
Hope that helps. My requirement for dmpmqcfg was always that it duplicated the functionality of saveqmgr, plus a few more things. I'd vote for this.
I am comparing two almost identical folders which include hidden .svn folders which should be ignored and I want to continually quickly compare the folders as some files are patched to compared the difference without checking the unchanged matching files again.
edit:
Because there are so many options I'm interested in a solution that clearly exploits the knowledge from the previous compare because any other solution is not really feasable when doing repeated comparisons.
If you are willing to spend a bit of money, Beyond Compare is a pretty powerful diffing tool that can do folder based diffing.
Beyond Compare
I personally use WinMerge and find it very useful. It has filters that exclude svn file. Under linux i prefer Meld.
One option would be to use rsync. Something like:
rsync -n -r -v -C dir_a dir_b
The -n option does a dry-run so no files will be modified. -r does a recursive comparison. Optionally turn on verbose mode with -v. (You could use -i to itemize the changes instead of -v.) To ignore commonly ignored files such as .svn/ use -C.
This should be faster than a simple diff as I read the rsync manpage:
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time. Any changes in the other preserved
attributes (as requested by options) are made on the destination file
directly when the quick check indicates that the file's data does not
need to be updated.
Since the "quick check" algorithm does not look at file contents directly, it might be fooled. In that case, the -c option, which performs a checksum instead, may be needed. It is likely to be faster than an ordinary diff.
In addition, if you plan on syncing the directories at some point, this is a good tool for that job as well.
Not foolproof, but you could just compare the timestamps.
Use total commander ! All the cool developers use it :)
If you are on linux or some variant, you should be able to do:
prompt$ diff -r dir1 dir2 --exclude=.svn
The -r forces recursive lookups. There are a bunch of switches to ignore stuff like whitespace etc.
Sometimes I have to work on code that moves the computer clock forward. In this case some .cpp or .h files get their latest modification date set to the future time.
Later on, when my clock is fixed, and I compile my sources, system rebuilds most of the project because some of the latest modification dates are in the future. Each subsequent recompile has the same problem.
Solution that I know are:
a) Find the file that has the future time and re-save it. This method is not ideal because the project is very big and it takes time even for windows advanced search to find the files that are changed.
b) Delete the whole project and re-check it out from svn.
Does anyone know how I can get around this problem?
Is there perhaps a setting in visual studio that will allow me to tell the compiler to use the archive bit instead of the last modification date to detect source file changes?
Or perhaps there is a recursive modification date reset tool that can be used in this situation?
I would recommend using a virtual machine where you can mess with the clock to your heart's content and it won't affect your development machine. Two free ones are Virtual PC from Microsoft and VirtualBox from Sun.
If this was my problem, I'd look for ways to avoid mucking with the system time. Isolating the code under unit tests, or a virtual machine, or something.
However, because I love PowerShell:
Get-ChildItem -r . |
? { $_.LastWriteTime -gt ([DateTime]::Now) } |
Set-ItemProperty -Name "LastWriteTime" -Value ([DateTime]::Now)
I don't know if this works in your situation but how about you don't move your clock forward, but wrap your gettime method (or whatever you're using) and make it return the future time that you need?
Install Unix Utils
touch temp
find . -newer temp -exec touch {} ;
rm temp
Make sure to use the full path when calling find or it will probably use Windows' find.exe instead. This is untested in the Windows shell -- you might need to modify the syntax a bit.
I don't use windows - but surely there is something like awk or grep that you can use to find the "future" timestamped files, and then "touch" them so they have the right time - even a perl script.
1) Use a build system that doesn't use timestamps to detect modifications, like scons
2) Use ccache to speed up your build system that does use timestamps (and rebuild all).
In either case it is using md5sum's to verify that a file has been modified, not timestamps.