I have been testing an older AWS Tools install using AWSToolsAndSDKForNet_sdk-3.3.398.0_ps-3.3.390.0_tk-1.14.4.1.msi and a newer install using AWSToolsAndSDKForNet_sdk-3.5.2.0_ps-4.1.0.0_tk-1.14.5.0.msi. The code that I am using to test with is
Set-AWSCredential -AccessKey:$ACCESSKEY -SecretKey:$SECRETKEY -StoreAs:default
$items = Get-S3Object -BucketName:$BUCKETNAME -Region:'eu-west-1' -Key:'revit/2020'
Write-Host "$($items.Length) items"
$count = 1
foreach ($item in $items) {
Write-Host "$count $($item.key)"
$count ++
}
I am seeing VERY different behavior, and can't figure out why. With 3.3 the code works as intended, I end up with a list of files in my bucket and key. Performance is pretty decent, it takes a moment but I have about 5000 files in may "subfolders".
When I run this with 4.1 it takes 3-5 times as long and returns nothing.
It seems that Help is a bit different too. A first run of get-help Get-S3Object -detailed will take as long as 10 minutes to run, with CPU, Memory and Disk access often at 99% utilization. A second run is quite quick. 3.3 Does nothing of the sort.
So, is this current build of AWS Tools for Powershell just not ready for prime time? My searches for AWS Tools 4.1 performance have turned up nothing.
For what it is worth, I am using the MSI installer because I need the install to actually work consistently, and the NuGet approach has been very problematic on a number of production workstations. But if there is another option I would love to look at it. The main issue is I need ultimately to do the install and immediately load the modules and work with AWS. I don't have that working with the MSI based install yet, but that's for a different thread.
It looks like they changed the results from Get-S3Object. You will need to add -Select S3Objects.Key to get the results you're looking for (or just -select *). Here's the excerpt from the change notes:
Most cmdlets have a new parameter: -Select. Select can be used to change the value returned by the cmdlet. For example the service API used by Get-S3Object returns a ListObjectsResponse object but the cmdlet is configured to return only the S3Objects field. Now you can specify -Select * to receive the full API response. You can also specify the path to a nested result property like -Select S3Objects.Key. In certain situations it may be useful to return a cmdlet parameter, this can be achieved with -Select ^ParameterName.
Found by going to the Change Notes and doing a CTRL+F for Get-S3Object. Hope this resolves it for you!
Related
Normally, in ${JENKINS_HOME}/updates/ there are several JSON files for automatically installing various tools. Namely, the one I need is hudson.tasks.Maven.MavenInstaller . Two others are suddenly missing: for Ant, and for JDK.
End result is, my build fails because it can't install Maven from Apache automatically (as detailed here).
I am deploying Jenkins to AWS. What's strange is, I have an AMI (image) that previously was working fine, that suddenly is encountering this problem. I've banged my head on this one extensively with no solution.
Looks like you can find the JSON that I'm failing to download here:
http://mirrors.jenkins-ci.org/updates/current/updates/
Except the JSON there is prepended with "downloadService.post()", indicating that hudson.model.DownloadService is probably doing something (other hints point to that, as well).
Any ideas?
EDIT: Actually, it looks like the last AMI that worked does, in fact, still work.
Should mention: The project is, creating a Jenkins AMI via Chef and Packer
Found the answer to this about a week after posing. Turns out, the issue was on Jenkins update center side of things, suddenly changing to a smaller RSA Key:
https://issues.jenkins-ci.org/browse/JENKINS-31089
At the time, the workaround was this:
sed -i s/'jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024'/'jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 512'/ /usr/lib/jvm/jre/lib/security/java.security
Which allowed Java to grab the updates, even though the update center was using a smaller RSA key.
I'd like to shelve old builds in all of my jobs for example
build numbers 1-10
I'm wondering if there is way to do that from the jenkins UI using a single command.
First of all in order to make changes to a bulk of jobs of I would use something called configuration slicer.
you can get to that from here: https://wiki.jenkins-ci.org/display/JENKINS/Configuration+Slicing+Plugin
Also you want to delete your build? or archive them?! in case of deleting I would use the Log rotation eaither by date or number of builds. In the configure section of the job click on Discard old build and you will see the options.
and finally you can always use Artifact deployer and somether examples from that plug in.
Link Here: https://wiki.jenkins-ci.org/display/JENKINS/ArtifactDeployer+Plugin
Link on how to use the CLI in Jenkins : https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI
EDIT 1
In regards to the comments below where you are asking about "Shelving Jobs" .
I think the phrase you are looking for here is "archive" and not shelving - that is a very Visual Studio/TFS concept - so I am not personally aware of any anything that does SHELVING per say.
In terms of Groovy script I believe that you are now asking a different question and so this should be raised specifically as different question - but as far as groovy script go you can use the following link as an intro :
http://groovy.codehaus.org/
I'm writing a library in order to help everyone using Amazon cloud search.
With cloud search when you update a document you need to specify the id of the document (of course) and the version of the document you want to upgrade to.
If the specify version number is smaller than the current version the update don't append.
So how to make sure I update my record every time I do an update?
The Ruby project aws_cloud_search is using a timestamp in order to keep the version number always higher but:
As the maximum version number is 4294967295 for AWS cloud search. So
it will not work any more after the 07 Feb 2106
If you run two updates within the same second then the last update
(the more important one) will be ignore
Aside from the timestamp approach, which appears to be the standard answer from everybody, including the docs, the only approach that I've found that works is to keep track of the version number elsewhere, and increment it as changes happen.
Of course, this approach only works if the object that you're trying to represent in the cloud search document can be accessed from somewhere else where presumably you have some sort of atomicity.
There is a directory where a buddy adds new builds of a product.
The listing looks like this
$ ls path-to-dir/
01
02
03
04
$
where the numbers listed are not files but names of directories containing the builds.
I have to manually go and check every time whether there is a new build or not. I am looking for a way to automate this, so that the program can send an email to some people (including me) whenever path-to-dir/ is updated.
Do we have an already existing utility or a Perl library that does this?
inotify.h does something similar, but it is not supported on my kernel (2.6.9).
I think there can be an easy way in Perl.
Do you think this will work?
Keep running a loop in Perl that does a ls path-to-dir/ after, say, every 5 minutes and stores the results in an array. If it finds that the new results are different from the old results, it sends out an email using Mail or Email.
If you're going for perl, I'm sure the excellent File::ChangeNotify module will be extremely helpful to you. It can use inotify, if available, but also all sorts of other file-watching mechanisms provided by different platforms. Also, as a fallback, it has its own watching implementation, which works on every platform, but is less efficient than the specialized ones.
Checking for different ls output would send a message even when something is deleted or renamed in the directory. You could instead look for files with an mtime newer than the last message sent.
Here's an example in bash, you can run it every 5 minutes:
now=`date +%Y%m%d%H%M.%S`
if [ ! -f "/path/to/cache/file" ] || [ -n "`find /path/to/build/dir -type f -newer /path/to/cache/file`" ]
then
touch /path/to/cache/file -t "$now"
sendmail -t <<< "
To: aaa#bbb.ccc
To: xxx#yyy.zzz
Subject: New files found
Dear friend,
I have found a couple of new files.
"
fi
Can't it be a simple shell script?
while :;do
n = 'ls -al path-to-dir | wc -l'
if n -gt old_n
# your Mail code here; set old_n=n also
fi
sleep 5
done
Yes, a loop in Perl as described would do the trick.
You could keep a track of when the directory was last modified; if it hasn't changed, there isn't a new build. If it has changed, an old build might have been deleted or a new one added. You probably don't want to send alerts when old builds are removed; it is crucial that the email is sent when new builds are added.
However, I think that msw has the right idea; the build should notify when it has completed the copy out to the new directory. It should be a script that can be changed to notify the correct list of people - rather than a hard-wired list of names in the makefile or whatever other build control system you use.
you could use dnotify it is the predecessor of inotify and should be available on your kernel. It is still supported by newer kernels.
Sometimes I have to work on code that moves the computer clock forward. In this case some .cpp or .h files get their latest modification date set to the future time.
Later on, when my clock is fixed, and I compile my sources, system rebuilds most of the project because some of the latest modification dates are in the future. Each subsequent recompile has the same problem.
Solution that I know are:
a) Find the file that has the future time and re-save it. This method is not ideal because the project is very big and it takes time even for windows advanced search to find the files that are changed.
b) Delete the whole project and re-check it out from svn.
Does anyone know how I can get around this problem?
Is there perhaps a setting in visual studio that will allow me to tell the compiler to use the archive bit instead of the last modification date to detect source file changes?
Or perhaps there is a recursive modification date reset tool that can be used in this situation?
I would recommend using a virtual machine where you can mess with the clock to your heart's content and it won't affect your development machine. Two free ones are Virtual PC from Microsoft and VirtualBox from Sun.
If this was my problem, I'd look for ways to avoid mucking with the system time. Isolating the code under unit tests, or a virtual machine, or something.
However, because I love PowerShell:
Get-ChildItem -r . |
? { $_.LastWriteTime -gt ([DateTime]::Now) } |
Set-ItemProperty -Name "LastWriteTime" -Value ([DateTime]::Now)
I don't know if this works in your situation but how about you don't move your clock forward, but wrap your gettime method (or whatever you're using) and make it return the future time that you need?
Install Unix Utils
touch temp
find . -newer temp -exec touch {} ;
rm temp
Make sure to use the full path when calling find or it will probably use Windows' find.exe instead. This is untested in the Windows shell -- you might need to modify the syntax a bit.
I don't use windows - but surely there is something like awk or grep that you can use to find the "future" timestamped files, and then "touch" them so they have the right time - even a perl script.
1) Use a build system that doesn't use timestamps to detect modifications, like scons
2) Use ccache to speed up your build system that does use timestamps (and rebuild all).
In either case it is using md5sum's to verify that a file has been modified, not timestamps.