Can you help me with that script? I would like to transfer files
to my bucket on S3 AWS.
My code:
$cmd = "s3cmd -v -c /root/.s3cfg put /var/project_db_" . $date . ".sql.xz s3://bucket600";
My second code - What I'm using. And doesn't work.
$cmd = "s3cmd expire -v -c /root/.s3cfg put /var/project_db_" . $date . ".sql.xz --expiry-day=90 s3://bucket600";
Thank you for your help
I searched Google for s3cmd expire and the first result took me to this page which says this:
Advanced features
Object Expiration with s3cmd
You can set an object
expiration policy on a bucket, so that objects older than a particular
age will be deleted automatically. The expiration policy can have a
prefix, an effective date, and number of days to expire after.
s3cmd v2.0.0 can be used to set or review the policy:
[lxplus-cloud]$ s3cmd expire s3://dvanders-test --expiry-days 2
Bucket 's3://dvanders-test/': expiration configuration is set.
[lxplus-cloud]$ s3cmd getlifecycle s3://dvanders-test
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>ir0smpb610i0lthrl31jpxzegwximbcz3rrgb1he2yfxgudm</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>2</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Additional s3cmd expire options include:
--expiry-date=EXPIRY_DATE
Indicates when the expiration rule takes effect. (only
for [expire] command)
--expiry-days=EXPIRY_DAYS
Indicates the number of days after object creation the
expiration rule takes effect. (only for [expire] command)
--expiry-prefix=EXPIRY_PREFIX
Identifying one or more objects with the prefix to
which the expiration rule applies. (only for [expire]
command)
So you use the --expiry-date or --expiry-days command line option to do what you want.
(This question has nothing at all to do with Perl.)
when you have to place any file into the s3 bucket. You need to provide the location inside the s3 where you would want the file to be stored. If the specified folder(abstract) does not exist then it will be created.
In case you want to put the file to / (root) of the s3 then the destination would be set as "s3://bucket600/" (notice the slash at the end).
Related
I have files uploaded to S3 on a daily basis and I want to have a scheduled Lambda that checks the upload date of the file and deletes any files older then 7 days. I tried using S3 Object Lifecycle but that deletes all of my directory structure as well.
I have this working on a local server with a crontab that runs the following.
#daily find /share/sftp -type f -iname '*' -mtime +7 -delete
Does anyone have example code for this use case on AWS Lambda? Google hasn't come up with any relevant results.
Does it really matter if the folder-structure is deleted? If the question is of ACL, then permissions can be given to the user to create a folder( if not existing ) within a specific parent-folder of choice. The point here is, life-cycle is the ideal way to expire objects in S3.
I want to write an automated job in which the job will go through my files stored on the ec2 storage and check for the last modified date.If the date is more than (x) days the file should automatically get archived to my s3.
Also I don't want to convert the file to a zip file for now.
What I don't understand is how to give the path of the ec2 instance storage and the how do i put the condition for the last modified date.
aws s3 sync your-new-dir-name s3://your-s3-bucket-name/folder-name
Please correct me if I understand this wrong
Your requirement is to archive the older files
So you need a script that checks the modified time and if its not being modified since X days you simply need to make space by archiving it to S3 storage . You don't wish to store the file locally
is it correct ?
Here is some advice
1. Please provide OS information ..this would help us to suggest shell script or power shell script
Here is power shell script
$fileList = Get-Content "c:\pathtofolder"
foreach($file in $fileList) {
Get-Item $file | select -Property fullName, LastWriteTime | Export-Csv 'C:\fileAndDate.csv' -NoTypeInformation
}
then AWS s3 cp to s3 bucket.
You will do the same with Shell script.
Using aws s3 sync is a great way to backup files to S3. You could use a command like:
aws s3 sync /home/ec2-user/ s3://my-bucket/ec2-backup/
The first parameter (/home/ec2-user/) is where you can specify the source of the files. I recommend only backing-up user-created files, not the whole operating system.
There is no capability for specifying a number of days. I suggest you just copy all files.
You might choose to activate Versioning to keep copies of all versions of files in S3. This way, if a file gets overwritten you can still go back to a prior version. (Storage charges will apply for all versions kept in S3.)
I want to have multiple lifecycles for many folders in my bucket.
This seems easy if I use web interface but this should be an automated process so, at least in my case, it must use s3cmd.
It works fine when I use:
s3cmd expire ...
But, somehow, everytime I run this my last lifecycle gets overwrited.
There's an issue on github:
https://github.com/s3tools/s3cmd/issues/863
My question is: is there another way?
You made me notice I had the exact same problem as you. Another way to access the expire rules with s3cmd is to show the lifecycle configuration of the bucket.
s3cmd getlifecycle s3://bucketname
This way you get some xml formatted text:
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>RULEIDENTIFIER</ID>
<Prefix>PREFIX</Prefix>
<Status>Enabled</Status>
<Expiration>
<Days>NUMBEROFDAYS</Days>
</Expiration>
</Rule>
<Rule>
<ID>RULEIDENTIFIER2</ID>
<Prefix>PREFIX2</Prefix>
<Status>Enabled</Status>
<Expiration>
<Days>NUMBEROFDAYS2</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
If you put that text in a file, changing the appropriate fields (put identifiers of your choice, set the prefixes you want and the number of days until expiration), you now can use the following command (changing FILE for the path where you put the rules):
s3cmd setlifecycle FILE s3://bucketname
That should work (in my case, now I see several rules when I execute the getlifecycle command, although I do not know yet if the objects actually expire or not).
I have about 2 million files nested in subfoldrs in a bucket and want to move all of them to another bucket. Spending much of time on searching ... i found a solution to use AWS CLI mv/sync command. use move command or use sync command and then delete all the files after successfully synced.
aws s3 mv s3://mybucket/ s3://mybucket2/ --recursive
or it can be as
aws s3 sync s3://mybucket/ s3://mybucket2/
But the problem is how would i know that how many files/folders have moved or synced and how much time would it take...
And what if some exception occurs(machine/server stops/ internet disconnection due to any reason )...i have to again execute the command or it will for surely complete and move/sync all files. How can i be sure about the number of files moved/synced and files not moved/synced.
or can i have something like that
I move limited number of files e.g 100 thousand.. and repeat until all files are moved...
or move files on the basis of uploaded time.. e.g files uploaded from starting date to ending date
if yes .. how?
To sync them use:
aws s3 sync s3://mybucket/ s3://mybucket2/
You can repeat the command, after it finish (or fail) without issue. This will check if anything is missing/different to the target s3 bucket and will process it again.
The time depends on what size are the files, how much objects you have. Amazon counts directories as an object, so they matter too.
In S3 REST API I am adding metadata to an existing object by using the PUT (Copy) command and copying a key to the same location with 'x-amz-metadata-directive' = 'REPLACE'
What I want to do is change the download file name by setting:
Content-Disposition: attachment; filename=foo.bar;
This sets the metadata correctly but when I download the file it still uses the key name instead of 'foo.bar'
I use a software tool S3 Browser to view the metadata and it looks correct (apart from 'Content-Disposition' being all lower case as that's was S3 ask me to sign)
Then using S3 Browser I just pressed, then save without changing anything and now it works???
What am I missing how come setting a metadata 'Content-Disposition: attachment; filename=foo.bar;' from my web app does not work but does work from S3 Browser?
Edited for clarity:
Content-Disposition must be set explicitly and not included as x-amz-meta-Content-Disposition. All metadata header names must start with "x-amz-meta-" and be all lowercase.
Thanks to #Eyal for clarifying.
Original:
>SOLVED:
>
>The Doco at http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html
>
>seems to be wrong it says:
>
>Notice how the 'x-amz-' headers are sorted, white-space trimmed, converted tolowercase, and multiple headers with the same name have been joined using a comma toseparate values.
>
>Note how only the Content-Type and Content-MD5HTTPentity headers appear in the StringToSign .The otherContent-* entity headers do not.
However Content-Disposition must be set specifically and not included as : x-amz-meta-Content-Disposition
>
>It now works fine.
here: this uses the cli to set the content-disposition header on all files in a path inside a bucket (and also sets them as public):
aws s3 ls s3://mybucket/brand_img/ios/|awk {'print $4'} > objects.txt
while read line; do aws s3api copy-object --bucket mybucket \
--copy-source /mybucket/brand_img/ios/$line --key brand_img/ios/$line \
--metadata-directive REPLACE --metadata Content-Disposition=$line --acl public-read; done < objects.txt