I have imported a vhd file from local network to the EC2 and have the data in the S3 bucket. It ran properly. I accidentally terminated the EC2 image that was created. I still have the data in the S3 bucket in parts. Can I use that data, or do I have to re-upload the image? It's 40gb and would take over one work day to push to the cloud.
Yes, if you do not specifically remove your file from your S3 bucket you should be able to use it again from any future EC2 instance. For example, you could use the aws-cli to "copy" the file from the S3 bucket to any number of EC2 instances.
If you had used the aws mv or aws rm command, then I would expect the file to be gone.
The bottom line is, if the file is still in the bucket, then you can still use it provided you have the permissions set correctly.
I re started the conversion task and re-uploaded the VM.
Related
I have a huge tar file in an s3 bucket that I want to decompress while remaining in the bucket. I do not have enough space on my local machine to download the tar file and upload it back to the s3 bucket. Whats the best way to do this?
Amazon S3 does not have in-built functionality to manipulate files (such as compressing/decompressing).
I would recommend:
Launch an Amazon EC2 instance in the same region as the bucket
Login to the EC2 instance
Download the file from S3 using the AWS CLI
Untar the file
Upload desired files back to S3 using the AWS CLI
Amazon EC2 instances are charged per-second, so choose a small machine (eg t3a.micro) and it will be rather low-cost (perhaps under 1 cent).
I want examples on how to backup an EC2 snapshots to S3 bucket, and import it back afterwards.
I found the AWS CLI can export the snapshots to S3, and was explained here
Copying aws snapshot to S3 bucket
I also found the import command from AWS CLI reference, but I failed to execute that command, as I don't follow understand the option
https://docs.aws.amazon.com/cli/latest/reference/ec2/import-snapshot.html
can someone explain how to use this command? especially on how to specific which file on the S3 bucket to import from?
EC2 snapshots are by default stored on S3 standard storage. However, you cannot copy the snapshot to a specific S3 bucket using the AWS CLI.
There may be some third party tool out there somewhere that can do it, but I do not see any reason why you would need to download a snapshot to your s3 bucket? It's like paying for the snapshot twice!!!
Could you mention why you have this requirement? An easier alternate to your problem might exist.
Note:
The two links that you shared in your question, do not copy a snapshot to S3.
The first link shows how to copy a snapshot from one region to another, while the second link is to export a disk image into an EBS snapshot and only the following disk formats are supported for this import:
Virtual Hard Disk (VHD/VHDX)
ESX Virtual Machine Disk (VMDK)
Raw
If I am reading your question correctly, you are having trouble with choosing the bucket from which to restore your backup. You might find this easier using the EC2 console.
In the console - Navigation bar - select Snapshots
Select the snapshot you want to copy from the list
Choose Copy from Action list, complete the dialog box and click Copy
When the confirmation dialog box comes up if you click Snapshots then you can monitor the progress.
Here's some additional information on AWS backups that might help you.
I'm currently debugging an ECS Task which basically grabs a message from a SQS queue, downloads a file from S3, manipulates it and uploads it back to S3. This script works fine locally and if I run it in a docker container it also works fine locally. When I create a task in ECS with the docker image and let it run it doesn't seem to process the file. In order to find the problem I created a very small script which simply uploads a file to s3
aws s3 cp hello-world.txt s3://my-buckt-name/hello-world.txt
This again works fine locally, and it works fine in a docker container (locally). When I create a task for it, it simply won't work. The ECS Task has a role that has "Full S3 Access"... any ideas?
Could it be that I need a bucket policy on my s3 bucket? I thought it would be sufficient if I grant access to the AWS services that need it, but apparently it's not working... and using my admin account I can use the awscli to do create objects in my bucket...
EDIT
Ok, it seems that the problem is the region. I created another bucket in a different region (it was Frankfurt before and now Ireland) and now I can copy and paste to the bucket as I would expect. Funnily I can create buckets programmatically (even from within my ECS task) but I can't seem the create objects in the buckets that are located in Frankfurt.
I have a zip file that is about 20 GB large and contains about 400'000 images that I was able to move to my EC2 instance by using wget. Now I want to unzip the files and save them to my S3.
Preferably it would be great if I didnt need to unzip them to the ec2 first. Can I by SSH somehow use unzip -options to extract each file to S3?
I have found answers like this https://stackoverflow.com/a/9722141/2335675. But I have no understanding of what he actually means by "unzipping it to S3". Can I do this while connected to my EC2 instance by SSH? Do Amazon have some kind of build in unzip command that extracts it to the s3 instead of the current server?
I can see other people have asked this questions, but I'm unable to find a direct answer of how to actually do it.
How I solved it:
I created a secondary volume on my EC2 instance to have space for the file x3 or so, to also include space for the extracted files. See guide here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-add-volume-to-instance.html
By being connected to the EC2 instance by SSH, I used the unzipcommand to unzip the file to the new volume.
I used aws s3 cp myfolder s3://mybucket/myfolder --recursive to move all my files into my S3 bucket.
I deleted my temporary volume and all files on it.
Everything was done using SSH. No script or programming was required.
Remember you need to use sudo to have permission to do many of the things.
The first solution:
Mount s3 on ec2 using s3fs.
Extract files to the mount point.
The second solution:
Using python and its aws library boto
extracting one file to the temporal location using zipfile
and uploading it to s3 using boto,
then delete the temporal file.
go to 2 while finishied
I would like to grab a file straight of the Internet and stick it into an S3 bucket to then copy it over to a PIG cluster. Due to the size of the file and my not so good internet connection downloading the file first onto my PC and then uploading it to Amazon might not be an option.
Is there any way I could go about grabbing a file of the internet and sticking it directly into S3?
Download the data via curl and pipe the contents straight to S3. The data is streamed directly to S3 and not stored locally, avoiding any memory issues.
curl "https://download-link-address/" | aws s3 cp - s3://aws-bucket/data-file
As suggested above, if download speed is too slow on your local computer, launch an EC2 instance, ssh in and execute the above command there.
For anyone (like me) less experienced, here is a more detailed description of the process via EC2:
Launch an Amazon EC2 instance in the same region as the target S3 bucket. Smallest available (default Amazon Linux) instance should be fine, but be sure to give it enough storage space to save your file(s). If you need transfer speeds above ~20MB/s, consider selecting an instance with larger pipes.
Launch an SSH connection to the new EC2 instance, then download the file(s), for instance using wget. (For example, to download an entire directory via FTP, you might use wget -r ftp://name:passwd#ftp.com/somedir/.)
Using AWS CLI (see Amazon's documentation), upload the file(s) to your S3 bucket. For example, aws s3 cp myfolder s3://mybucket/myfolder --recursive (for an entire directory). (Before this command will work you need to add your S3 security credentials to a config file, as described in the Amazon documentation.)
Terminate/destroy your EC2 instance.
[2017 edit]
I gave the original answer back at 2013. Today I'd recommend using AWS Lambda to download a file and put it on S3. It's the desired effect - to place an object on S3 with no server involved.
[Original answer]
It is not possible to do it directly.
Why not do this with EC2 instance instead of your local PC? Upload speed from EC2 to S3 in the same region is very good.
regarding stream reading/writing from/to s3 I use python's smart_open
You can stream the file from internet to AWS S3 using Python.
s3=boto3.resource('s3')
http=urllib3.PoolManager()
urllib.request.urlopen('<Internet_URL>') #Provide URL
s3.meta.client.upload_fileobj(http.request('GET', 'Internet_URL>', preload_content=False), s3Bucket, key,
ExtraArgs={'ServerSideEncryption':'aws:kms','SSEKMSKeyId':'<alias_name>'})