I am using an EC2 instance and I have enabled the log service of Elastic Load Balancer. The logs are stored in Amazon S3 and I want that data to be used as dataset for Elasticsearch which is present on my EC2 instance. Is there a way I can transfer the data to my EC2 instance or access the data directly from S3 only to be used for Elasticsearch ?
The AWS Command Line Interface (CLI) has commands that make it easy to copy to/from Amazon S3.
You can run them on your EC2 instance to download data from Amazon S3.
aws s3 cp s3://bucket/path/file .
aws s3 cp s3://bucket/path . --recursive
aws s3 sync s3://bucket/path .
Related
I'm using AWS CLI to download big file from S3 bucket ( heroku db backup ) to my EC2 instance and upload big file ( about 110gb) to my AWS S3 bucket from my AWS EC2
Problem is when download ( from bucket that I don't own ) or upload to my bucket ( they are all in same region us-east-1 ) . Speed when download/upload start at about 60MB/s but decrease to 7-8MB/s after first 15gb even when I enable transfer acceleration .
So this is problem with AWS CLI config or my EC2 instance ( I'm testing with t2.micro ) ?Tks
Here's a list of items you can try:
Use enhanced networking on the EC2 instance.
Use parallel workloads for the data transfer.
Customize the upload configurations on the AWS Command Line Interface (AWS CLI).
Use an Amazon Virtual Private Cloud (Amazon VPC) endpoint for Amazon S3.
Upgrade your EC2 instance type.
Use chunked transfers.
Read more here: https://aws.amazon.com/premiumsupport/knowledge-center/s3-transfer-data-bucket-instance/
How can I go about downloading from this s3 bucket:
aws s3 cp --request-payer requester s3://adatascientist/parler/v1/post.ndjson.zip
I am not very technically minded, do I need a particular software or programme to do this? I have already created my AWS account I am just not sure on the next steps.
The website is here for any extra information: https://parler.adatascienti.st/research
Assuming that you have installed the AWS Command-Line Interface (CLI), then:
Use aws configure and provide your Access Key and Secret Key
Use aws s3 cp --request-payer requester s3://adatascientist/parler/v1/post.ndjson.zip . to copy the file to the current local directory
There is no Data Transfer charge if you do this from an Amazon EC2 instance in the same Region. If you download to your own computer, you will be charged for Data Transfer (9c/GB).
The question is mostly self-explanatory.
If I run aws s3 cp <s3_bucket> <ec2_host> locally (assuming both the bucket and the instance are on the same region), does the data go from s3 to ec2 internally (in which case there are no data transfer costs), or does it go first from the bucket to my computer, and only then to the instance (in which case there are s3 download data transfer costs)?
Thanks a lot.
AWS S3 pricing writes:
Transfers between S3 buckets or from Amazon S3 to any service(s) within the same AWS Region are free.
Please note that if you are using NAT gateway to access S3 from private subnet you will pay for NAT transfers. Use VPC S3 gateway endpoint instead. So if you are using aws s3 cp from private subnet through NAT, you will pay for them.
The documentation for the cp command states this at the beginning:
cp
<LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
In other words, each of the x and y in aws s3 cp <x> <y> can be either a S3 URI or a local reference. There is no option there to point it at an EC2 instance. If you did try to run aws s3 cp s3://bucketname/key.name 1.2.3.4, you would just end up with a file called 1.2.3.4 on your machine.
I'm familiar with running the AWS CLI command to copy from a folder to S3 or from one S3 bucket to another S3 bucket:
aws s3 cp ./someFile.txt s3://bucket/someFile.txt
aws s3 cp s3://bucketSource/someFile.txt s3://bucketDestination/someFile.txt
But is it possible to copy files from S3 to an EC2-Instance when you're not on the EC2-Instance? Something like:
aws s3 cp s3://bucket/folder/ ec2-user#1.2.3.4:8080/some/folder/
I'm trying to run this from Jenkins which is why I can't simply run the command on the EC2 like this:
aws s3 cp s3://bucket/folder/ ./my/destination/folder/on/the/ec2
Update:
I don't think this is possible so I'm going to look into using https://docs.aws.amazon.com/cli/latest/reference/ssm/send-command.html
No.
The AWS CLI calls the AWS API. The APIs for Amazon S3 do not have the ability to interact with the operating system on an Amazon EC2 instance.
Your idea of using AWS Systems Manager is a good idea. You can send the command to the instance itself, and the instance can then upload/download objects to Amazon S3.
Since you have SSH access, you could also just run
ssh ec2-user#1.2.3.4:8080 "aws s3 cp s3://bucket/folder/ ./my/destination/folder/on/the/ec2"
... to run the command on the EC2 instance directly.
It's not as efficient as using send-command (because ssh will necessarily pipe the output of that command to your local terminal) but, if you're not transferring millions of files, the tradeoff in simplicity may be acceptable for you.
Using AWS System Manager send command :
#Copying file from S3 bucket to EC2 instance :
$Instance_Id='i-0123456xxx'
aws ssm send-command --document-name "AWS-RunShellScript" --document-version "\$DEFAULT" --targets "Key=instanceids,Values='$Instance_Id'" --parameters '{"commands":["aws s3 cp s3://s3-bucket/output-path/file-name /dirName/ "]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region REGION_NAME
I'm new to cloud infrastructure for Deep Learning and trying to use AWS for deep learning first time and I don't know how to access my data from EC2 launched instance.
My data is stored is S3 bucket but I'm not able to find a way how to get it together and start training.
In that EC2 instance. login via ssh.
install aws cli if its not there
configure credentials are add permission for ec2 instance to use s3 bucket.
otherwise add aws secret and access key
get files to your local system
aws s3 cp s3://mybucket/test.txt test2.txt
Get files from local to s3
aws s3 cp test.txt s3://mybucket/test2.txt
https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#examples