I am a newbie on aws and server in general. Hope someone can help me :)
What I want to do:
I have some image files on remote server and I would like to copy them to aws s3 bucket.
I can ssh this remote server once I ssh intermediate server.
so it looks like
me(local) <-----> server1 <----> server2 where image files on
What I did:
*I first ssh server1.
*I installed aws-cli on that server, and set configure.
*Then I entered aws s3 sync <username-I-use-for-server2>#<server2-IP>:/images/pc s3://<s3 bucket name>
Problem:
What I expected was to get password prompt from server2 then sync the files in images/pc to s3 bucket
but I got the error saying The user-provided path <username-I-use-for-server2>#<server2-IP>:/images/pc does not exist.
can anyone tell me What am I doing wrong?
I've checked that I can ssh server2 and that there's files inside /images/pc as well.
Any help would be appreciated!!
It is expected that this is a local path to the current server.
The available combinations are <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>.
You will need to run this command against a local path on the server you're trying to copy the objects from.
Remember that all this command is actually doing is running aws s3api putObject under the hood along with some management to perform operations such as recursion on your local disk. It does not handle any SSH interactions.
If you wanted to share any assets between servers you would need to mount them between servers (for example using an NFS mount) before you are able to sync them up, otherwise run the command on the other server.
Related
How to copy files from a remote server to google cloud storage bucket over SSH? This question outlines how to copy files from a google compute engine instance to google cloud storage which is not what I need.
The files are too large to download to my local machine before uploading them. Is it possible to do something like this:
ssh -p 2455 skiventist#myremoteserver | gsutil cp -r /path/to/remote_folder :gs//mybucket
Error:
CommandException: No URLs matched: /path/to/remote_folder
it doesn't matter if it's GCE or another server, they are both remote servers. you need to:
1- ssh into your remote server
2- make sure that you have gsutil installed and that you are authorized to manage the bucket.
3- copy the files to that remote server:
gsutil cp -r /path/to/remote_folder gs//mybucket
you can use also -m flag which will perform a parallel multi-threaded/multi-processing and make the transfer even faster.
check this doc for more info
In my Amazon EC2 instance, my application is located at /usr/local/apps/[my_app]. My problem is that I cannot transfer my file from local machine to the remote server. Supposed on my local path directory, I have a file named "dog.txt". On sftp connection with my Amazon EC2 instance, I can put dog.txt successfully; however, when I jump into /usr folder, I cannot even put dog.txt anymore. It said that
Uploading dog.txt to /usr/dog/txt
remote open("/usr/dog.txt"): Permission denied
Please help
it has nothing to do with AWS or EC2. It is a simple UNIX concept. To explain this in a simple way, just sftp all your files first to /tmp on remote server.
in order to solve your problem execute this command
$ chmod 777 -R /usr/local/apps/[my_app]
After this command is executed, you can copy files directly from your local machine or from /tmp.
I tried to follow advice provided at https://stackoverflow.com/a/18136205/6608952 but was unsure how to share myAmazonKeypair path in a .pem file on the remote server.
scp -i yourAmazonKeypairPath.pem fileNameThatYouWantToTransfer.php ec2-user#ec2-00-000-000-15.us-west-2.compute.amazonaws.com:
The command completed after a few minutes with this display:
ssh: connect to host
myBucketEndpointName
port 22: Connection timed out
lost connection
I have a couple of very large files to transfer and would prefer not to have to download the files to my local computer and then re-upload them to the S3 bucket.
Any suggestions?
There is no direct way to upload files to S3 from a remote location. i.e a URL
So to achieve that, you have two options :
Download the file on your local machine and then upload it via AWS Console or AWS CLI.
Download the file in AWS EC2 Instance and upload to S3 by AWS CLI.
The first method is pretty simple, not much explanation needed.
But for the second method, you'll need to do :
Create an EC2 Instance in the same region as the S3 Bucket is. Or if you already have an Instance, then login/ssh to it.
Download the file from the source to the EC2 Instance. via wget or curl whichever is comfortable.
Install AWS CLI on the EC2 Instance.
Create IAM User and Grant him Permission for your S3 Bucket.
Configure your AWS CLI with your IAM Credentials.
Upload your file to S3 Bucket with AWS CLI S3 CP Utility.
Terminate the Instance, if you set up the instance only for this.
Do it with shell script easily. If you have a list of URLs in files.txt do it like it is described here:
#!/bin/bash
input="files.txt"
while IFS= read -r line do
name=$(basename "$line")
echo $name
wget $line
aws s3 mv $name <YOUR_S3_URI>
done < "$input"
Or for one file:
wget <FILE_URL> | aws s3 mv <FILE_NAME> <YOUR_S3_URI>
I created a fresh Adobe Media Server 5 (aka Flash Media Server) instance on Amazon EC2 AWS, I connected over SSH and changed the default password, everything looks ok. But the amsadmin account doesn't seem to have enough access to administrate the instance. I can read most of the files and folders (access denied for /lib; /lost+found and /root).
I can connect ok
ssh -i key.pem amsadmin#public-dns.amazonaws.com
I can upload to the media folder
scp -i key.pem test.f4v amsadmin#public-dns.amazonaws.com:/mnt/applications/vod/media
I can't upload content to webroot; for example if I wanted to add a crossdomain.xml file to the web root.
scp -i key.pem crossdomain.xml amsadmin#public-dns.amazonaws.com:/mnt/webroot/
I can't create a directory; I want to setup S3 to store content, step 5 says to:
Create a directory on the instance to mount the S3 bucket, for example, /mnt/s3fs.
But under the amsadmin account I get an access denied error, is this expected?
How can I get access to these restricted areas?
Update:
I tried executing the commands with sudo and I entered the correct password for amsadmin, got "Sorry, try again".
Do I need to setup a root account?
Figured it out, I needed to run the commands using sudo sh after connecting to the instance via SSH.
How to use scp command to upload file to aws server
I have .pem file in /Downloads in local machine
I am trying to copy file to /images folder in AWS server
What command can i use ?
Thanks,
You can use plain scp:
scp -i ~/Downloads/file.pem local_image_file user#ec2_elastic_ip:/home/user/images/
You need to put an Elastic IP to the EC2 instance, open port 22 to your local machine IP in the EC2 instance security group, and use the right user (it can be ec2-user, admin or ubuntu (look at the AMI documentation)).
Diego's answer works.. However, if you're unaware of your elastic IP, then you can simply scp using following command (check the order of arguments)
scp -i path-to-your-identifier.pem file-to-be-copied ubuntu#public-IP:/required-path
just for reference, here ubuntu is your AWS user and public-IP is somewhat like 54.2xx.xxx.xxx e.g. 54.200.100.100 or such
(If order is messed up: filename before identifier, then you'll get a Permission denied (publickey).lost connection error)
Also, keep in mind the permissions of .pem file.. Should be 400 or 600. Not public to all.
Hope it helps!
there are number of ways to achieve what you want
use s3cmd http://s3tools.org/s3cmd
or use cyberduck http://cyberduck.ch/
or write a tool using amazon Java API
You can try kitten utility which is a wrapper around boto3. You can easily upload/download files and run commands on EC2 server or on multiple servers at once for that matter.
kitten put -i ~/.ssh/key.pem cat.jpg /tmp [SERVER NAME][SERVER IP]
Where server name is e.g ubuntu or ec2-user etc.
This will upload cat.jpg file to /tmp directory of server
This is the correct way uploading from local to remote.
scp -i "zeus_aws.pem" ~/Downloads/retail_market_db-07-01-2021.sql ubuntu#ec2-3-108-200-27.us-east-2.compute.amazonaws.com:/var/www/
Could be a better approach
Another alternative way to scp is rsync.
Some of the benefits of rsync
faster - uploads only the deltas
enable compression
you can exclude some files from the upload
resume
limit the transfer bandwidth
The rsync cmd
rsync -ravze "ssh -i /home/your-user/your-key.pem " --exclude '.env' --exclude '.git/' /var/www/your-folder-to-upload/* ubuntu#xx.xxx.xxx.xxx:/var/www/your-remote-folder
Now, in case you find this syntax a little bit verbose you can use aws-upload which does all the above but you just tabbing.