I am trying to create a job to connect sftp server from aws services to bring files into s3 storage in aws. It will be an automated job which runs every night and bring data into S3. I have seen documentation about how to connect aws and import data into S3 manually. However there is nothing I found about connecting external SFTP server to bring data into S3. I don't know if it is doable?
You can now use the managed SFTP service by AWS. It provides a fully managed SFTP server which is easy to setup and is reliable, scalable and durable. It uses S3 as backend for storing files.
Use S3FS to configure sftp connection directly to S3.
All you need to do is install S3FS
https://github.com/s3fs-fuse/s3fs-fuse/wiki/Installation-Notes
Install dependencies for fuse and s3cmd.
CentOS/RHEL Users:
# yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap
Ubuntu Users:
$ sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support
Download and Compile latest fuse
https://github.com/libfuse/libfuse/releases/download/fuse-2.9.7/fuse-2.9.7.tar.gz
# cd fuse-2.9.7
# ./configure --prefix=/usr/local
# make && make install
# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
# ldconfig
# modprobe fuse
Download and Compile latest S3FS
https://code.google.com/archive/p/s3fs/downloads
# cd /usr/src/
# wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
# tar xzf s3fs-1.74.tar.gz
# cd s3fs-1.74
# ./configure --prefix=/usr/local
# make && make install
4. setup Access Keys
# echo AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY > ~/.passwd-s3fs
# chmod 600 ~/.passwd-s3fs
Mount S3 Bucket
# mkdir /tmp/cache
# mkdir /s3mnt
# chmod 777 /tmp/cache /s3mnt
# s3fs -o use_cache=/tmp/cache mydbbackup /s3mnt
Make your mount point as ftp user home directory this will direct the files transferred using sftp to S3.
NOTE: Donot forget to add permissions to your S3 Bucket to allow Authenticated AWS users
Related
I am running vertica-ce in docker and I created a table , now i wanna export that table as parquet to s3 bucket, when i tried to export using EXPORT TO PARQUET(directory = 's3://s3-bucket-name/data') I got error ERROR 8198: Unable to verify if directory [s3://eucloid-vertica-migration/data/] exists due to 'Access Denied' I know why im getting this error because i need to give the access_key,secret_key but im unable to use awslib inside my docker container so i tried docker exec -it vertica-ce bash -l and accessed my container and tried to install the awscli but apt-get,yum,apk, nothing is working.
if any one have solution for this please let me know!!
You have a couple of options.
Set AWS parameters in the session.
There are a bunch of S3 settings that you can set. For instance, if you need to set is the access and secret key, you can do this:
=> ALTER SESSION SET AWSAuth='access_key:secret_key';
=> EXPORT TO PARQUET(directory = 's3://s3-bucket-name/data');
Depending on your setup, you may need to set additional config options (e.g. region, endpoint url, etc). All of the settings are documented here: https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/AdministratorsGuide/ConfiguringTheDB/S3Parameters.htm
Create a new version of the image that has aws cli in it.
docker-ce is currently a CentOS based OS, so it uses the yum package manager. You can create a new image using the sample Dockerfile:
FROM vertica/vertica-ce:latest
USER root
RUN set -x \
&& yum -q -y makecache \
&& yum install -y unzip \
&& yum clean all \
&& rm -rf /var/cache/yum \
&& cd /tmp \
&& curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscli.zip \
&& unzip awscli.zip \
&& /tmp/aws/install -i /usr/bin/aws-cli -b /usr/bin
USER dbadmin
Mount a volume so that you can access it from your host PC.
With this approach you can use the aws cli that you have installed locally to copy them to an s3 bucket.
Run this docker command to start the ce container persisting everything in /data to a local vertica-data directory.
docker run -p 5433:5433 \
--mount type=bind,source=$(pwd)/vertica-data,target=/data \
--name vertica_ce \
vertica/vertica-ce:latest
Run EXPORT TO PARQUET using the in-container path /data.
Access the parquet files from your PC in the vertica-data directory.
My Docker container works perfectly locally and using the default context and the command "docker compose up". I'm trying to run my docker image on ECS in AWS following this guide - https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/
I've followed all of the steps on the guide, after I've set the context to my new context (I've tried all 3 options) - after I run "docker compose up" I get the above error, here again for detail:
INFO trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
I've also set the user and added all of the permissions I can think of - image below
I've looked everywhere and I can't find traction, please help :)
The image is located on AWS ECS and Docker hub - I've tried both
Here is my Docker file:
FROM php:7.4-fpm
# Arguments defined in docker-compose.yml
ARG user
ARG uid
# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
curl \
libpng-dev \
libonig-dev \
libxml2-dev \
zip \
unzip
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
RUN curl -sS https://getcomposer.org/installer | php -- --
install-dir=/usr/local/bin --filename=composer
# Install PHP extensions
RUN docker-php-ext-install pdo_mysql mbstring exif pcntl bcmath
gd
# Get latest Composer
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
# Create system user to run Composer and Artisan Commands
# RUN useradd -G www-data,root -u $uid -d /home/$user $user
RUN mkdir -p /home/$user/.composer && \
chown -R $user:$user /home/$user
# Set working directory
WORKDIR /var/www
USER $user
I have a user-data bootstrap script that creates a folder called content in root directory and downloads files from an S3 bucket.
#!/bin/bash
sudo yum update -y
sudo yum search docker
sudo yum install docker -y
sudo usermod -a -G docker ec2-user
id ec2-user
newgrp docker
sudo yum install python3-pip -y
sudo pip3 install docker-compose
sudo systemctl enable docker.service
sudo systemctl start docker.service
export PATH=$PATH:/usr/local/bin
mkdir content
docker network create web_todos
docker run -d -p 80:80 --name nginx-proxy --network=web_todos -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy
aws s3 cp s3://jv-pocho/docker-compose.yaml .
aws s3 cp s3://jv-pocho/backup.sql .
aws s3 cp s3://jv-pocho/dns-updater.sh .
aws s3 sync s3://jv-pocho/images/ ./content/images
aws s3 sync s3://jv-pocho/themes/ ./content/themes
docker-compose up -d
sleep 30
docker exec -i db_jv sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < backup.sql
rm backup.sql
chmod +x dns-updater.sh
This bootstrap works ok, it creates the folder and download the files (it has permissions to download the files) i.e.:
download: s3://jv-pocho/dns-updater.sh to ./dns-updater.sh
[ 92.739262] cloud-init[3203]: Completed 32.0 KiB/727.2 KiB (273.1 KiB/s) with 25 file(s) remaining
so it's copying all the files correctly. The thing is that when i enter via SSH to the instance, i don't have any files inside
[ec2-user#ip-x-x-x-x ~]$ ls
[ec2-user#ip-x-x-x-x ~]$ ls -l
total 0
all commands worked as expected, all the yum installs, python, docker, etc were successfully installed, but no files.
Are files deleted after the bootstrap script ran?
thanks!
Try to copy them in a specific path, then look for it. Because here we don't know which path it's going to use.
Use the following command for specific path:
aws s3 cp s3://Bucket-name/Objet /Path
else you can do one thing,
use pwd command to get the current directory and print it using echo command so that you will get the present working directory.
I am trying to connect my raspberry pi4 device running raspy OS lite with AWS Iot Greengrass v2 and i do following steps:
From AWS Greengrass console i setup a core device
On my raspberry i install Java 8 runtime
$ sudo apt.get update
$ sudo apt-get install openjdk-8-jdk
On my raspberry i download the installer:
curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip > greengrass-nucleus-latest.zip && unzip greengrass-nucleus-latest.zip -d GreengrassCore
On my device i run the installer:
sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --aws-region eu-west-1 --thing-name GreengrassQuickStartCore-1773dec1ad2 --thing-group-name GreengrassQuickStartGroup --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true
All seems to be done, my core device was created in aws console and status is "Healty" but on my raspberry the folder /greengrass/v2 does not exist and i cannot see logs etc.
If i read documentation for troubleshooting device issues everyone report /greengrass/v2/logs/ as a log folder but on my device greengrass folder does not exist.
Everyone have some suggestion about?
So many thanks in advance
Did you install the AWS CLI V1 (the V2 version is not supported on the raspberry pi). Be sure to do this before installing Greengrass Core software.
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Had a similar error, be careful with the paths, sometimes you are using relative and absolute paths.
Example: GGv2 folder in the filesystem root directory (/greengrass/v2)
cd /greengrass/v2
Example: GGv2 folder relative to the current directory
cd ./greengrass/v2
Example: GGv2 folder at current user home directory (/usr/home/greengrass/v2)
cd ~/greengrass/v2
I assume your log files shall be located at filesystem root:
cd /greengrass/v2/logs
If you cannot access the logs folder, try changing its permissions:
sudo chmod 755 /greengrass/v2/logs
cd /greengrass/v2/logs
Actually i wants to use my own stun/Turn server instance and i want to use Amazon EC2 .If anybody has any idea regarding this please share with me the steps to create or any reference link to follow.
do an ssh login to your ec2 instance, then run the below commands for installing and starting the turn server.
simple way:
sudo apt-get install coturn
If you say no, I want the latest cutting edge, you can download source code from their downloads page in install it yourself, example:
sudo -i # ignore if you already in admin mode
apt-get update && apt-get install libssl-dev libevent-dev libhiredis-dev make -y # install the dependencies
wget -O turn.tar.gz http://turnserver.open-sys.org/downloads/v4.5.0.3/turnserver-4.5.0.3.tar.gz # Download the source tar
tar -zxvf turn.tar.gz # unzip
cd turnserver-*
./configure
make && make install
sample command for running TURN server:
turnserver -a -o -v -n -u user:root -p 3478 -L INT_IP -r someRealm -X EXT_IP/INT_IP --no-dtls --no-tls
command description:
-X - your amazon instance's external IP, internal IP: EXT_IP/INT_IP
-p - port to be used, default 3478
-a - Use long-term credentials mechanism
-o - Run server process as daemon
-v - 'Moderate' verbose mode.
-n - no configuration file
--no-dtls - Do not start DTLS listeners
--no-tls - Do not start TLS listeners
-u - user credentials to be used
-r - default realm to be used, need for TURN REST API
in your WebRTC app, you can use trun server like:
{
url: 'turn:user#EXT_IP:3478',
credential: 'root'
}
One method to install a turnserver on Amazon EC2 would be to choose Debian and to install the coturn package, which is the successor of the RFC5766-server.
The configuration file at /etc/turnserver.conf includes EC2 specific instructions. The information provided within this file is very exhaustive in general and should answer the majority of configuration questions.
Once configured, the coturn server can be stopped an started however you would any other service.