cannot access s3 bucket using vertica - amazon-web-services

Question

I am running vertica-ce in docker and I created a table , now i wanna export that table as parquet to s3 bucket, when i tried to export using EXPORT TO PARQUET(directory = 's3://s3-bucket-name/data') I got error ERROR 8198: Unable to verify if directory [s3://eucloid-vertica-migration/data/] exists due to 'Access Denied' I know why im getting this error because i need to give the access_key,secret_key but im unable to use awslib inside my docker container so i tried docker exec -it vertica-ce bash -l and accessed my container and tried to install the awscli but apt-get,yum,apk, nothing is working.
if any one have solution for this please let me know!!

Answer 1

You have a couple of options.
Set AWS parameters in the session.
There are a bunch of S3 settings that you can set. For instance, if you need to set is the access and secret key, you can do this:
=> ALTER SESSION SET AWSAuth='access_key:secret_key';
=> EXPORT TO PARQUET(directory = 's3://s3-bucket-name/data');
Depending on your setup, you may need to set additional config options (e.g. region, endpoint url, etc). All of the settings are documented here: https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/AdministratorsGuide/ConfiguringTheDB/S3Parameters.htm
Create a new version of the image that has aws cli in it.
docker-ce is currently a CentOS based OS, so it uses the yum package manager. You can create a new image using the sample Dockerfile:
FROM vertica/vertica-ce:latest
USER root
RUN set -x \
&& yum -q -y makecache \
&& yum install -y unzip \
&& yum clean all \
&& rm -rf /var/cache/yum \
&& cd /tmp \
&& curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscli.zip \
&& unzip awscli.zip \
&& /tmp/aws/install -i /usr/bin/aws-cli -b /usr/bin
USER dbadmin
Mount a volume so that you can access it from your host PC.
With this approach you can use the aws cli that you have installed locally to copy them to an s3 bucket.
Run this docker command to start the ce container persisting everything in /data to a local vertica-data directory.
docker run -p 5433:5433 \
--mount type=bind,source=$(pwd)/vertica-data,target=/data \
--name vertica_ce \
vertica/vertica-ce:latest
Run EXPORT TO PARQUET using the in-container path /data.
Access the parquet files from your PC in the vertica-data directory.