How to copy s3 objects into EKS pods directly - amazon-web-services

Question

I am working on Kubernetes pods and I have some pods running on an application. Now, I want to copy some files inside my pods but the situation here is my files present in my s3 bucket. So, I want to know an automated way that can directly copy my s3 files into my pod's folder directory. I don't know how to do this.
If anyone knows it then please reply.
Thanks

Answer 1

There are multiple ways to achieve this, few of them are:
Using Kubectl (very basic not automated method): first need to download s3 files on local and copy them into application pods can use kubectl command line to perform copy operation from host fs to pod fs.
kubectl cp <local-file> <pod-name>:<file-path>
Using Init-Container (Automated): Write a small script to download files from s3/cloud provider and place into shared volume between init-container and main container. In that way whenever your pod is spin up, the init-container will prepare volume before running actual container.
Sample Pod yaml:
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
volumeMounts:
- mountPath: /tmp
name: data
initContainers:
- name: init-downloader
image: busybox:1.28
command: ['sh', '-c', "aws s3 cp s3://xyz/ /tmp"]
volumeMounts:
- mountPath: /tmp
name: data
volumes:
- emptyDir: {}
name: data
Using CrobJob (Periodically Update): If the data needs to be updated in volume on periodic basis and not depends on pod restarts, then additional utility can be utilised to perform this action.
Either a script to download on local and perform kubectl copy operation on all pods
Use other opensource project to fullfill that, for example
https://github.com/maorfr/skbn, Skbn is a tool for copying files and directories between Kubernetes and cloud storage providers
Sample Job Yaml
apiVersion: batch/v1
kind: Job
metadata:
labels:
app: skbn
name: skbn
spec:
template:
metadata:
labels:
app: skbn
annotations:
iam.amazonaws.com/role: skbn
spec:
serviceAccountName: skbn
containers:
- name: skbn
image: maorfr/skbn
command: ["skbn"]
args:
- cp
- --src
- k8s://namespace/pod/container/path/to/copy/from
- --dst
- s3://bucket/path/to/copy/to
env:
- name: AWS_REGION
value: us-east-1
There can be multiple ways to implement this requirement but mostly it will be using above mentioned methods only.
Thanks

Answer 2

You can use the init container & cronjobs with AWS CLI to copy the files to S3 bucket.
To add further you can also use the Volume mount with datshim it's nice option :
apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
name: example-dataset
spec:
local:
type: "COS"
accessKeyID: "{AWS_ACCESS_KEY_ID}"
secretAccessKey: "{AWS_SECRET_ACCESS_KEY}"
endpoint: "{S3_SERVICE_URL}"
bucket: "{BUCKET_NAME}"
readonly: "true" #OPTIONAL, default is false
region: "" #OPTIONAL
https://github.com/datashim-io/datashim
If you are looking for something with FUSE you should also checkout the https://github.com/s3fs-fuse/s3fs-fuse