I'm looking to set up my staging server (many instances) to be able to spin up new instances at the press of a button. Ideally I'd just bring up a new docker instance whenever I need it, however each instance needs its own s3 and dynamo instance. If I have to I'll bring up real s3 and dynamodb instances through aws api or similar but I'd prefer to have containers to mimic s3 and dynamo. Any suggestions would be appreciated.
You can run localstack in a Docker container. Image can be found here.
LocalStack - A fully functional local AWS cloud stack
Then you need to override the AWS URL in the AWS SDK client to point to this container.
In Java it would look like this:
DynamoDbClient dynamoDbClient = DynamoDbClient.builder()
.endpointOverride(localstackUrl)
.build();
Related
Currently I have the ec2 in a private subnet within VPC. This ec2 has to get a "template-file" from a github repo. Ideally, I would like it to fetch the "template-file" only if changes are made. In short, the changes will tell the ec2 to fetch the new template. What is the best way to accomplish this?
I was thinking of using github-actions to sync the changes into S3 bucket, and have the ec2 constantly pull from it.
You can use SNS to handle the event of a new object created in the bucket and make sure that the EC2 is consuming this event.
You can try this approach:-
sync changes with the s3 bucket.
configure bucket notifications on upload using sns.
Run a script in your ec2 instance, which is running continuously and and checking whether there is object in sns or not, if yes download the updated file in your ec2 instance
As I'm following a multi-instance deployment strategy opposed to a multi-tenant, I'm deploying my entire infrastructure again for every new customer. This results in a lot of work as I have to
Deploy a new API instance on Elastic Beanstalk + env variables
Deploy a new webapp instance via s3
Deploy a new file storage via s3
Deploy a new backup file storage via s3
Setup a new data pipeline backing up the file storage to the backup bucket
Mapping the API and web app instance to a new customer-specific URL (e.g. mycustomer.api.mycompany.com and mycustomer.app.mycompany.com) via Route 53 + CloudFront
...
Is there a way to automate all of this deployment? I've looked into CodeDeploy by AWS but that doesn't seem to fit my needs.
The AWS tool that you can use to build infrastructure again and again is CloudFormation. We call this technique Infrastructure as a Code (IaaC). You can also use Terraform if you don't want to use AWS Specific tool.
You can use either YAML or JSON to define the template for your infrastructure.
And, you'll be using Git to do templates change management.
Watch this reinvent video to clear the whole picture.
I have one spring boot application which is in our internal data center, which process files from a specific folder on the host.
we wanted to deploy this to aws and wanted to use s3 bucket to upload files for processing.
is there any way we can add s3 bucket space as docker volume?
UPD: See at the bottom of this answer.
Other answers mistakenly say that AWS S3 is an object store and you can not mount it as volume to docker. Which is not correct. AWS S3 has a 3rd party FUSE driver, which allows it to be mounted as local filesystem and operate on objects as if those were files.
However it does not seem this FUSE driver has been made available as storage plugin for Docker just yet.
Edit: well, i have to correct myself after just a couple of minutes posting this. There in fact is a FUSE based driver for Docker to get volume mounted from AWS S3. See REX-ray and also here for possible configuration issue.
Other answers have correctly pointed out that :
AWS S3 is an object store and you can not mount it as volume to docker.
That being said, using S3 with spring application is super easy and there is framework developed called spring-cloud. spring-cloud works excellent with AWS.
Here is sample code :
public void uploadFiles(File file, String s3Url) throws IOException {
WritableResource resource = (WritableResource) resourceLoader.getResource(s3Url);
try (OutputStream outputStream = resource.getOutputStream()) {
Files.copy(file.toPath(), outputStream);
}
}
You can find detailed blog over here.
S3 is an object store, not a file system. You should have S3 trigger a message to SQS when new objects are added to the bucket. Then you can code your application running in the Docker container to poll SQS for new messages, and us the S3 location in the message to copy the object from S3 to local storage (using the appropriate AWS SDK) for processing.
No docker volume is for mounting drives on the machine (https://docs.docker.com/storage/volumes/)
You can use the S3 api to manage your bucket from the docker container (https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)
I want to use DynamoDB in an EC2 instance in Python. I have tested it locally, and set up my DynamoDB resource locally by using:
dynamodb = boto3.resource('dynamodb', aws_access_key_id=ACCESS_ID,
aws_secret_access_key= ACCESS_KEY, region_name='us-west-2', endpoint_url='http://localhost:8000')
I am wondering if, once it is running on an EC2 instance, the endpoint_url should be changed (to something different than http://localhost:8000), or if I should set up the resource in a completely different way. Thank you!
Firstly, you should avoid putting credentials in your source code. This can lead to security breaches and is difficult to update Instead:
When running on an Amazon EC2 instance: Assign an IAM Role to the instance. The code will automatically find credentials.
When running on your own system: Store credentials in the ~.aws/credentials file (or run aws configure to create the file).
If you wish to connect with DynamoDB, leave out the endpoint parameter. I assume that you have been using DynamoDB Local, which runs on your own computer. To use the 'real' DynamoDB, leave out the endpoint.
Also, it is a good idea to include a region, such as:
dynamodb = boto3.resource('dynamodb', region_name='ap-southeast-2')
I have a very restrictive policy on QA environment's aws in my company. I need a way to clean up dynamo db tables using jenkins. One of the thing I could think of using aws cli commands but I could not find a way to wipe out the content of dynamo db by just using aws cli. If there was a command I could have easily done it using aws cli commands on Jenkins. Any insights would be really helpful.
We had the same problem. Deleting bulk records is time consuming, costly proceess.
We delete the table and recreate it and recreate the data as needed.
I have not seen anything special with jenkins except running the aws cli shell script.
Hope it helps.
There are couple of concerns in terms, if you allow Jenkins to directly access DynamoDB and delete content, make sure to grant fine grained access control with IAM policies given to the AWS CLI execution to restrict permission to delete the data in particular tables.
Another approach is to use a SNS trigger (HTTP, Email & etc.) by Jenkins where it will run a Lambda function to delete the content. Here you do not need to give AWS DynamoDB Access Permission to your Jenkins Server and Script can be version controlled and managed outside Jenkins.
You can also consider using DynamoDB Local for Testing purposes.
The downloadable version of DynamoDB lets you write and test
applications without accessing the DynamoDB web service. Instead, the
database is self-contained on your computer. When you're ready to
deploy your application in production, you can make a few minor
changes to the code so that it uses the DynamoDB web service.