I received an assignment from a company. For that assignment, I created a Kinesis Firehose Delivery Stream, S3 buckets, a Lambda function, table and views in Athena, and a Quicksight dashboard by using the AWS console web site.
Then I developed a Python script for sending test data to the Kinesis Firehose Delivery Stream
The company requested an easily reproducible environment.
I created a virtual environment and requirements.txt file for the Python code.
How can I create a reproducible environment for the AWS user, role, policies, lambda function and all resources (stream, buckets etc)?
Thanks!
To create a reproducible AWS environment, you have to leverage a concept called Infrastructure as Code (IaC). Using native AWS, this can be done using AWS CloudFormation (declarative, using JSON or YAML) or the AWS Cloud Development Kit (CDK) (imperative, using TypeScript, JavaScript, Python, Java, or C#).
AWS CloudFormation provides a common language for you to model and provision AWS and third party application resources in your cloud environment.
The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to model and provision your cloud application resources using familiar programming languages.
As a side note: If you're interviewing for a non-entry-level job involving AWS (which I assume, since you're expected to build something using Kinesis Firehose, Lambda functions, Athena, etc.), please be aware that you should be very familiar with the basic cloud concepts, like IaC.
Related
Sorry, I’m sort of a newbie when it comes to Amazon AWS Cloud so sorry if I sound naïve.
For .NET developers, I’ve used Visual Studio 2019’s AWS Lambda project to code Lambda functions and ultimately deploy it to Amazon AWS cloud
However, my concern is that there is No way to version and/or back up the Configurations for the AWS Services ( i.e., S3 bucket, Amazon SNS & SQS, etc.) that are invoked and/or trigger the various AWS Lambda Functions
The problem is that IT developers who configure said AWS Services have to use the ADFS AWS Services console website’s GUI to configure the various AWS Services, and if someone mistakenly deletes an AWS Services then they lose the configuration settings as well?
How do we go about versioning and/or backing up Configurations for the AWS Services?
There are Infrastructure as a Code frameworks like Terraform and Ansible designed to address that.
You can't really delete an AWS service.
it seems like you guys are kind of "new" to AWS so I will recommend using CloudFormation templates as Infrastructure As Code tool. All the configuration of how your AWS resources are supposed to look like can be added to the template and you deploy the template to create your AWS resources. Its AWS-Native and does not cost you anything.
On top of it, you also want to add your CloudFormation templates to version control system.
Is there an AWS equivalent to the Azure Building Blocks CLI?
Doing a Google search has not revealed anything so far.
I'm not familiar with Azure Building Blocks, but based on AWS vs Azure vs Google Cloud: Storage and Compute Comparison, the closest equivalent is AWS CloudFormation.
AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; AWS CloudFormation handles all of that. The following scenarios demonstrate how AWS CloudFormation can help.
If you want a programmatic equivalent to AWS CloudFormation, AWS Cloud Development Kit (AWS CDK):
Use the AWS CDK to define your cloud resources in a familiar programming language. The AWS CDK supports TypeScript, JavaScript, Python, Java, and C#/.Net.
We were doing most of our cloud processing (and still do) using AWS. However, we also now have some credits on GCP and would like to use and want to explore interoperability between the cloud providers.
In particular, I was wondering if it is possible to use AWS S3 from within GCP. I am not talking about migrating the data but whether there is some API which will allow AWS S3 to work seamlessly from within GCP. We have a lot of data and databases that are hosted on AWS S3 and would prefer to keep everything there as it still does the bulk of our compute.
I guess one way would be to transfer the AWS keys to the GCP VM and then use the boto3 library to download content from AWS S3 but I was wondering if GCP, by itself, provides some other tools for this.
From an AWS perspective, an application running on GCP should appear logically as an on-premises computing environment. This means that you should be able to leverage the services of AWS that can be invoked from an on-premises solution. The GCP environment will have Internet connectivity to AWS which should be a pretty decent link.
In addition, there is a migration service which will move S3 storage to GCP GCS ... but this is distinct from what you were asking.
See also:
Getting started with Amazon S3
Storage Transfer Service Overview
I am developing the application where "form" plays an important role. I use the "form" for data collection from the users which I store it on DynamoDB and then these data should be displayed in my application synchronously or immediately after some trigger functions when the data has been inserted into the dynamoDB.
What is the best way to achieve this? How should I frame my infrastructure on the AWS Cloud? What are the services should I rely on?
For "realtime" applications you should use something like graphQL.
On AWS you can use: AWS App Sync
https://console.aws.amazon.com/appsync/home?region=us-east-1#/
There are multiple ways you can achieve this. However, one of most modernized way of building your applications today is using a Serverless Architecture. You can host your website on S3 and can go serverless with sample architecture as below
(Note: you can just replace the Amazon Aurora with Dynamo DB in the architecture reference)
You can create a server less application on AWS using following AWS services:
AWS Lambda: AWS Lambda is a compute service that lets you run code without provisioning or managing servers.
AWS API Gateway: Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
AWS S3: AWS S3 is Object storage built to store and retrieve any amount of data from anywhere
AWS DynamoDB: Amazon
DynamoDB is a fast and flexible nonrelational database service for
all applications that need consistent, single-digit millisecond latency at any scale.
AWS Route53: For creating and registering a domain name for the web app.
AWS IAM: AWS IAM for creating users,roles and policies.
AWS Cognito: for authentication, access control to your web app.
I have an AWS S3 Bucket holding a development website. I would like to FTP(SSL) into the S3 Bucket, and also be able to create username and password credentials for others. Is this possible, and how can I do this?
Thanks!
Before giving up on S3 remember that sometimes frustration with a new product or technology comes from lack of knowledge and experience. The Amazon Cloud platform has some amazing services to work with.
FTP is an old technology that is not as popular today. The new style is using REST interfaces. S3 supports REST. Also you can easily copy files to / from S3 using command line tools. Look into the AWS Command Line Interface (CLI). Link below.
If your goal is to use S3 as your source repository look into AWS CodeCommit. Very similar to GIT. There is also CodePipeline, CodeBuild and CodeDeploy. Combine these tools with other Amazon services such as CloudFormation and you have real developer power.
AWS Command Line Interface
AWS Code Services
AWS CloudFormation