Is there any way I can launch e.g. 2 instances of the same cloud foundry app where each instance has its own instance specific settings passed down as parameters or environment variables e.g. instance1 subscribes to Kafka topic A and instance2 subscribes to Kafka topic B?
Is there any way I can launch e.g. 2 instances of the same cloud foundry app where each instance has its own instance specific settings passed down as parameters or environment variables e.g. instance1 subscribes to Kafka topic A and instance2 subscribes to Kafka topic B?
Not really. What you're talking about is two different applications, at least by the way Cloud Foundry defines an "application". For CF, your application will have source code, a droplet, environment variables, and everything required to run your app like services, which are the same for all instances of your application.
If you have the same source code that you would like to run with different configuration, services or environment variables, then you need to push that as a separate application (in Cloud Foundry's terminology).
The simplest way to do this would be to cf push to separate app names. It's a little redundant because you have to push the same code and stage it multiple times. I would suggest you go this route if it results in you pushing a handful of time. Where you might look at the next option is if you have lots (tens, hundreds or more) of different application configurations based off the same source code.
If you have a lot of different apps, what you can do is to download the droplet from your first app and then cf push --droplet for every other app. This will deploy the droplet that you've downloaded from the first app and reuse it with different configurations (probably by using different manifest.yml files) for all the copies of your app.
Hope that helps!
Related
I need to run a pipeline of data transformation that is composed of several scripts in distinct projects = Python repos.
I am thinking of using Compute Engine to run these scripts in VMs when needed as I can manage resources required.
I need to be able to orchestrate these scripts in the sense that I want to run steps sequentially and sometimes asyncronously.
I see that GCP provides us with a Worflows components which seems to suit this case.
I am thinking of creating a specific project to orchestrate the executions of scripts.
However I cannot see how I can trigger the execution of my scripts which will not be in the same repo as the orchestrator project. From what I understand of GCE, VMs are only created when scripts are executed and provide no persistent HTTP endpoints to be called to trigger the execution from elsewhere.
To illustrate, let say I have two projects step_1 and step_2 which contain separate steps of my data transformation pipeline.
I would also have a project orchestrator with the only use of triggering step_1 and step_2 sequentially in VMs with GCE. This project would not have access to the code repos of these two former projects.
What would be the best practice in this case? Should I use other components than GCE and Worflows for this or there is a way to trigger scripts in GCE from an independent orchestration project?
One possible solution would be to not use GCE (Google Compute Engines) but instead create Docker containers that contain your task steps. These would then be registered with Cloud Run. Cloud Run spins up docker containers on demand and charges you only for the time you spend processing a request. When the request ends, you are no longer charged and hence you are optimally consuming resources. Various events can cause a request in Cloud Run but the most common is a REST call. With this in mind, now assume that your Python code is now packaged in a container which is triggered by a REST server (eg. Flask). Effectively you have created "microservices". These services can then be orchestrated by Cloud Workflows. The invocation of these microservices is through REST endpoints which can be Internet addresses with authorization also present. This would allow the microservices (tasks/steps) to be located in separate GCP projects and the orchestrator would see them as distinct callable endpoints.
Other potentials solutions to look at would be GKE (Kubernetes) and Cloud Composer (Apache Airflow).
If you DO wish to stay with Compute Engines, you can still do that using shared VPC. Shared VPC would allow distinct projects to have network connectivity between each other and you could use Private Catalog to have the GCE instances advertize to each other. You could then have a GCE instance choreograph or, again, choreograph through Cloud Workflows. We would have to check that Cloud Workflows supports parallel items ... I do not believe that as of the time of this post it does.
This is a common request, to organize automation into it's own project. You can setup service account that spans multiple projects.
See a tutorial here: https://gtseres.medium.com/using-service-accounts-across-projects-in-gcp-cf9473fef8f0
On top of that, you can also think to have Workflows in both orchestrator and sublevel project. This way the orchestrator Workflow can call another Workflow. So the job can be easily run, and encapsuled also under the project that has the code + workflow body, and only the triggering comes from other project.
I'm new to AWS. For simplicity I have been working with a number of separate Elastic Beanstalk Environments, within the same Elastic Beanstalk Application
I have come to understand that Applications are just folders for Environments.
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.html
I'm just trying to get more organized. Is it even possible?
First of all, applications are not literally folders for environments, according to the documentation an application is "a logical collection of Elastic Beanstalk components".
The idea behind it is to have a single codebase that get deployed to several environments with different configuration depending on your needs. You can have several application versions (or the same version if you want it that way) deployed in several scenarios (environments) with varying configurations, let's say different environment variables, or different values for them, different infrastructure, etc. So, from my point of view there's not much logic in cloning an environment to a different application, since usually different applications would require different configurations.
But, if for your particular needs, you have very similar applications (they use the same platform, consume some of the same resources and basically you wanna deploy them in a very similar way infrastructure-wise) I think a good idea could be writing configuration files for your desired environment and just add those to your different application's code (you can find the official documentation on .ebextensions here, they even give you this link with several common configuration files). Another approach could be directly using CloudFormation templates as discussed here and just recreate the entire application from a CF template.
I hope this helped
Just for someone like me who came here for solutions...
Actually you can clone a environment from one Application to another Application with some steps.
Just follow this AWS - How do I migrate my Elastic Beanstalk environment from one AWS account to another AWS account?
This doc tells you how to do it between 2 AWS accounts but you can do it in just one account. The idea is just:
Save Config in one Application
Load Config in another Application
I was looking around aws code deploy to perform deployment. My application will have multiple services like apache, tomcat, database, cassandra, kafka and etc..
Each services will run in different machines. As per my knowledge it looks like i need to create different deployment group for each services(because each services running in different instances) and different deployment for each services.
So for example, if i have around 5 different services in my application and each running in different instance, do i need to create 5 deployment group and 5 different deployment in aws code deploy ? Is there any option to perform deployment for all the services using a single deployment/appspec file ? I would like to get some ideas from experts on how we can accomplish this effectively.
If I understand your use case correctly, you have an application that is made up of 5 different services. Every service runs on a different set of instances. You want to know if you can deploy all 5 services to a different set of instances with one deployment? That's not possible without doing some really hacky things in your appspec.yml, which would not be recommended. A deployment is tied to one deployment group and one revision.
The Recommended Way
If you want to do this with CodeDeploy, you can break it up with applications or deployment groups. There is no way to have a deployment group with a mix of different services for different instances.
Ideally, each service in your application has it's own CodeDeploy application, and each application would have one deployment group. You could use a deployment group for that as well in the same application, but conceptually, a deployment group is intended to break up a fleet with the same service running, though it doesn't really matter.
Installing applications you don't code
If you're talking about installing applications you don't own like a DB or Apache, the EC2 user data is probably the place you want to install that with.
I have an application that have several environments (all running in Amazon Elastic Beanstalk), namely, Production, Worker and Debug. Each environment have corresponding git branch that is different from master in some ways (like, configuration is changed and some code is deleted).
I use eb deploy to deploy the new version of application from its branch. It zips current git branch using git zip and sends the information to Amazon. Then it deploys to running instances.
The problem, however, is that deploying takes some time (about 5 minutes). Thus, between deploying, say, worker and production it have different code. Which is bad, because my changes might have change the queue protocol or something like that.
What I want is to be able to upload the information and to do its processing on all the environments, but not actually replace the code, just prepare it. And after I did it for all the environments issue command like "finish deploy" so that the code base is replaced on all the environments simultaneously.
Is there a way to do it?
You need to perform a "blue-green" deploy and not do this in-place. Because your deployment model requires synchronization of more than one piece, a change to the protocol those pieces use means those pieces MUST be deployed at the same time. Treat it as a single service if there's a frequently-breaking protocol that strongly binds the design.
"Deployed" means that the outermost layer of the system is exposed and usable by other systems. In this case, it sounds like you have a web server tier exposing an API to some other system, and a worker tier that reads messages produced by the web tier.
When making a breaking queue protocol change, you should deploy BOTH change-sets (web server layer and queue layer) to entirely NEW beanstalk environments, have them configured to use each other, then do a DNS swap on the exposed endpoint, from the old webserver EB environment to the new one. After swapping DNS on the webserver tier and verifying the environment works as expected, you can destroy the old webserver and queue tiers.
On non-protocol-breaking updates, you can simply update one environment or the other.
It sounds complex because it is. If you are breaking the protocol frequently, then your system is not decoupled enough to expect to version the worker and webserver tiers, which is why you have to do this complex process to version them together.
Hope this helps!
I'm planning on transfer an application from Heroku to AWS Elastic Beanstalk. On the Heroku, I have two different applications, one for staging and the other for production, and both have their web and workers dynos.
I'd like to setup something like that on AWS EB. I've read the difference about Web Tier and Worker Tier, but here goes some questions:
Do I setup two different applications for production and staging? Or the same application and two different environments? If so, I would have to create 4 environments, two for production web/worker and two for staging web/worker? What's the correct structure? I'll use the same Rails application for web and worker. In that case, will I have to deploy them separate or is there a command to deploy both environments together?
I'll use the same Rails application for web and worker.
This tells me that you should have a single application. Applications manage application versions, which is basically just deployment history.
You will want to create 4 environments. This allows you to "promote to prod" by cname swapping, or by deploying a previously deployed version.
You will have to deploy your web/worker separately, but you could very easily create a script that deploys to both at the same time.
For future reference, AWS Elastic Beanstalk later has created a solution for that called Environment Links:
https://aws.amazon.com/about-aws/whats-new/2015/11/aws-elastic-beanstalk-adds-support-for-environment-links/
With that feature, we're now able to link both environments with the same code (so we deploy it only once instead of twice). To make the worker run the worker process and the web to run the web server, you can set different environment variables and customize the EB initialization scripts to check for those vars and run specific process.