I've been looking into Argo as a Gitops style CD system. It looks really neat. That said, I am not understanding how to use Argo in across multiple GCP projects. Specifically, the plan is to have environment dependent projects (i.e. prod, stage dev). It seems like Argo is not designed to orchestrate deployment across environment dependent clusters, or is it?
Your question is mainly about security management. You have several possibilities and several point of views/level of security.
1. Project segregation
The most simple and secure way is to have Argo running in each project without relation/bridge between each environment. No risk in security or to deploy on the wrong project. Default project segregation (VPC and IAM role) are sufficient.
But it implies to deploy and maintain the same app on several clusters, and to pay several clusters (Dev, Staging and prod CD aren't used at the same frequency)
In term of security, you can use the Compute Engine default service account for the authorization, or you can rely on Workload identity (preferred way)
2. Namespace segregation
The other way is to have only one project with a cluster deployed on it and a kubernetes namespace per delivery project. By the way, you can reuse the same cluster for all the projects in your company.
You still have to update and maintain Argo in each namespace, but the cluster administration is easier because the node are the same.
In term of security, you can use the Workload identity per namespace
(and thus to have 1 service account per namespace authorized in the delivery project) and to keep the permission segregated
Here, the trade off is the private IP access. If your deployment need to access to private IP inside the delivery project (for testing purpose or to access to private K8S master), you have to set up a VPC peering (and you are limited to 25 peering per project) or set up a shared VPC.
3. Service account segregation
The latest solution isn't recommended, but it's the easiest to maintain. You have only one GKE cluster for all the environment, and only 1 namespace with Argo deployed on it. By configuration, you can say to Argo to use a specific service account to access to the delivery project (with service account key files (not recommended solution) stored in GKE secrets or in secret manager, or (better) by using service account impersonation).
Here also, you have 1 service account authorized per delivery project. And the peering issue is the same in case of private IP access required in the delivery project.
Related
Now I am working with our company product developed with spring boot , angular and PostgreSQL technologies where front end angular is communicating with 138 back end ReST API end points. And these 138 end points are from 35 different spring boot project. And all these end points need to separately deploy for 5 different tenant. Actually end point working is same.But databases are different for different tenant. And we decided to go with AWS cloud. And we are looking for cost effective deployment method from AWS.
Our Current Development/Test strategy - Current we are developing application(final stage of development) and testing our application using our On-premise server. Here we are using 5 ubuntu machines. And we created kubernetes cluster with 2 master nodes and 3 worker nodes.And from our SVN repository and Jenkins server we implemented CI/CD pipeline deployment to this 5 machines.
Proposed Cloud Solution - Now we are thinking with to use either EKS deployment method or any of CodeDeploy/CodePipeline method to implement this big project.
So by considering cost and control over infrastructure management which solution is better for my product? Now I am not that much experienced as solution architect and still in cloud learning curve. So can any one suggest/guide me to think properly to achieve my goal please?
Company consideration
Control over infrastructure
Cost effective
Easy management of aws services for multi-tenant deployment
Data security ( Installing database on ec2/ RDS)
Management of load balances
Control over infrastructure
it would be better to manage it on Github, Gitlab, and or AWS code build, or cloud build.
indeed AWS code build, and repo is great tools but again consider the limitation of extra users it allows only 5 users if your team is very big you might have to pay to compare to managing projects at the Github & GitLab level.
Cost effective
EKS would be a good option compared to ECS or others as it has limitations of we can not run the Daemon set or Privilege PODs.
If you are looking for running everything On POD and auto-scalable with little less flexibility and don't want to manage much ECS also a good idea, but again you have to derive the capacity and compare both pricing ECS vs EKS.
Note : EKS will also charge the per hour charges $0.10 for each cluster + worker nodes. it's not just worker nodes like in on-prem we run.
Data security ( Installing database on ec2/ RDS)
RDS would be better as it's managed service compare to managing the EC2 and database performance and encryption etc.
it would be better to use RDS and EKS so the K8s service can connect to RDS easily on a private network.
RDS would be a cost-effective option considering the management of DB over EC2.
Management of load balances
NLB or ALB will take care of that you can use any of them as per the requirement with EKS.
Cloud front could be also a great option with cloud storage to serve static assets, which will reduce calls, improve performance and be cost-effective also.
I managed to get multicluster istio working following the documentation.
However this requires the kubeconfig of the clusters to be setup on each other. I am looking for an alternative to doing that. Based on presentation from solo.io and admiral, it seems that it might be possible to setup ServiceEntries to accomplish this manually. Istio docs are scarce in this this area. Does anyone have pointers on how to make this work?
There are some advantages to setting up the discovery manually or thru our CD processes...
if one cluster gets compromised, the creds to other clusters dont leak
allows us to limit the which services are discovered
I posted the question on twitter as well and hope to get some feedback from the Istio contributors.
As per Admiral docs:
Admiral acts as a controller watching k8s clusters that have a credential stored as a secret object which the namespace Admiral is running in. Admiral delivers Istio configuration to each cluster to enable services to communicate.
No matter how you manage contol-plane configuration (manually or with controller) - you have store and provision credentials somehow. In this case with use of the secrets
You can store your secrets securely in git with sealed-secrets.
You can read more here.
Can we run an application that is configured to run on multi-node AWS EC2 K8s cluster using kops (project link) into local Kubernetes cluster (setup using kubeadm)?
My thinking is that if the application runs in k8s cluster based on AWS EC2 instances, it should also run in local k8s cluster as well. I am trying it locally for testing purposes.
Heres what I have tried so far but it is not working.
First I set up my local 2-node cluster using kubeadm
Then I modified the installation script of the project (link given above) by removing all the references to EC2 (as I am using local machines) and kops (particularly in their create_cluster.py script) state.
I have modified their application yaml files (app requirements) to meet my localsetup (2-node)
Unfortunately, although most of the application pods are created and in running state, some other application pods are unable to create and therefore, I am not being able to run the whole application on my local cluster.
I appreciate your help.
It is the beauty of Docker and Kubernetes. It helps to keep your development environment to match production. For simple applications, written without custom resources, you can deploy the same workload to any cluster running on any cloud provider.
However, the ability to deploy the same workload to different clusters depends on some factors, like,
How you manage authorization and authentication in your cluster? for example, IAM, IRSA..
Are you using any cloud native custom resources - ex, AWS ALBs used as LoadBalancer Services
Are you using any cloud native storage - ex, your pods rely on EFS/EBS volumes
Is your application cloud agonistic - ex using native technologies like Neptune
Can you mock cloud technologies in your local - ex. Using local stack to mock Kinesis, Dynamo
How you resolve DNS routes - ex, Say you are using RDS n AWS. You can access it using a route53 entry. In local you might be running a mysql instance and you need a DNS mechanism to discover that instance.
I did a google search and looked at the documentation of kOps. I could not find any info about how to deploy to local, and it only supports public cloud providers.
IMO, you need to figure out a way to set up your local EKS cluster, and if there are any usage of cloud native technologies, you need to figure out an alternative way about doing the same in your local.
The true answer, as Rajan Panneer Selvam said in his response, is that it depends, but I'd like to expand somewhat on his answer by saying that your application should run on any K8S cluster given that it provides the services that the application consumes. What you're doing is considered good practice to ensure that your application is portable, which is always a factor in non-trivial applications where simply upgrading a downstream service could be considered a change of environment/platform requiring portability (platform-independence).
To help you achieve this, you should be developing a 12-Factor Application (12-FA) or one of its more up-to-date derivatives (12-FA is getting a little dated now and many variations have been suggested, but mostly they're all good).
For example, if your application uses a database then it should use DB independent SQL or no-sql so that you can switch it out. In production, you may run on Oracle, but in your local environment you may use MySQL: your application should not care. The credentials and connection string should be passed to the application via the usual K8S techniques of secrets and config-maps to help you achieve this. And all logging should be sent to stdout (and stderr) so that you can use a log-shipping agent to send the logs somewhere more useful than a local filesystem.
If you run your app locally then you have to provide a surrogate for every 'platform' service that is provided in production, and this may mean switching out major components of what you consider to be your application but this is ok, it is meant to happen. You provide a platform that provides services to your application-layer. Switching from EC2 to local may mean reconfiguring the ingress controller to work without the ELB, or it may mean configuring kubernetes secrets to use local-storage for dev creds rather than AWS KMS. It may mean reconfiguring your persistent volume classes to use local storage rather than EBS. All of this is expected and right.
What you should not have to do is start editing microservices to work in the new environment. If you find yourself doing that then the application has made a factoring and layering error. Platform services should be provided to a set of microservices that use them, the microservices should not be aware of the implementation details of these services.
Of course, it is possible that you have some non-portable code in your system, for example, you may be using some Oracle-specific PL/SQL that can't be run elsewhere. This code should be extracted to config files and equivalents provided for each database you wish to run on. This isn't always possible, in which case you should abstract as much as possible into isolated services and you'll have to reimplement only those services on each new platform, which could still be time-consuming, but ultimately worth the effort for most non-trival systems.
I'm working on a Datalake project composed by many services : 1VPC (+ subnets, security groups, internet gateway, ...), S3 buckets, EMR cluster, Redshift, ElasticSearch, some Lambdas functions, API Gateway and RDS.
We can say that some resources are "static" as they will be created only once and will not change in the future, like : VPC + Subnets and S3 buckets
The other resources will change during the developement and production project lifecycle.
My question is what's the best way to manage the structure of the project ?
I first started this way :
-modules
.rds
.main.tf
.variables.tf
.output.tf
-emr
-redshift
-s3
-vpc
-elasticsearch
-lambda
-apigateway
.main.tf
.variables.tf
So this way i only have to do a terraform apply and it deploys all the services.
The second option (i saw some developers using it) is that each service will be in a seperate folder and then we only go the folder of the service that we want to launch it and then execute terraform apply
We will be 2 to 4 developers on this project and some of us will only work on a seperate resources.
What strategy do you advice me to follow ? Or maybe you have other idea and best practice ?
Thanks for your help.
The way we do it is separate modules for each service, with a “foundational” module that sets up VPCs, subnets, security policies, CloudTrail, etc.
The modules for each service are as self-contained as possible. The module for our RDS cluster for example creates the cluster, the security group, all necessary IAM policies, the Secrets Manager entry, CloudWatch alarms for monitoring, etc.
We then have a deployment “module” at the top that includes the foundational module plus any other modules it needs. One deployment per AWS account, so we have a deployment for our dev account, for our prod account, etc.
The deployment module is where we setup any inter-module communication. For example if web servers need to talk to the RDS cluster, we will create a security group rule to connect the SG from the web server module to the SG from the RDS module (both modules pass back their security group ID as an output).
Think of the deployment as a shopping list of modules and stitching between them.
If you are working on a module and the change is self-contained, you can do a terraform apply -target=module.modulename to change your thing without disrupting others. When your account has lots of resources this is also handy so plans and applies can run faster.
P.S. I also HIGHLY recommend that you setup remote state for Terraform stored in S3 with DynamoDB for locking. If you have multiple developers, you DO NOT want to try to manage the state file yourself you WILL clobber each other’s work. I usually have a state.tf file in the deployment module that sets up remote state.
I am having a unique opportunity to suggest a workflow for IaC for a part of a big company which has number of technical agencies working for it.
I am trying to work out a solution that would be enterprise-level safe but have as much self-service as possible.
In scope:
Code management [repository per project/environment/agency/company]
Environment handling [build promotion/statefile per env, one statefile, terraform envs etc]
Governance model [Terraform Enterprise/PR system/custom model]
Testing and acceptance [manual acceptance/automated tests(how to test tf files?)/infra test environment]
I have read many articles, but most of them describe a situation of a development team in-house, which is much easier in terms of security and governance.
I would love to learn how what is the optimal solution for IaC management and govenance in enterprise. Is Terraform Enterprise a valid option?
I recommend using Terraform modules as Enterprise "libraries" for (infrastructure) code.
Then you can:
version, test, and accept your libraries at the Enterprise level
control what variables developers or clients can set (e.g. provide a module for AWS S3 buckets with configurable bucket name, but restricted ACL options)
provide abstractions over complex, repeated configurations to save time, prevent errors and encourage self-service (e.g. linking AWS API Gateway with AWS Lambda and Dynamodb)
For governance, it helps to have controlled cloud provider accounts or environments where every resource is deployed from scratch via Terraform (in addition to sandboxes where users can experiment manually).
For example, you could:
deploy account-level settings from Terraform (e.g. AWS password policy)
tag all Enterprise module resources automatically with
the person who last deployed changes (e.g. AWS caller ID)
the environment they used (with Terraform interpolation: "${terraform.workspace}")
So, there are lots of ways to use Terraform modules to empower your clients / developers without giving up Enterprise controls.