I understand that Kubernetes make great language-agnostic distributed computing clusters, easy to deploy, etc.
However, it seems that each platform has his own set of tools to deploy and manage Kubernetes.
So for example, If I use Amazon Elastic Container Service for Kubernetes (Amazon EKS), Google Kubernetes engine or Oracle Container Engine for Kubernetes, how easy (or hard) is to switch between them ?
"It depends". The core APIs of Kubernetes like pods and services work pretty much the same everywhere, or at least if you are getting into provider specific behavior you would know it since the provider name would be in the annotation. But each vendor does have their own extensions. For example, GKE offers integration with GCP IAM permissions as an alternative to Kuberenetes' internal RBAC system. If you use that, then switching is that much harder. The more provider-specific annotations and extensions you use, the more work it will be to switch.
Related
I have an EKS cluster that has a slightly modified kube-prometheus running on it for metrics monitoring(includes alertmanager, node exporters, prometheus-operator etc). I am thinking of replacing this setup with the AWS managed Prometheus & AWS managed Grafana. What are the upsides and downsides of making this change? One upside that I believe is this could free up my resources in the cluster to be used for actual applications and might also be a cheaper way of monitoring(not 100% sure on this though).
AWS Managed Prometheus is fully managed, secure and highly available. Customers only pay for the number of metrics ingested. You get improved scalability, availability and security without having to manage the underlying infrastructure.
AWS Managed Graphana is fully managed, scalable and highly available service providing native integration with AWS services including AWS SSO.
In both cases, you are getting managed services which provides native integration with AWS services and are based on open source standards.
Can we run an application that is configured to run on multi-node AWS EC2 K8s cluster using kops (project link) into local Kubernetes cluster (setup using kubeadm)?
My thinking is that if the application runs in k8s cluster based on AWS EC2 instances, it should also run in local k8s cluster as well. I am trying it locally for testing purposes.
Heres what I have tried so far but it is not working.
First I set up my local 2-node cluster using kubeadm
Then I modified the installation script of the project (link given above) by removing all the references to EC2 (as I am using local machines) and kops (particularly in their create_cluster.py script) state.
I have modified their application yaml files (app requirements) to meet my localsetup (2-node)
Unfortunately, although most of the application pods are created and in running state, some other application pods are unable to create and therefore, I am not being able to run the whole application on my local cluster.
I appreciate your help.
It is the beauty of Docker and Kubernetes. It helps to keep your development environment to match production. For simple applications, written without custom resources, you can deploy the same workload to any cluster running on any cloud provider.
However, the ability to deploy the same workload to different clusters depends on some factors, like,
How you manage authorization and authentication in your cluster? for example, IAM, IRSA..
Are you using any cloud native custom resources - ex, AWS ALBs used as LoadBalancer Services
Are you using any cloud native storage - ex, your pods rely on EFS/EBS volumes
Is your application cloud agonistic - ex using native technologies like Neptune
Can you mock cloud technologies in your local - ex. Using local stack to mock Kinesis, Dynamo
How you resolve DNS routes - ex, Say you are using RDS n AWS. You can access it using a route53 entry. In local you might be running a mysql instance and you need a DNS mechanism to discover that instance.
I did a google search and looked at the documentation of kOps. I could not find any info about how to deploy to local, and it only supports public cloud providers.
IMO, you need to figure out a way to set up your local EKS cluster, and if there are any usage of cloud native technologies, you need to figure out an alternative way about doing the same in your local.
The true answer, as Rajan Panneer Selvam said in his response, is that it depends, but I'd like to expand somewhat on his answer by saying that your application should run on any K8S cluster given that it provides the services that the application consumes. What you're doing is considered good practice to ensure that your application is portable, which is always a factor in non-trivial applications where simply upgrading a downstream service could be considered a change of environment/platform requiring portability (platform-independence).
To help you achieve this, you should be developing a 12-Factor Application (12-FA) or one of its more up-to-date derivatives (12-FA is getting a little dated now and many variations have been suggested, but mostly they're all good).
For example, if your application uses a database then it should use DB independent SQL or no-sql so that you can switch it out. In production, you may run on Oracle, but in your local environment you may use MySQL: your application should not care. The credentials and connection string should be passed to the application via the usual K8S techniques of secrets and config-maps to help you achieve this. And all logging should be sent to stdout (and stderr) so that you can use a log-shipping agent to send the logs somewhere more useful than a local filesystem.
If you run your app locally then you have to provide a surrogate for every 'platform' service that is provided in production, and this may mean switching out major components of what you consider to be your application but this is ok, it is meant to happen. You provide a platform that provides services to your application-layer. Switching from EC2 to local may mean reconfiguring the ingress controller to work without the ELB, or it may mean configuring kubernetes secrets to use local-storage for dev creds rather than AWS KMS. It may mean reconfiguring your persistent volume classes to use local storage rather than EBS. All of this is expected and right.
What you should not have to do is start editing microservices to work in the new environment. If you find yourself doing that then the application has made a factoring and layering error. Platform services should be provided to a set of microservices that use them, the microservices should not be aware of the implementation details of these services.
Of course, it is possible that you have some non-portable code in your system, for example, you may be using some Oracle-specific PL/SQL that can't be run elsewhere. This code should be extracted to config files and equivalents provided for each database you wish to run on. This isn't always possible, in which case you should abstract as much as possible into isolated services and you'll have to reimplement only those services on each new platform, which could still be time-consuming, but ultimately worth the effort for most non-trival systems.
I am very new to Kubernetes so apologies for gaps in my understanding and possibly incorrect wording.
I am developing on my local MacBook Pro, which is somewhat resource constrained. My actual payload is a database, which is already running in a Docker container, but obviously needs some sort of persistent storage.
The individual containers also need to talk to each over network and some of them need a channel (port open) to the outside world.
I would like to set up a single Kubernetes cluster for dev and testing purposes that I can later easily deploy to to bare metal servers or a cloud vendor - Google and AWS.
From reading so far it looks that I can, for example use minikube and orchestrate that cluster on top VirtualBox that I am already running.
How would that then map to an actual deployment in the cloud?
What additional tools do I need to get it all running, especially with regards to persistent storage and network?
Will it map easily to the cloud?
What configuration management software would you recommend to maintain all that configuration?
A very short answer is that it's hard to do this properly.
One of the best options I know of is LinuxKit, it allows you to build identical images that you can run on any of the popular cloud providers or in a data centre of your own, or desktop hypervisor. In fact, this is what Docker for Mac is based on.
Disclaimer: I am one of the LinuxKit contributors.
Generally you get more or less the same kubernetes, regardless of the method you spin up the cluster. Although, comparing to cloud, other deployments will usually lack in what cloud provides by default with kubes built-in cloud providers. Some very important features it relates to are things like out of the box support for LoadBalancer type of services or automatic PersistentVolume provisioning.
If you're ok with not having them, or configuring them additionally for your dev/test env then you should be quite fine.
In scope of PVC/PV, the lack of automatic PV provisioner (unless you set up something like ie. GlusterFS with Heketi to support this) will mean that you will have to provision every PV manualy on the dev/test cluster in opposite to ability of this happening in automatic fashion on cloud.
Also, as you begin, there are ought to be some minor differences between your dev/test setup and prod, so you might really want to investigate manifest templating and management solutions like helm from thew day one of your work with deployments to kubernetes. I know it would save m a lot of headache if I did that my self when I started doing kube.
Focusing a bit on your inquiry on the database, I think you have two options (assuming cloud is still an option for you):
use a docker database image and mount volumes
use an RDS instance in case of aws
I believe that in case of databases the case of volumes is generally not recommended.
What I would suggest you do is (once you grasp a bit the basic concepts, mainly Services, to
create an RDS instance and your needed databases therein
expose this RDS instance as a Service as type ExternalName
I have been doing the following and so far is working:
apiVersion: v1
kind: Service
metadata:
name: my-database-service
namespace: some-namespece
spec:
type: ExternalName
externalName: <my-rds-endpoint>
After that, you rest of k8s services can reach this service via my-database-service
I think this approach is more db-wise consistent and saves the volumes' hussle.
That being said, I acknowledge that the guidelines in terms of "select-this-if-you-go-for-cloud" or "that-if-you-go-on-prem" are not quite clear yet.
My experience so far indicates that:
most likely for on prem (not just your localhost) the way to go is kubeadm
for aws I have been having a pleasant experience with kops so far.
there is also the Canonical solution that seems to use a stack (conjure-up/juju) to help deploy their own slightly modified version of Kubernetes that they claim suits both cloud/on-prem (haven't tried it at all).
I'm wondering how people are deploying a production-caliber Kubernetes cluster in AWS and, more importantly, how they chose their approach.
The k8s documentation points towards kops for Debian, Ubuntu, CentOS, and RHEL or kube-aws for CoreOS/Container Linux. Among these choices it's not clear how to pick one over the others. CoreOS seems like the most compelling option since it's designed for container workloads.
But wait, there's more.
bootkube seems to be next iteration of the CoreOS deployment technology and is on the roadmap for inclusion within kube-aws. Should I wait until kube-aws uses bootkube?
Heptio recently announced a Quickstart architecture for deploying k8s in AWS. This is the newest approach and so probably the least mature approach but it does seem to have gained traction from within AWS.
Lastly kubeadm is a thing and I'm not really sure where it fits into all of this.
There are probably more approaches that I'm missing too.
Given the number of options with overlapping intent it's very difficult to choose a path forward. I'm not interested in a proof-of-concept. I want to be able to deploy a secure, highly-available cluster for production use and be able to upgrade the cluster (host OS, etcd, and k8s system components) over time.
What did you choose and how did you decide?
I'd say pick anything which fit's your needs (see also Picking the right solution)...
Which could be:
Speed of the cluster setup
Integration in your existing toolchain
e.g. kops integrates with Terraform which might be a good fit for some prople
Experience within your team/company/...
e.g. how comfortable are you with the related Linux distribution
Required maturity of the tool itself
some tools are very alpha, are you willing to play to role of an early adaptor?
Ability to upgrade between Kubernetes versions
kubeadm has this on their agenda, some others prefer to throw away clusters instead of upgrading
Required integration into external tools (monitoring, logging, auth, ...)
Supported cloud providers
With your specific requirements I'd pick the Heptio or kubeadm approach.
Heptio if you can live with the given constraints (e.g. predefined OS)
kubeadm if you need more flexibility, everything done with kubeadm can be transferred to other cloud providers
Other options for AWS lower on my list:
Kubernetes the hard way - using this might be the only true way to setup a production cluster as this is the only way you can fully understand each moving part of the system. Lower on the list, because often the result from any of the tools might just be more than enough, even for production.
kube-up.sh - is deprecated by the community, so I'd not use it for new projects
kops - my team had some strange experiences with it which seemed due to our (custom) needs back then (existing VPC), that's why it's lower on my list - it would be #1 for an environment where Terraform is used too.
bootkube - lower on my list, because it's limitation to CoreOS
Rancher - interesting toolchain, seems to be too much for a single cluster
Offtopic: If you don't have to run on AWS, I'd also always consider to rather run on GCE for production workloads, as this is a well managed platform rather than something you've to build yourself.
I'm facing a choice terraform of gcloud deployment manager.
Both tools provide similar functionality and unfortunately lacks all resources.
For example:
gcloud can create service account (terraform cannot)
terraform can manage DNS record set (gcloud cannot)
and many others ...
Questions:
Can you recommend one tool over the other?
What do you think, which tool will have a richer set of available resources in long run?
Which solution are you using in your projects?
Someone may say this is not a question you should ask on stackoverflow, but I will answer anyway.
It is possible to combine multiple tools. The primary tool you should run is Terraform. Use Terraform to manage all resources it supports natively, and use external provider to invoke gcloud (or anything else). While it will be not very elegant sometimes it will make the work.
Practically I do same approach to invoke aws-cli in external.
I personally found deployment manager harder to get started with for what I wanted to do. Although I had previous experience with terraform, therefore I may be biased. Terraform for me was easier.
Thats said though, the gcloud command line tool is extremely good and as Anton has said, you can feed that in when you need it via external. Also note, this is what terraform does and has been doing for a long time. They are also quite good in my experience of adding new features etc. Yes Gcloud Deployment Manager might have them first, as its google in house, but terraform would never be far behind.
In the long run terraform may be easier to integrate with other services, and there's always the options of going to other providers. On top of that, you have one configuration format to use. As this is what terraform does, I find the way you structure and work with it very logical and easily understood. Something thats valuable if your going to be sharing and working with other team members.
Deployment Manager is a declarative deployment orchestration tool specifically for Google Cloud Platform. So, if you're all in on Google, or just want to automate your processes on our infrastructure, you can certainly do so with Deployment Manager. Deployment Manager also allows you to integrate with other GCP services such as Identity Access Management. Cross platform alternatives such as Puppet, Chef, and Terraform work across multiple cloud providers. They aren't hosted, and you're ending up setting up your own infrastructure to support those. Cloud Formation from AWS is only structured to work within AWS infrastructure, and it integrates well with AWS services.