I'm getting ready to setup HashiCorp Vault with my web application, and while the examples HashiCorp provides make sense, I'm a little unclear of what the intended production setup should be.
In my case, I have:
a small handful of AWS EC2 instances serving my web application
a couple EC2 instances serving Jenkins for continuous deployment
and I need:
My configuration software (Ansible) and Jenkins to be able to read secrets during deployment
to allow employees in the company to read secrets as needed, and potentially, generate temporary ones for certain types of access.
I'll probably be using S3 as a storage backend for Vault.
The types of questions I have are:
Should vault be running on all my EC2 instances, and listening at 127.0.0.1:8200?
Or do I create an instance (maybe 2 for availability) that just run Vault and have other instances / services connect to those as needed for secret access?
If i needed employees to be able to access secrets from their local machines, how does that work? Do they setup vault locally against the S3 storage, or should they be hitting the REST API of the remote servers from step 2 to access their secrets?
And to be clear, any machine that's running vault, if it's restarted, then vault would need to be unsealed again, which seems to be a manual process involving x number of key holders?
Vault runs in a client-server architecture, so you should have a dedicated cluster of Vault servers (usually 3 is suitable for small-medium installations) running in availability mode.
The Vault servers should probably bind to the internal private IP, not 127.0.0.1, since they they won't be accessible within your VPC. You definitely do not want to bind 0.0.0.0, since that could make Vault publicly accessible if your instance has a public IP.
You'll want to bind to the IP that is advertised on the certificate, whether that's the IP or the DNS name. You should only communicate with Vault over TLS in a production-grade infrastructure.
Any and all requests go through those Vault servers. If other users need to communicate with Vault, they should connect to the VPC via a VPN or bastion host and issue requests against it.
When a machine that is running Vault is restarted, Vault does need to be unsealed. This is why you should run Vault in HA mode, so another server can accept requests. You can setup monitoring and alerting to notify you when a server needs to be unsealed (Vault returns a special status code).
You can also read the production hardening guide for more tips.
Specifically for point 3 & 4:
They would talk to the Vault API which is running on one/more machines in your cluster (If you do run it in HA mode with multiple machines, only one node will be active at anytime). You would provision them with some kind of authentication, using one of the available authentication backends like LDAP.
Yes, by default and in it's recommended configuration if any of your Vault nodes in your cluster get restarted or stopped, you will need to unseal it with however many keys are required; depending on how many key shards were generated when you stood up Vault.
Related
We are developing applications and devices that communicate with our servers. We have one "main" Java Spring server which handles almost all the HTTP requests including user authentication, storing relevant user data and giving that data to the applications. Furthermore, we have a few smaller HTTP servers (written in golang) which are both used by the "main" server to perform certain tasks but also have some public API's that apps and devices use directly.
In our current non-production setup we run all the servers locally on one machine with an apache2 in front which directs the requests. So the servers can be accessed via the apache2 by a user by their respective subdomains but they also perform some communication between each other. When doing so, currently we simply send the request to localhost:{PORT} since they all run on the same machine. They furthermore all utilize the same mysql-server running on that same machine.
We are now looking to get it more production-ready and are looking to deploy it to AWS. They are currently not containerized so a solution that requires containerization (ECS? K8s?) would most likely require more work. What would be the most straightforward way to do the following:
Deploy a number of servers on AWS where they are exposed publicly with their respective domains but can also communicate internally with one another (or would they just communicate with one another using their public domains?)
Deploy a managed SQL database (Amazon RDS?) which is accessible for all the servers.
Setup the routing of the requests. Currently run our own configured apache2 but I assume we can add a managed API Gateway in AWS and configure it for our servers.
Q. Deploy a number of servers on AWS where they are exposed publicly
with their respective domains but can also communicate internally with
one another (or would they just communicate with one another using
their public domains?)
On AWS you create a VPC(1st default VPC is created when you login for the first time).
You can deploy a number of EC2 instances(virtual servers) with just private IP addresses and without any public access and put them behind an ELB(elastic load balancer). The ELB will take all the traffic and distribute the load onto the servers based on endpoint.
However the EC2 instances won't have public IPs A VPC(virtual Private Gateway) allows your services to communicate to each other via private IPs (something like 172.31.xx.xx), You can also provide domain/sub-domain names to these private IP addresses using Route53 service of AWS.
For example You launch 2 servers:
Your Java Application - on 172.31.1.1 (you name it
xyz.myjavaapp.something.com on Route53)
Your Angular Application - on 172.31.1.2
The angular application can reach your java application on 172.31.1.1:8080 or
xyz.myjavaapp.something.com:8080
Q. Deploy a managed SQL database (Amazon RDS?) which is accessible for
all the servers.
Yes you can deploy an SQL database on RDS and it will be available to the EC2 instances. Just make sure you create proper security groups to allow only your servers to access it, and not leave it open for public internet.
Example for a VPC only security group entry is 172.31.0.0/16 This will allow only ther servers in you VPC to connect to the RDS DB. given that your VPC subnet has the range 172.31.x.x
Q. Setup the routing of the requests. Currently run our own configured
apache2 but I assume we can add a managed API Gateway in AWS and
configure it for our servers.
You can set up public/private APIs and manage different endpoints using API Gateway.
Another way it to put your application server behind an Application ELB. The ELB can take care of load balancing as well as endpoint management.
for example :
if you decide to deploy 2 servers for /getData and 1 server for /doSomethingElse. It can be easily managed by ELB.
I would suggest you use at-least servers for critical services and load balance them behind and ELB for production env.
On another note, containerizing and deploying to kubernetes is not that difficult or time consuming. But yes it has got some learning curve, but the benefits outweigh it.
Feel free to ask questions.
I am setting up a Tomcat application in EC2. For reliability, I am running two or more instances. If one server goes down, my users should be redirected to the other instance. This suggests that session state should be kept in an external source, or mirrored between the servers.
AWS offers a hosted service, Elasticache, which seems like it would work well. I even found a nice library, memcached-session-manager. However, I soon ran into some issues.
Unless someone can convince me otherwise, I need the session states to be encrypted in transit. Otherwise someone could intercept the network traffic and pretend to be someone else on my site. I don't see any built-in Amazon method to keep traffic off the internet. (Is peering available here?)
The library mentioned earlier does have Redis support with SSL, but it does not support a Redis cluster. Someone put in a pull request for this but it has not been incorporated and this library is a complex build. I may talk myself into living without the cluster, but that puts us back at a single point of failure.
Tomcat is running on EC2 in your VPC, and ElastiCache is in your VPC. Your AWS VPC is an isolated network. Nobody can intercept the traffic between the EC2 and Elasticache servers unless your VPC network becomes compromised in some way.
If you want to use Redis instead, with SSL connections, then I believe at this time you would need a Tomcat Session Manager implementation that uses Jedis. This one uses Jedis, but you would need to upgrade the version of Jedis it uses in order to use SSL connections.
AWS seems a little daunting with too many overlapping services so I'm looking for some advice and direction.
We have a mobile app for which we've developed a sync server (i.e. user will sign-up, sync data kept on AWS). Currently we've setup an EC2 instance with a web server, Django end-points and a postgres server. However we need the following:
Ensure the service is available from different regions of the
world for faster access
If that requires putting the postgres server outside of the EC2, what service do we need and how would replication work?
We will have larger file attachments stored on S3 separately, but need to do this securely and encrypt the files
Eventually we will host a web-app (i.e. an Angular 2 app) that would
connect to the same database.
We also would need to do all this in the most economical way and then scale up as the load increases.
Please any guidance would be appreciated. I'm struggling with terminologies at the moment. We also setup an Amazon SSL Certificate however that requires an Elastic Load Balancer but we only have one EC2 instance. What do we do to get this all working securely?
Based on the information provided, I would recommend you to start with AWS Elastic Beanstalk, where it will manage autoscaling and loadbalancing while providing you with a DNS URL for external domain mapping.
To ensure that the service is available from different regions for faster access, you can cache the static Angular App using Cloudfront. Then you will be able to add SSL Certificate to Cloudfront instead of ELB. If you plan to create multiple environments for different regions, you can use Route53 for geo based routing.
To take Postgres server outside EC2, you can use AWS RDS and it supports synchronous replication with fail-over for Multi-AZ deployments and also Postgres in RDS also supports Cross Region Replication if you plan to setup multiple deployment environments in different regions. Also you can create Read Replicas to improve reading speeds which will be asynchronously replicated.
You can encrypt the files in S3 using AES256 using Keys from KMS or from your client and I would recommend using Signed URLs with Cloudfront in front of S3 serving these files, so that clients can securely and directly access them improving the performance by getting advantage from distributed caching.
You can host the Angular App in AWS S3 and Cache using Cloudfront for faster access. Another option is to cache the static asset path in Cloudfront so that subsequent requests for static assets will be served from Cloudfront.
FAQs from Amazon
Who should use AWS Elastic Beanstalk?
Those who want to deploy and manage their applications within minutes
in the AWS Cloud. You don’t need experience with cloud computing to
get started. AWS Elastic Beanstalk supports Java, .NET, PHP, Node.js,
Python, Ruby, Go, and Docker web applications.
Your current environment isn't scalable (either load-responsive or to another region). If you need scalability then it should be re-arranged. It is difficult to provide you with details because the required environment depends on the applications architecture, however there are some suggestions:
DB: For better stability multi-AZ RDS setup for the DB is recommended. Benefit is RDS is fully managed service so you don't need to worry about replication, maintenance etc.
Web/app servers: you can deploy a copy in any region you want and connect to the same DB.
S3: you can enable crosss-region replication as well as encryption, but make sure it is used wisely (e.g. files are served to the client from bucket in closest region)
You can set up your own SSL on the server and it does not require ELB. However, you can use ELB with one webnode only.
I do NOT suggest to use Beanstalk because despite it really makes the first steps more easier you may have trouble trying to configure something non-standard in the future (unless you're very well familiar with EBT, of course).
To add efficiency you may want to add CDN (either AWS ot another vendor).
Make sure your environment configuration is really secure. You may need for your team someone who is familiar with AWS because every topic can be converted to a separate article.
I'm investigating a move to Kubernetes (coming from AWS ECS). But I haven't solved the local development issue when depending on internal services.
Let me elaborate:
When developing and testing microservices, before they are deployed as a Kubernetes Service I want to be able to talk to other, internal Kubernetes Services. As there are > 20 microservices I have a Kubernetes cluster running latest development versions. I can't run a MiniKube.
example:
I'm developing an user-service which needs access to the email service. The Email service is already on Kubernetes and is an internal service.
So before the user-service is deployed I want to be able to talk to the internal email service for dev/testing. I can't make use of K8S nice service discovery env vars.
As we currently already have a VPN up to restrict DEV env to testers/development only, could I use this VPN to provide access to the Kubernetes-Service IP-addresses? I do have Kubernetes DEV-env on the same VPC as the VPN is in.
If you deploy your internal services as type NodePort, then you can access them over your VPN via that nodePort. NodePorts can be dynamically allocated or you can customize them to be 'static' where they are known by you up front.
When developing an app on your local machine, you can access the dependent service by that NodePort.
As an alternative, you can use port-forwarding from kubectl (https://kubernetes.io/docs/user-guide/connecting-to-applications-port-forward/) to forward a pod to your local machine. (Note: This only handles traffic to a pod not a service).
Telepresence (http://telepresence.io) is designed for this scenario, though it presumes developers have kubectl access to the staging/dev cluster.
We are planning to have a terminal server setup to access all the AWS instances including EMR etc. So we have planned to store key pair of all the instances on terminal server. Is this a best practice ? If not then how to access the AWS infra
That should work. It sounds like the Bastion Server concept. It gives you a single exposed server to harden against attacks, which generally simplifies your infrastructure security.