I'm running a gunicorn+flask service in a docker container with Google Container Engine. I set up the cluster following the tutorial at http://kubernetes.io/docs/hellonode/
The REMOTE_ADDR environmental variable always contains an internal address in the Kubernetes cluster. What I was looking for is HTTP_X_FORWARDED_FOR but it's missing from the request headers. Is it possible to configure the service to retain the external client ip in the requests?
If anyone gets stuck on this there is a better approach.
You can use the following annotations depending on your kubernetes version:
service.spec.externalTrafficPolicy: Local
on 1.7
or
service.beta.kubernetes.io/external-traffic: OnlyLocal
on 1.5-1.6
before this is not supported
source: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
note that there are caveats:
https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips
I assume you set up your service by setting the service's type to LoadBalancer? It's an unfortunate limitation of the way incoming network-load-balanced packets are routed through Kubernetes right now that the client IP gets lost.
Instead of using the service's LoadBalancer type, you could set up an Ingress object to integrate your service with a Google Cloud HTTP(s) Load Balancer, which will add the X-Forwarded-For header to incoming requests.
Related
I'm looking to host multiple services on a single compute instance. I'm using docker for the one existing service, which has been configured to serve the http on the usual ports. And since I'm using docker I figured it would be easier to set a routing setting than set up a new apache/nginx server.
Could I route the traffic from one address to a specific port? Or, more specifically, is it possible to map a specific port on the server to the http/s ports for a certain domain name?
If it is possible I'm sure it must be a simple setting, but I'm not intimately familiar with GCloud so I'm also sure that I'm missing something.
Yes, you can route ports using IP Tables or setting up a container for virtual hosts which will use Apache or Nginx or similar). However, there are very good reasons to not expose Docker containers to the Internet. Deploy Apache or Nginx as your frontend or deploy a Google Cloud HTTP(S) Load Balancer.
This is not how virtual host works - only cannot simply remap :443 without breaking SSL.
Use Cloud DNS to provide name resolution & use virtual host to tell apart the host headers.
External HTTP(S) Load Balancing would be required to map different internal portswhich also requires named virtual hosts - else the backend will not know which site it is.
With named virtual hosts one can also define multiple SSL certificates.
I have an AWS EC2 instance set up running my back-end, and it's able to communicate with my front-end (locally), but not with front-end deployed (on Netlify).
Is it necessary to create a domain name for my EC2 instance so I can use SSL? There's no point to have a domain name to my back end since it's just there for the API calls.
How do I use SSL for my backend server without a domain name? Every video and blog I've found requires a domain name. If anyone can point me to the right resource, would appreciate it.
You can enable SSL on an EC2 instance without a domain using a combination of Caddy and nip.io.
nip.io is allows you to map any IP Address to a hostname without the need to edit a hosts file or create rules in DNS management.
Caddy is a powerful open source web server with automatic HTTPS.
Install Caddy on your server
Create a Caddyfile and add your config (this config will forward all requests to port 8000)
<EC2 Public IP>.nip.io {
reverse_proxy localhost:8000
}
Start Caddy using the command caddy start
You should now be able to access your server over https://<IP>.nip.io
I wrote an in-depth article on the setup here: Configure HTTPS on AWS EC2 without a Custom Domain
Sadly yes to use SSL-certificates you need to have a valid DNS name so it can process it when you are calling it, anyways if what you want to encrypt is the info you could just use your own encryption method and send the data encrypted to frontend, then use something like crypto.js to use it once decrypted, but the best practice would be giving the backend it's own DNS, that way if at some point the API grows to the point it can be used by others for business you can have them point at something named (and also you don't need to deal with the whole manual encryption/decryption).
I'm running Istio Ingress Gateway in a GKE cluster. The Service runs with a NodePort. I'd like to connect it to a Google backend service. However we need to have an health check that must run against Istio. Do you know if Istio expose any HTTP endpoint to run health check and verify its status?
Per this installation guide, "Istio requires no changes to the application itself. Note that the application must use HTTP/1.1 or HTTP/2.0 protocol for all its HTTP traffic because the Envoy proxy doesn't support HTTP/1.0: it relies on headers that aren't present in HTTP/1.0 for routing."
The healthcheck doesn't necessarily run against Istio itself, but against the whole stack behind the IP addresses you configured for the load balancer backend service. It simply requires a 200 response on / when invoked with no host name.
You can configure this by installing a small service like httpbin as the default path for your gateway.
You might also consider changing your Service to a LoadBalancer type, annotated to be internal to your network (no public IP). This will generate a Backend Service, complete with healthcheck, which you can borrow for your other load balancer. This method has worked for me with nesting load balancers (to migrate load) but not for a proxy like Google's IAP.
I have enabled Private Google API access for a VPC and I use this HTTP proxy solution described to connect my offsite datacenter to the Google Cloud backend.
Using the solution, I have verified that the Google object storage api's work, by using gsutil to move files across the offsite network.
However I am unable to connect to mqtt.googleapis.com that is required for cloud IOT.
I think this is because the MQTT broker running at mqtt.googleapis.com cannot be accessed via a private network unless it is also proxied like the HTTP proxy solution described above.
Meanwhile actual gsutil IOT commands work fine because I presume they are running over the Google HTTP API.
To solve this I see we'd need any one of the below, unless someone has different way to do this?
Run an MQTT broker proxy in the private VPC and route MQTT packets to the mqtt.googleapis.com . Is there a suitable MQTT proxy broker that we can use in this case?
If we get a range of public IP's that the mqtt bridge (mqtt.googleapis.com) is running at then we can simply build the network routes for this one use case. Is this available?
Would it work via the HTTP protocol bridge in IoT Core? Is using HTTP instead of MQTT an option?
I managed to get this to work using NGINX as a reverse proxy and stream the TCP traffic directly to mqtt.googleapis.com. Here are the steps to achieve this
Install Nginx with the --with-stream configuration flag . This builds Nginx with the functionality of a TCP streaming proxy
My Nginx conf file contains the following to point to Google's broker. The Nginx server is running in an instance in the VPC
/etc/nginx/nginx.conf
stream {
upstream google_mqtt {
server mqtt.googleapis.com:8883;
}
server {
listen 8883;
proxy_pass google_mqtt;
}
}
The internal private VPC has a DNS Server that resolves mqtt.googleapis.com to the IP of the Nginx server
I'm running a 4 node (2 cluster nodes, 2 arbiter nodes) neo4j 2.2.5 enterprise cluster on 2 virtual machines in a private subnet within AWS EC2 (Linux). I'm able to start the clusters and have members join them.
I have also configured an AWD Elastic Load Balancer to access the 7474 port of the different machines. The load balancer health check is configured with the end point below.
/db/manage/server/ha/available
However, the servers return a 401 unauthorized as they don't contain an authorization header. I'm not sure if AWS allows you to pass auth values for healthcheck. Is it possible to have neo4j NOT require auth for "just" these health check URLs? (i don't want to disable auth for the entire db).
Thx,
NN
AWS ELB doesn't support custom header, I suggest to you use HAproxy instead of ELB - Here is the great description of the configuration - http://blog.armbruster-it.de/2015/08/neo4j-and-haproxy-some-best-practices-and-tricks/
The following is available in 2.3.3:
http://neo4j.com/docs/stable/ha-configuration.html#config_dbms.security.ha_status_auth_enabled
dbms.security.ha_status_auth_enabled
Description: Require authorization for access to the HA status endpoints.
Valid values: dbms.security.ha_status_auth_enabled is a boolean.
Default value: true
Changing that to false will allow the HA status check endpoint to be accessible without security. I'm currently using this as a means to use AWS ELB with Neo4j and it's working just fine.
you need to set these two variables to the false in neo4j.conf default value of these two variables is true.
These variables help ELB for health check without authentication
dbms.security.ha_status_auth_enabled=false
dbms.security.causal_clustering_status_auth_enabled=false
for setting up elastic load balancer
Following listeners need to be configured for the load balancer
Set up two elastic load balancer. One for reading and is used in neo4j desktop and other is used for writing the data to neo4j
For status endpoint please check the URL according to Neo4j version -
https://neo4j.com/docs/operations-manual/current/monitoring/causal-cluster/http-endpoints/
ELB for reading data.
Health check - http:7474/db/manage/server/causalclustering/available
ELB for writing data.
Health check - http:7474/db/manage/server/causalclustering/writable
Answering to your question - there is no possibility to suppress authentication for just one path in Neo4j. As we see there - whitelist is hardcoded and can't be extended.
Maybe you can use nginx as reverse proxy in your case?
Solution:
place nginx in front of Neo4j instance.
configure standard proxy setup
for /db/manage/server/ha/available automatically add authorization headers
There is couple of articles to help you with setup:
NGINX REVERSE PROXY by Nginx
Understanding Nginx HTTP Proxying, Load Balancing, Buffering, and Caching
Example (not tested):
location /db/manage/server/ha/available {
proxy_set_header HOST "Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==";
proxy_pass http://localhost:7474/db/manage/server/ha/available;
}
Actually FylmTM's solution is the right approach! This just needs some minor tweaks - pl see below..
location /db/manage/server/ha/available {
proxy_set_header Authorization "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==";
proxy_pass_header Authorization;
proxy_pass http://ip-10-0-1-98:7474/db/manage/server/ha/available;
}
This addresses the above issue and you can use your AWS ELB for healthchecks!
Now I still need to figure out why I'm getting a 404 error on these urls (pl see this post below)
neo4j HA cluster end points return 404