Hosting django on aks behind nginx-ingress - django

I am trying to host a django website on Azure kubernetes service behide nginx-ingress, and I would like my django web show under a path.
e.g. when access the default admin site, I would like to access it at http://example.com/django/admin instead of http://example.com/admin
I tried the configure below, when I access http://example.com/django/admin it will forward me to http://example.com/admin and show me 404 error from default ingress backend, as I set django debug to ture I assume this mean ingress did not send my request to my django service
# path example
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /django(/|$)(.*)
so I try to curl -I -k http://example.com/django/admin, and it show something like below
HTTP/1.1 301 Moved Permanently
Server: openresty/1.15.8.2
Date: Wed, 06 Nov 2019 04:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /admin/
The same thing happen to any valid page in the site, if I curl -I -k http://example.com/django/any_valid_page it show below:
HTTP/1.1 301 Moved Permanently
Server: openresty/1.15.8.2
Date: Wed, 06 Nov 2019 04:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /any_valid_page/
I wonder it is caused by I am doing the test with the default django development web server? (i.e. python manage.py runserver)
If I try to host it at root like below, everything is fine...
# root example
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /

Trying adding this
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /django
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /django

Starting in Version 0.22.0, ingress definitions using the annotation
nginx.ingress.kubernetes.io/rewrite-target are not backwards
compatible with previous versions. In Version 0.22.0 and beyond, any
substrings within the request URI that need to be passed to the
rewritten path must explicitly be defined in a capture group. So make
sure you have right version.
When using SSL offloading outside of cluster it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: "true" annotation in the particular resource.
I think your Ingress configuration file should look like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: example.com
http:
paths:
- path: /django(/|$)(.*)
backend:
serviceName: django-service
servicePort: 80
If you get 404 error, there is possible solution:
Please change https to http in the curl command?:
curl --resolve your-host:80:xx.xxx.xx.xxx http://my-host:80
To get the IP from kubectl get ing command, it is necessary to
enable the reporting Ingress status feature. Take a look on: reporting-ingress-status.
There is the default server in the Ingress controller. It returns
the Not Found page with the 404 status code for all requests for
domains for which there are no Ingress rules defined. Those requests
are not shown in the access log.
Since you're getting a 404, this means that the host header of your
requests doesn't match with the host field in the Ingress resource.
To set the host header in curl, please see previous curl
commands. Optionally, you can also do:
curl http://<ip> -H "host: example.com"
Please take a look on ngnix-ingress, server-side-https-enforcement-nginx.

This is a problem from Django's side. Whenever the admin is not logged in, the /django/admin results in a redirect to /admin/. In this case, if you just replace /django/admin/ with /django/admin/ in the browser URL field it will work and open django admin login.
So basically Django's built-in redirect conflicts with the Ingress's rewrite module.

Related

502 Server Error when creating http load balancer in google/GKE

My application consists of play web application deployed using GKE. The application was running fine (using Deployment and Loadbalancer service) and then I decided to use Ingress. I made the following changes which has made the application unreachable. I get 502 error when I try to connect with the application using ingress IP.
The application is of kind Deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
spec:
replicas: 2
selector:
matchLabels:
app: webapp
It has a service associated with it
apiVersion: v1
kind: Service
metadata:
name: webapp-service
spec:
selector:
app: webapp
ports:
- protocol: TCP
port: 9000 #this service is reachable at this port
targetPort: 9000 #this service will forward the request to correspoding nodes of the service at this port
#type: LoadBalancer
type: NodePort
Then I applied the following file to create ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: webapp-https-loadbalancer-ingress
annotations:
kubernetes.io/ingress.class: "gce"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: webapp-service
servicePort: 9000
I can see that there is an IP address (which is also reachable from outside) when I run
kubectl describe ingress webapp-https-loadbalancer-ingress
Name: webapp-https-loadbalancer-ingress
Namespace: default
Address: 3x.yyy.zzz.pq
Default backend: default-http-backend:80 (10.88.0.5:8080)
Rules:
Host Path Backends
---- ---- --------
*
... Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 10m loadbalancer-controller default/webapp-https-loadbalancer-ingress
Normal CREATE 9m10s loadbalancer-controller ip: 3x.yyy.zzz.pq
But I am not able to reach the application using https://3x.yyy.zzz.pq. I haven't yet associated the domain with the IP. I tried to connect using curl and got error- 502 bad gateway error
curl -v 3x.xxx.xxx.xxx
* Expire in 0 ms for 6 (transfer 0x55d4c5258f90)
* Trying 3x.xxx.xxx.xxx...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55d4c5258f90)
* Connected to 3x.xxx.xxx.xxx (3x.xxx.xxx.xxx) port 80 (#0)
> GET / HTTP/1.1
> Host: 3x.xxx.xxx.xxx
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/html; charset=UTF-8
< Referrer-Policy: no-referrer
< Content-Length: 332
< Date: Tue, 22 Dec 2020 22:27:23 GMT
<
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
* Connection #0 to host 3x.xxx.xxx.xxx left intact
The issue was that the load balancer's IP was not in the allowed-hosts list in the server. As temporary fix, I used wild card in server configuration to allow traffic from all hosts. I am still figuring out how to restrict it to load balancer's internal IPs

Nginx Ingress rules not find a match

I'm struggling on an Ingress configuration in yaml because the pattern matching seems not to work.
I would like the frontend-lb ClusterIP Service for the frontend deployment to respond to any of these:
https://example.com
https://example.com/home
https://example.com/login
... any other without /api/
And the backend-lb ClusterIP Service for the backend deployment to respond to any of these:
https://example.com/api/...
The yaml for the ingress rules is the following:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- secretName: example-com-tls
hosts:
- example.com
rules:
- host: example.com
http:
paths:
- path: /api
backend:
serviceName: gateway-lb
servicePort: 80
- path: /
backend:
serviceName: frontend-lb
servicePort: 80
The result is that every backend url is recognized as a frontend url and gives back 404 Not Found.
I've tried many other regexp and also I've tried to exclude /api in the frontend path (!?api).* but with no success.
UPDATE:
What I see with the logs it's like in the backend the url path gets blanked because if I call:
https://example.com/api/javalin-api-gateway/login
I get this error:
Not found. Request is below context-path (context-path:
'/javalin-api-gateway')
While when I call the frontend with a specific url path:
https://example.com/home
The /home controller is effectively called (it doesn't get blanked).
If I call the backend service directly (if the service is a LoadBalancer) with the same url:
http://192.168.64.17:31186/javalin-api-gateway/login
I get the right response, signal that the backend part is working properly.
How is possible that only the backend service doesn't receive the complete path?

Ingress Controller on Minikube not routing correctly flask POST request

I have a simple Flask app. It worked fine when I connected to it via port-forwarding to send the HTTP Post request directly to the Service.
from flask import Flask, request
import redis
from rq import Queue
from worker import job_worker
UPLOAD_FOLDER = './uploads/'
app = Flask(__name__)
r = redis.Redis()
q = Queue(connection = r)
#app.route('/', methods=['POST'])
def upload():
scale = int(request.form['scale'])
q.enqueue(job_worker, scale)
return ""
if __name__ == "__main__":
app.run()
I also have a simple index.html file in an nginx container which is served at port 80. It does an ajax POST request to "/upload". Which if you look at the ingress controller, should convert that to a port 5000 request and strip away the "upload"
The flask app gets served at port 5000
Here is the ingress controller:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: emoji-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /upload
backend:
serviceName: emoji-backend
servicePort: 5000
- path: /
backend:
serviceName: emoji-frontend
servicePort: 80
And for completeness, the emoji-backend service:
apiVersion: v1
kind: Service
metadata:
name: emoji-backend
labels:
app: emoji-backend
tier: backend
spec:
type: LoadBalancer
ports:
- port: 5000
selector:
app: emoji-backend
tier: backend
I get a 502 bad gateway without really any indication except the ingress log does say this:
2019/09/29 21:41:04 [error] 2021#2021: *78651 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.64.1, server: _,
request: "POST /upload HTTP/2.0", upstream: "http://172.17.0.4:5000/", host: "192.168.64.5", referrer: "https://192.168.64.5/"
"http://172.17.0.4:5000/" is the correct endpoint and port for the emoji-backend service.
Adding the following line fixed it:
app.run(debug=True,host='0.0.0.0',port=5000)
However, it took me a while to figure that out because at first when I tried it my docker image was not updating when I re-deployed.

Istio HTTPMatchRequest seems to match request using OR logic instead of the documented AND logic

As per https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/#HTTPMatchRequest ,
HttpMatchRequest specifies a set of criterion to be met in order for the rule to be applied to the HTTP request. For example, the following restricts the rule to match only requests where the URL path starts with /ratings/v2/ and the request contains a custom end-user header with value jason.
i take that to mean the matching should be of type AND .
Below is an istio virtual service definition. As per the above definition, I'd assume that this virtual service only permits requests of the form POST /status/...
However, It seems that the logic is actually OR i.e. either POST requests or (for instance, GET /status/xxx) requests go through . Can someone explain or correct my config.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: "httpbin-virtual-service"
spec:
hosts:
- "*"
gateways:
- my-istio-gateway
http:
- match:
- method:
exact: POST
- uri:
prefix: /status
route:
- destination:
host: "httpbin"
port:
number: 80 # application port
Output
$ siege -c1 -d1 --content-type "application/json" '127.0.0.1:31380/delay/2 POST {"ids": ["1","2","3"]}' ==> not a request to /status/.. , why does this match
HTTP/1.1 200 2.00 secs: 1072 bytes ==> POST http://127.0.0.1:31380/delay/2
HTTP/1.1 200 2.01 secs: 1072 bytes ==> POST http://127.0.0.1:31380/delay/2
..
$ siege -c1 -d1 127.0.0.1:31380/status/200 ====================> not a POST request , why does this match
HTTP/1.1 200 0.00 secs: 0 bytes ==> GET /status/200
HTTP/1.1 200 0.00 secs: 0 bytes ==> GET /status/200
..
Solved, I had a "-" before uri
The correct config should be
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: "httpbin-virtual-service"
spec:
hosts:
- "*"
gateways:
- my-istio-gateway
http:
- match:
- method:
exact: POST
uri:
prefix: /status
route:
- destination:
host: "httpbin"
port:
number: 80 # application port

Impossible to use Google Cloud CDN cache with ingress controller, Transfer-Encoding header

I try to configure Google Cloud CDN to my container engine project.
Following the documentation It has either a Content-Length header or a Transfer-Encoding header in order to be cached.
My backend use gzip compression so I have Transfer-Encoding: chunked
The problem is it seems the ingress load balancer remove the Transfer-encoding header so I can't have a "cache hit"
I used "kubectl port-forward" to connect direclty to an instance backend and I have the Transfer-encoding header.
But when I connect to the external IP, the header has disappear.
Here my ingress configuration
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-preprod3-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: gateway-preprod2-static-ip
spec:
tls:
- secretName: gateway-preprod-secret-2018-with-ca-7
backend:
serviceName: gateway-preprod
servicePort: 80
Here my deployment configuration
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gateway-preprod
spec:
replicas: 1
strategy:
type: RollingUpdate
minReadySeconds: 50
template:
metadata:
labels:
app: gateway-preprod
spec:
containers:
- name: gateway-preprod
image: eu.gcr.io/writecontrol-1055/gateway:v305
env:
- name: writecontrolEnv
value: preprod
ports:
- containerPort: 80
In the opposite, for certain ressources not GZIP compressed, the Content-length header is given and I have a successful "cache hit"
The kubernetes version is 1.7.12-gke.1
Here an URL to test it : https://preprod-writecontrol.ovh
I am not sure why your response has Transfer-Encoding header. This header is not meant to be at the source (your application) and it is usually added by other hops, such as proxies like Cloud HTTP Load Balancer (=Ingress).
More info on Transfer-Encoding here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
I was able to get CDN working with a GZIP response myself using the Content-Encoding header instead:
First read the caching requirements at https://cloud.google.com/cdn/docs/caching. You need a Content-Length and/or Cache-Control: public header. So code these headers into your application server.
After enabling CDN on the load balancer created by Ingress, make two requests, and if you see Age header on the second request (as I did below), it means your request is now being served from CDN.
curl -vH Accept-Encoding:gzip 35.186.195.233
> [...]
>
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Date: Tue, 27 Mar 2018 19:38:20 GMT
< Content-Length: 87
< Content-Type: application/x-gzip
< Via: 1.1 google
< Cache-Control: max-age=600,public
��H����Q(�/�IQ�
* Connection #0 to host 35.186.195.233 left intact
K-*��ϳR0�3�3���/.�K�M�R�)+OM�55575��L�4ѭ�N+L���K��4A
And the second request (note the Age header):
$ curl -vH Accept-Encoding:gzip 35.186.195.233
[...]
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=600,public
< Content-Encoding: gzip
< Date: Tue, 27 Mar 2018 19:42:01 GMT
< Content-Length: 87
< Content-Type: application/x-gzip
< Via: 1.1 google
< Age: 314
<
��H����Q(�/�IQ�
* Connection #0 to host 35.186.195.233 left intact
K-*��ϳR0�3�3���/.�K�M�R�)+OM�55575��L�4ѭ�N+L���K��4A