Summary
I have a flask application deployed to Kubernetes with python 2.7.12, Flask 0.12.2 and using requests library. I'm getting a SSLError while using requests.session to send a POST Request inside the container. When using requests sessions to connect to a https url , requests throws a SSLError
Some background
I have not added any certificates
The project works when I run a docker image locally but after deployment to kubernetes, from inside the container - the post request is not being sent to the url
verify=false does not work either
System Info - What I am using:
Python 2.7.12, Flask==0.12.2, Kubernetes, python-requests-2.18.4
Expected Result
Get HTTP Response code 200 after sending a POST request
Error Logs
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 511, in send
raise SSLError(e, request=request)
SSLError: HTTPSConnectionPool(host='dev.domain.nl', port=443): Max retries exceeded with url: /ingestion?LrnDevEui=0059AC0000152A03&LrnFPort=1&LrnInfos=TWA_100006356.873.AS-1-135680630&AS_ID=testserver&Time=2018-06-22T11%3A41%3A08.163%2B02%3A00&Token=1765b08354dfdec (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:661)'),))
/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Reproduction Steps
import requests
from flask import Flask, request, jsonify
from requests import Request, Session
sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(max_retries = 200)
sess.mount('http://', adapter)
sess.mount('https://', adapter)
sess.cert ='/usr/local/lib/python2.7/site-packages/certifi/cacert.pem'
def test_post():
url = 'https://dev.domain.nl/ingestion/?'
header = {'Content-Type': 'application/json', 'Accept': 'application/json'}
response = sess.post(url, headers= header, params= somepara, data= json.dumps(data),verify=True)
print response.status_code
return response.status_code
def main():
threading.Timer(10.0, main).start()
test_post()
if __name__ == '__main__':
main()
app.run(host="0.0.0.0", debug=True, port=5001, threaded=True)
Docker File
FROM python:2.7-alpine
COPY ./web /web
WORKDIR /web
RUN pip install -r requirements.txt
ENV FLASK_APP app.py
EXPOSE 5001
EXPOSE 443
CMD ["python", "app.py"]
The problem may be in the Alpine Docker image that lacks CA certificates. On your laptop code works as it uses CA certs from you local workstation. I would think that running Docker image locally will fail too - so the problem is not k8s.
Try to add the following line to the Dockerfile:
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
It will install CA certs inside the container.
Related
I've built a docker image using the command "docker build -t flask-app-testing .". Later I've run the image with the command docker run --name test-flask -p 5060:5060 flask-app-testing using a port of 5060 later it generates a url like : image
if i click here it gives the content from the flask app on the browser page but it's not giving the desired. and showing ....
This site can’t be reached The webpage at http://localhost:5060/ might be temporarily down or it may have moved permanently to a new web address.
ERR_UNSAFE_PORT
The app.py file is
import uuid
from flask import Flask
instanceId = uuid.uuid4().hex
app = Flask(__name__)
#app.route("/")
def get_instance_id():
return f"instance {instanceId}"
if __name__ == "__main__":
app.run(port=5060, host="0.0.0.0")
The docker file is:
FROM python:3.9.12
WORKDIR /app
COPY . .
RUN pip3 install -r requirements.txt
CMD ["gunicorn", "--bind=0.0.0.0:5060","app:app"]
requirements.txt:
flask
gunicorn
I deployed my Django project on Heroku with gunicorn. It's basically only the api (no templates).
When I deploy heroku and access <heroku url>/api/login for example in the browser
and post login data already in json format, it always returns
"detail": "Unsupported media type \"application/x-www-form-urlencoded\" in request."
But when I do the same on localhost, the user gets authenticated and I receive a response with user data...
post data example for login:
{ "email": "ana#test.com", "password": "ana1234567890" }
The parser_classes = [JSONParser] is added on every view where I don't have images or files (there I am using FileUploadParser).
I deployed on Heroku with gunicorn, because on localhost I permanently received "Unauthorized" from backend ( Permission Class is "isAuthenticated" for most of my views). After some research I figured out that probably the authorization header is not sent (here the APACHE WSGIPassAuthorization On would be a solution) but I don't have an apache server running and I don't want no webserver running on my machine.
I thought if I would deploy it on heroku with gunicorn, I could continue with the development without the "Unautorized" header but instead I run in other errors, like "detail": "Unsupported media type \"application/x-www-form-urlencoded\" in request." .. or the database connection is refused.
In my frontend I am sending the following header
// Headers
const token_config = {
headers: {
'content-type': 'application/json',
accept: 'application/json',
'Authorization': 'Bearer ${token}',
//withCredentials : true,
},
};
And the frontend is of course also receiving the "Wrong mediatype Error" .
I would be so greatfull if anybody could provide an detailed "howTo" for Django DRF + postgres on Heroku... I've wen't through lots of howtos, but none of them provides a fitting solution for this..
My main questions now are...
What does settings.py need so that the backend works on heroku just as fine as on localhost?
Do I really need to start the heroku postgres database manually with
heroku pg:psql postgresql-shaped-60432 --app every time after deployment? - Because If I don't, the herocu log shows:
2020-09-09T11:11:15.028985+00:00 app[web.1]: connection = Database.connect(**conn_params)
2020-09-09T11:11:15.028985+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
2020-09-09T11:11:15.028986+00:00 app[web.1]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2020-09-09T11:11:15.028986+00:00 app[web.1]: django.db.utils.OperationalError: could not connect to server: Connection refused
2020-09-09T11:11:15.028987+00:00 app[web.1]: Is the server running on host "localhost" (127.0.0.1) and accepting
2020-09-09T11:11:15.028987+00:00 app[web.1]: TCP/IP connections on port 5432?
With this error + accessing the /api/login for example, I get an HTTP status of 503 or 500...
My requirements.txt:
Django==3.1.1
django-allauth==0.42.0
django-cors-headers==3.5.0
django-environ==0.4.5
django-rest-auth==0.9.5
django-sslserver==0.20
djangorestframework==3.11.1
djangorestframework-simplejwt==4.1.3
gunicorn==20.0.4
image==1.5.27
importlib-metadata==1.7.0
oauthlib==3.1.0
Pillow==7.2.0
psycopg2-binary==2.8.2
PyJWT==1.7.1
Thanks in advance and best regards!
Ok I fixed it!
I basically had to add all the settings.py stuff:
import django_heroku
import dj_database_url
ALLOWED_HOSTS = ['*']
DATABASES = {'default': dj_database_url.config(default='URI from Heroku-color- postgres database')}
# ALL THE STATIC PATHS
MEDIA_ROOT = os.path.join(BASE_DIR,'media')
MEDIA_URL = '/media/'
#location where django collect all static files
STATIC_ROOT = os.path.join(BASE_DIR,'static')# location where you will store your static files
STATICFILES_DIRS = [os.path.join(BASE_DIR,'walkAdog/static')]
STATIC_URL = '/static/'
# for deployement on heroku
django_heroku.settings(locals())
AT THIS POINT the origin error “detail”: “Unsupported media type \”application/x-www-form-urlencoded\“ in request.” did not appear anymore
.. But the next error occured: No module named 'django_heroku'...
So I figured out, that django-heroku was not installed correctly, since the pipenv install psycopg crashed - worked for me as I did installed pipenv install psycopg2==2.7.7 and then django whitenoise dj-database-url psycopg2
instead, the database was "empty" - quite predictable..
So I did makemigrations on localhost
then commit the project/migrations folder (and the rest)
and ran heroku run python manage.py migrate
-> everything working fine now!!
Basically, I just had to follow the steps from https://medium.com/#hdsingh13/deploying-django-app-on-heroku-with-postgres-as-backend-b2f3194e8a43 .. I just kinda didn't understand the neccessarity of all the stuff..
My use-case requires my app to return CORS headers when error response is 401.
This functionality was added by AWS last year (See this). It can be done using Cloudformation and Swagger template but I'm not sure if it's yet possible with Chalice.
I solved my problem by using a python script that adds the CORS headers for 401 response and redeploys the API. This redeploying of API takes a second or two since it doesn't have to deploy all Lambdas like Chalice.
deploy.sh
#!/usr/bin/env bash
cd services
A="$(chalice deploy --stage $1)"
cd ..
python update_api_response_headers.py "$A" "$1"
update_api_response_headers.py
import boto3
import sys
import re
if len(sys.argv) != 3:
print("usage: python script.py <CHALICE_DEPLOYMENT_RESULT> <STAGE>")
exit()
search = re.search('URL: https:\\/\\/([a-zA-Z0-9]+).+', sys.argv[1])
api_id = search.group(1)
print(api_id)
if not api_id:
print(sys.argv[1])
exit()
client = boto3.client('apigateway')
response = client.put_gateway_response(
restApiId=api_id,
responseType='UNAUTHORIZED',
statusCode='401',
responseParameters={
"gatewayresponse.header.Access-Control-Allow-Origin": "'*'",
"gatewayresponse.header.Access-Control-Allow-Headers": "'*'"
}
)
response = client.create_deployment(
restApiId=api_id,
stageName=sys.argv[2])
print(sys.argv[1])
Services folder contains my chalice app. deploy.sh and update_api_response_headers.py are placed one level above the chalice app. To deploy the app I simply have to use:
./deploy.sh stage_name
I want to build a python client to talk to my python Google Cloud Endpoints API. My simple HelloWorld example is suffering from an HTTPException in the python client and I can't figure out why.
I've setup simple examples as suggested in this extremely helpful thread. The GAE Endpoints API is running on localhost:8080 with no problems - I can successfully access it in the API Explorer. Before I added the offending service = build() line, my simple client ran fine on localhost:8080.
When trying to get the client to talk to the endpoints API, I get the following error:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 526, in getresponse
raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://localhost:8080/_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=%3A%3A1
I've tried extending the http deadline. Not only did that not help, but such a simple first call on localhost should not be exceeding a default 5s deadline. I've also tried accessing the discovery URL directly within a browser and that works fine, too.
Here is my simple code. First the client, main.py:
import webapp2
import os
import httplib2
from apiclient.discovery import build
http = httplib2.Http()
# HTTPException happens on the following line:
# Note that I am using http, not https
service = build("helloworldendpoints", "v1", http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest"))
# result = service.resource().method([parameters]).execute()
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-type'] = 'text/plain'
self.response.out.write("Hey, this is working!")
app = webapp2.WSGIApplication(
[('/', MainPage)],
debug=True)
Here's the Hello World endpoint, helloworld.py:
"""Hello World API implemented using Google Cloud Endpoints.
Contains declarations of endpoint, endpoint methods,
as well as the ProtoRPC message class and container required
for endpoint method definition.
"""
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
# If the request contains path or querystring arguments,
# you cannot use a simple Message class.
# Instead, you must use a ResourceContainer class
REQUEST_CONTAINER = endpoints.ResourceContainer(
message_types.VoidMessage,
name=messages.StringField(1),
)
package = 'Hello'
class Hello(messages.Message):
"""String that stores a message."""
greeting = messages.StringField(1)
#endpoints.api(name='helloworldendpoints', version='v1')
class HelloWorldApi(remote.Service):
"""Helloworld API v1."""
#endpoints.method(message_types.VoidMessage, Hello,
path = "sayHello", http_method='GET', name = "sayHello")
def say_hello(self, request):
return Hello(greeting="Hello World")
#endpoints.method(REQUEST_CONTAINER, Hello,
path = "sayHelloByName", http_method='GET', name = "sayHelloByName")
def say_hello_by_name(self, request):
greet = "Hello {}".format(request.name)
return Hello(greeting=greet)
api = endpoints.api_server([HelloWorldApi])
Finally, here is my app.yaml file:
application: <<my web client id removed for stack overflow>>
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /_ah/spi/.*
script: helloworld.api
secure: always
# catchall - must come last!
- url: /.*
script: main.app
secure: always
libraries:
- name: endpoints
version: latest
- name: webapp2
version: latest
Why am I getting an HTTP Deadline Exceeded and how to I fix it?
On your main.py you forgot to add some variables to your discovery service url string, or you just copied the code here without it. By the looks of it you were probably suppose to use the format string method.
"http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest".format(api='helloworldendpoints', apiVersion="v1")
By looking at the logs you'll probably see something like this:
INFO 2015-11-19 18:44:51,562 module.py:794] default: "GET /HTTP/1.1" 500 -
INFO 2015-11-19 18:44:51,595 module.py:794] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 3109
INFO 2015-11-19 18:44:52,110 module.py:794] default: "GET /_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=127.0.0.1 HTTP/1.1" 200 3719
It's timing out first and then "working".
Move the service discovery request inside the request handler:
class MainPage(webapp2.RequestHandler):
def get(self):
service = build("helloworldendpoints", "v1",
http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest")
.format(api='helloworldendpoints', apiVersion='v1'))
Hi i need help to get info from web service on secure site with pfx certificate with password.
i tried more then one example..
code example:
import requests
wsdl_url = 'blabla'
requests.get(wsdl_url, cert='cert.pfx', verify=True)
other example:
import urllib3
import certifi
wsdl_url = 'blabla'
http = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED', # Force certificate check.
ca_certs=certifi.where() # Path to the Certifi bundle.
)
certifi.where()
# You're ready to make verified HTTPS requests.
try:
r = http.request('GET', wsdl_url)
print r
except urllib3.exceptions.SSLError as e:
print "wrong"
# Handle incorrect certificate error.
Error type:
connection aborted
an existing connection was forcibly closed by the remote host
help please