AWS HealthCheck endpoint doesn't send any headers which causes Cowboy (v 1.1.2) to return 400. This is causing container restarts.
Is there any way around the issue?
Related github issue: https://github.com/phoenixframework/phoenix/issues/2437
curl request to reproduce the error:
curl http://localhost:4000/ping -H 'Host:'
Log:
[error] Cowboy returned 400 and there are no headers in the connection.
This may happen if Cowboy is unable to parse the request headers,
for example, because there are too many headers or the header name
or value are too large (such as a large cookie).
You can customize those values when configuring your http/https
server. The configuration option and default values are shown below:
protocol_options: [
max_header_name_length: 64,
max_header_value_length: 4096,
max_headers: 100
]
endpoint configuration:
config :my_app, MyAppWeb.Endpoint,
load_from_system_env: true,
url: [host: System.get_env("MY_HOST"), port: 443],
force_ssl: [rewrite_on: [:x_forwarded_proto]]
I ended up running the server with the following Endpoint config:
config :my_app, MyAppWeb.Endpoint,
load_from_system_env: true,
http: [port: 4000]
and the problem was resolved.
It probably had something to do with the fact that cowboy was running https server behind ELB.
Related
I'm working on a React and Django API project, I'm using postgreeSQL as Database, and I deployed my website using nginx and gunicorn,
I have a problem on my deployed website when I try to insert a lot of data (add studies),
I'm getting this error:
Access to XMLHttpRequest at 'http://192.168.85.126:8000/api/new-study/'
from origin 'http://192.168.85.126' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
PS: I'm not getting this error when I try to add less data, in my development environment I can add whatever data I want, the problem is only happening in production
Let me guess, you're not using gunicorn in your development environment, but only use manage.py runserver, aren't you? That's why it was running, as "development server" is more easy and loose on security than gunicorn.
This is a security issue, as your system is blocking request from an unknown origin:
Your server: http://192.168.85.126:8000
The request origin: http://192.168.85.126 <<< no port, so it is considered different server
To fix the issue, in the settings.py, add the CSRF_TRUSTED_ORIGINS parameter like this
CSRF_TRUSTED_ORIGINS = ['http://192.168.85.126:8000', 'http://192.168.85.126']
Furthermore, if you're going to use SSL later (https), then you will also have to add CSRF_TRUSTED_ORIGINS for each https server
I have a non-simple cross-origin HTTP request to my POST method in API Gateway.
I have set up CORS support as per : https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.html
The POST request succeeds from Postman/AWS console but fails from a local development server where the origin is of type localhost.
After browsing through many answers and trying to debug i have concluded that all the required headers are present on the integration response of both POST and OPTIONS. However when making a call to OPTIONS from postman i see that it returns 500 Internal server error. This looks like a configuration issue with OPTIONS. How do i fix it? I don't have a serverless yaml, the gateway has been configured manually. Any way of resolving this via the console ?
We are catching a BigCommerce webhook event in our Google Cloud Run application. The request looks like:
Headers
host: abc-123-ue.a.run.app
AccountId: ABC
Content-Type: application/json
Password: Goodbye
Platform: BC
User-Agent: akka-http/10.1.10
Username: Hello
Content-Length: 197
Connection: keep-alive
Body
{"created_at":1594914374,"store_id":"1001005173","producer":"stores/gy68868uk5","scope":"store/product/created","hash":"139fab64ded23b3e1b8473ba24ab21bedd3f535b","data":{"type":"product","id":132}}
For some reason, this causes a 400 response from Google Cloud Run. Our application doesn't even seem to be passed the request. All other endpoints work (including other post requests).
Any ideas?
Edit
In the original post, I had the path in the host header. This was a mistake made in creating this post and not the actual value passed to us. We can only inspect the request via Requestbin (I can't find the request values anywhere in Google logs) so I'm speculating on the host value and made a mistake writing it out here.
Research so far...
So upon further testing, it seems that BigCommerce Webhooks also fail to send to any Google Cloud Function we set up. As a workaround, I'm having Pipedream catch the webhook and send the payload to our application. No problems there. This endpoint also works with mirror payloads from local and Zapier which seems to eliminate authentication errors.
We are running FastAPI on Google Run and the simplest function on Google Cloud Functions. This seems to be an error with how Google Serverless and BigCommerce Webhook Events communicate with each other. I'm just not sure how...
Here are the headers we managed to capture on one of the only times a BigCommerce Webhook Event came through to our Google Cloud Function:
Content-Length: 197
Content-Type: application/json
Host: us-central1-abc-123.cloudfunctions.net
User-Agent: akka-http/10.1.10
Forwarded: for="0.0.0.0";proto=https
Function-Execution-Id: unes7v34vzyo
X-Appengine-Country: ZZ
X-Appengine-Default-Version-Hostname: f696ddc1d56c3fd66p-tp.appspot.com
X-Appengine-Https: on
X-Appengine-Request-Log-Id: 5f10e15c00ff082ecbb02ee3a70001737e6636393664646331643536633366643636702d7470000165653637393633633164376565323033383131366437343031613365613263303a36000100
X-Appengine-Timeout-Ms: 599999
X-Appengine-User-Ip: 0.0.0.0
X-Cloud-Trace-Context: a62207698d141465d0f38488492d088b/9870406606828581415
X-Forwarded-For: 0.0.0.0
X-Forwarded-Proto: https
Accept-Encoding: gzip
Connection: close
> host: abc-123-ue.a.run.app/bigcommerce/webhooks/
This is most likely the issue. Host headers must contain only the hostname, not the request /paths.
You can clearly see this will fail:
$ curl -IvH 'Host: pdf-2wvlk7vg3a-uc.a.run.app/foo' https://pdf-2wvlk7vg3a-uc.a.run.app
...
HTTP/2 400
However if you don't craft the Host header yourself, it will work.
I got my React/Django website running on an AWS EC2 instance using Nginx and Supervisor. I'm trying to do a proper deployment rather than just put up my development environment.
When I set DEBUG to False in Django's settings.py, the supervisor logs give me the error "CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False." However, every combination I've tried so far has given me errors. I know that Nginx is essentially sending every request I get through my IP to localhost on the server (right?) so I figured just having 'localhost' would work but that gives me 404 errors when trying to retrieve the static files. Then I tried using my domain (currently just my public DNS from AWS since I haven't connected the domain yet) and that gives me a 400 (Bad Request) error. Using both also gets me a 404. Then I tried the wildcard ('*') since I figure Nginx is handling the traffic anyway but I STILL get a 404.
TL;DR table
ALLOWED_HOSTS value Result
-----------------------------------------------------
['localhost'] 404 (Not Found)
-----------------------------------------------------
['http://....amazonaws.com/'] 400 (Bad Request)
-----------------------------------------------------
['localhost', 404 (Not Found)
'http://....amazonaws.com/']
-----------------------------------------------------
['*'] 404 (Not Found)
-----------------------------------------------------
all three 404 (Not Found)
-----------------------------------------------------
['34.xxx.xx.xx'] 400 (Bad Request)
-----------------------------------------------------
I don't know what else to try. Any direction would be greatly appreciated.
I have used ShimmerCat with sc-tool to connect to my development sites as described here, and everything has worked always like a charm with it, but I also wanted to follow the "old way" configuring my /etc/hosts. In this case I had a small problem, the server ran ok, and I could access to my development site (let's say that I used https://www.example.com:4043/), but I'm also using a reverse proxy as described on this article, and on the config file reference. It redirects to a Django app I'm using. Let's say it is my devlove.yaml config file:
---
shimmercat-devlove:
domains:
www.example.com:
root-dir: site
consultant: 8080
cache-key: xxxxxxx
api.example.com:
port: 8080
The problem is that when I try to access to a URL that requests the API, a 404 response is sent from the API. Let me try to explain it through an example. I try to access to https://www.example.com:4043/country/, and on this page I do a request to the API: /api/<country>/towns/, then the API endpoint is returning a 404 response so it is not finding this URL, which does not happen when using Google Chrome with sc-tool. I had set both domains www.example.com, and api.example.com on my /etc/hosts. I have been trying to solve it, but without any luck, is there something I'm missing? Any help will be welcome. Thanks in advance.
With a bit more of data, we may be able to find the issue. In the meantime, here is a list of troubleshooting tips:
Possible issue: DNS is cached in browser, /etc/hosts is not being used (yet)
This can happen if somehow your browser has not done a DNS lookup since before you changed your /etc/hosts file. Then the connection is going to a domain in the Internet that may not have the API endpoint that you are calling.
Troubleshooting: Check ShimmerCat's log for the requests. If this is the issue, closing and opening the browser may solve the issue.
Possible issue: the host header is incorrect
ShimmerCat uses the Host header in HTTP/1.1 requests and the :authority header in HTTP/2 requests to distinguish the domains. It always discards any port number present in them. If these headers are not set or are set to a domain other than the ones ShimmerCat is configured to listen, the server will consider the situation so despicable that it will just close the connection.
Troubleshooting: This is not a 404 error, but a connection close (if trying to connect un-proxied, directly to the SSL port where ShimmerCat is listening), or a Socks Connection Failed (if trying to connect through ShimmerCat's built-in SOCKS5 proxy). In the former case, the server will print the message "Rejected request to Just https://some-domain-or-ip/some/path" in his log, using the actual value for the domain, or "Rejected request to Nothing", if no header was present. The second case is more complicated, because the SOCKS5 proxy is before the HTTP routing algorithm.
In any case, the browser will put a red line in the network panel of the developer tools. If you are accessing the server using curl, like this:
curl -k -H host:api.incorrect-domain.com https://127.0.0.1:4043/contents/blog/data-density/
or like
curl -k -H host:api.incorrect-domain.com
(notice the --http2 parameter in the second form), you will get a response:
curl: (56) Unexpected EOF
Extra-tip: There is a field for the network address in the browser's developer tools. Check it, it may tell you something!
Possible issue: something gets messed up when passing the request to the api back-end.
API backends are also sensitive to the host header, and to additional things like authentication cookies and request parameters.
Troubleshooting: A way to diagnose things is invoking ShimmerCat using the --show-proxied-headers command-line option. It makes ShimmerCat to report the proxied headers to the log:
Issuing request with headers :authority: api.example.com
:method: GET
:path: /my/api/endpoint/path/
:scheme: https
accept: */*
user-agent: curl/7.47.0
Possible issue: there are two instances or more of ShimmerCat running
...and they are using different configurations. ShimmerCat uses port sharing among several processes to increase availability. A downside of this is that is perfectly possible to mistakenly start ShimmerCat, forget about stopping it, and start it again after changing some configuration bit. The two instances will be running at the same time, and any of them will pick connections made to the listening port.
Troubleshooting: Shutdown all instances of ShimmerCat, then double-check there are none running by using the corresponding form of the ps command, and start the server with the configuration you want.