How to match subdomain with gorilla mux - regex

I need to build a route which matches two subdomains(prefix.api.example.com and prefix.api.sandbox.example.com) using gorilla mux router. So far I have the regex below but the router returns 404 on requests. Any idea why is that?
router := mux.NewRouter()
route := router.Host(`prefix.api{_:(^$|^\.sandbox$)}.example.com`)
More code
package main
import(
"github.com/gorilla/mux"
"net/http"
)
type handler struct{}
func (_ handler)ServeHTTP(w http.ResponseWriter, r *http.Request){
w.Write([]byte("hello world"))
w.WriteHeader(200)
}
func main() {
router := mux.NewRouter().StrictSlash(true)
route := router.Host(`prefix.api{_:(^$|^\.sandbox$)}.example.com`)
route.Handler(handler{})
http.Handle("/", router)
panic(http.ListenAndServe(":80", nil))
}
Request:
$ curl prefix.api.sandbox.example.com/any -v
* Trying 127.0.0.1...
* Connected to prefix.api.sandbox.example.com (127.0.0.1) port 80 (#0)
> GET /some HTTP/1.1
> Host: prefix.api.sandbox.example.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 01 Jun 2016 22:08:21 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host prefix.api.sandbox.example.com left intact

The ^ and $ metacharacters for matching beginning and end of lines should be removed, the parens can be as well.
route := router.Host(`prefix.api{_:|\.sandbox}.example.com`)`
My hosts file:
○ grep prefix /etc/hosts
127.0.0.1 prefix.api.example.com
127.0.0.1 prefix.api.sandbox.example.com
127.0.0.1 prefix.api.xsandbox.example.com
Gives me the following:
○ curl prefix.api.example.com:8000
hello world%
○ curl prefix.api.sandbox.example.com:8000
hello world%
○ curl prefix.api.xsandbox.example.com:8000
404 page not found
Updated:
Here are the regexes generated by the two different .Host()'s:
route := router.Host(`prefix.api{_:(^$|^\.sandbox$)}.example.com`)
regexp: ^prefix\.api(?P<v0>(^$|^\.sandbox$))\.example\.com$
route := router.Host(`prefix.api{_:|\.sandbox}.example.com`)
regexp: ^prefix\.api(?P<v0>|\.sandbox)\.example\.com$
Example tests for both regexes can be played with
here at play.golang

Related

AWS Api Gateway cannot rewrite path - 400 Bad Request error

I'm trying to set up an AWS API Gateway for something which was previously handled by an nginx reverse proxy. My endpoints are EC2 instances inside a VPC. I've already set it up so the gateway can access these instances.
The previous nginx setup looked like this:
http {
server {
listen 80;
location /host1/ {
proxy_pass http://host1:8000/;
}
location /host2/ {
proxy_pass http://host2:8070/;
}
...
}
}
The Problem arises when I try to rewrite the request path. I've set up a test route in the Gateway: ANY /test/{proxy+}, which I passed to the corresponding EC2 instance. I've verified, that requests pass through, but they contain the complete paths of the requests:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:15:05] "GET /test/hello_world/test/a HTTP/1.1" 404 -
(the 404 is expected, the important part is the request hitting the server)
I then tried to rewrite the request path to remove the leading /test using a parameter mapping: I specified "all incoming requests", Parameter to modify: path, Modification type: overwrite, Value: $request.path.proxy (the catch-all field defined in the route).
Now I get a 400 error, and the requests don't hit my server anymore:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 400
< date: Sat, 18 Dec 2021 09:19:53 GMT
< content-type: text/html
< content-length: 122
< server: awselb/2.0
< apigw-requestid: KiclDhxXFiAEMhg=
<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
# server:
-nothing-
When I map the $request.path.proxy to querystring.path, instead of path the requests hit the server:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:21:42] "GET /test/hello_world/test/a?path=hello_world%2Ftest%2Fa HTTP/1.1" 404 -
notice the value of the path query parameter is exactly the correct value which I would have wanted to replace the original requests path.
Is this a bug with AWS, or am I just missing some documentation, stating that you cannot rewrite path that way? Notably, when the {proxy+} path parameter is empty, requests get routed through correctly...
The problem was with the value of the path rewrite: It should have been /$request.path.proxy instead of $request.path.proxy.

Downloading empty file when accessing to subdirectory

I have trouble with setting default file for subdirectory.
http://d2770ni5llmjzr.cloudfront.net/songinfo/index.html works properly, but http://d2770ni5llmjzr.cloudfront.net/songinfo ends up with downloading blank file.
I already set static domain of my bucket as origin of distribution.
Following is log of curl -v http://d2770ni5llmjzr.cloudfront.net/songinfo
* Trying 13.225.105.93...
* TCP_NODELAY set
* Connected to d2770ni5llmjzr.cloudfront.net (13.225.105.93) port 80 (#0)
> GET /songinfo HTTP/1.1
> Host: d2770ni5llmjzr.cloudfront.net
> User-Agent: curl/7.55.1
> Accept: */*
>
HTTP/1.1 302 Moved Temporarily
Content-Type: text/html; charset=utf-8
Content-Length: 313 < Connection: keep-alive
x-amz-error-code: Found
x-amz-error-message: Resource Found
Location: /songinfo/
Date: Wed, 31 Jul 2019 20:21:42 GMT
Server: AmazonS3
Age: 9262
X-Cache: Hit from cloudfront
Via: 1.1 2f061b9f7c9e6f3de68229cddaa32e46.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: ICN54-C1
X-Amz-Cf-Id: KgKM9xl5ggcwOWvoUng9rWiVE12UI0q0oPvCUJ2IvJxmyPlJtvmMRg==
<html> <head><title>302 Moved Temporarily</title></head> <body> <h1>302 Moved Temporarily</h1> <ul> <li>Code: Found</li> <li>Message: Resource Found</li> <li>RequestId: 454F29ADAC061274</li> <li>HostId: FsYd6DZeb9nuWA/wQazkp+doZmsriSoakDrcsyQy7jkFQFw/y8ngKW0lR1yT1sYprBJ6CtjK3KU=</li> </ul> <hr/> </body> </html>
* Connection #0 to host d2770ni5llmjzr.cloudfront.net left intact
Web files named index (ex. index.html, index.php) are the default files for their directory. The index.html file loaded automatically on safari, chrome, and firefox for me.
>HTTP/1.1 302 Moved Temporarily
>Location: /songinfo/
This is an HTTP redirect, and it is exactly what S3 is supposed to do when you ask for a "directory" that will serve an index document, but you don't supply the trailing slash in the request.
Your site is working correctly, but curl doesn't follow these redirects by default. You need to tell it to, with the --location (long form) or -L (short form) option.
curl -v -L https://...
You should then find that curl does what you expect.

django : don't return cookie for a particular endpoint

I need to return a response from Django without returning a cookie.
I'm trying to implement a webhook client API that requires:
the use of https
response within 5 seconds
no body in the response
no cookies in the response headers
a 401 unauthorised status code for invalid hmac signatures
I'm working on Django 1.10 (soon to be upgraded to 2.x) where the rest of the app is protected by user validation via sessions.
Part of the endpoint view is as follows:
response200 = HttpResponse(status=200)
response401 = HttpResponse(status=401)
response401.close() # attempt not to set cookie
signature = request.META.get('HTTP_WEBHOOK_SIGNATURE')
if not request.method == 'POST':
return response401
if not signature:
return response401
and so on.
However my attempt to avoid setting the session using response401.close() doesn't work. I've also tried del response401['Set-Cookie']see Django docs
The cookie LocalTest... is still set in this curl session:
$ curl -d "param1=value1&param2=value2" \
-H "webhook-signature: $SIGVAL" \
-H "Content-Type: application/x-www-form-urlencoded" \
-X POST http://127.0.0.1:8000/invoices/webhookendpoint \
-w "\n" -v
...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> POST /invoices/webhookendpoint HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.52.1
> Accept: */*
> x-xero-signature: ZSlYlcsLbYmas53uHNrBFiVL0bLbIKetQI6x8JausfA=n
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 27
>
* upload completely sent off: 27 out of 27 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Date: Thu, 11 Apr 2019 08:32:50 GMT
< Server: WSGIServer/0.1 Python/2.7.13
< Vary: Cookie
< Content-Type: text/html; charset=utf-8
< Set-Cookie: LocalTest=gwx7jhsshy2qvtct1rmzv86h7xshe6ot; httponly; Path=/
<
* Curl_http_done: called premature == 0
* Closing connection 0
It appears that this works:
# ensure no cookie header is set
del request.session
response200 = HttpResponse(status=200)
response401 = HttpResponse(status=401)
...
as shown in the curl response:
< HTTP/1.0 200 OK
< Date: Thu, 11 Apr 2019 08:49:28 GMT
< Server: WSGIServer/0.1 Python/2.7.13
< Content-Type: text/html; charset=utf-8
<
Naturally, if you go to this endpoint as a logged in user, you will have to log in again.

Pre-Flight OPTIONS requests are canceled in Chrome (CORS already supported)

I'm working on a project that use Angular + Django(Django Rest Framework). During the development, the CORS support is done by using django-cors-headers, with CORS_ORIGIN_ALLOW_ALL = True and CORS_ALLOW_CREDENTIALS = True.
When I'm trying to send POST requests to create some resources in frontend (Angular), some pre-flight OPTIONS requests are sent by Chrome and responded successfully by backend server (python manage.py runserver), but others are not. These requests are canceled due to unknown reason, backend server logs indicate that requests are received and accepted by server, details are shown in fig below.
The headers of failed requests are shown below.
However, if a copy the content of the headers and try sending it with curl, it works as expected.
$ curl -v -X OPTIONS -H "Access-Control-Request-Headers: authorization,content-type" -H "Access-Control-Request-Method: POST" -H "DNS: 1" -H "Origin: http://localhost:4200" -H "Referer: http://localhost:4200" -H "User-Agent: Mozilla/5.0" http:/localhost:8000/api/user-permissions/
* Unwillingly accepted illegal URL using 1 slash!
* Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8000 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> OPTIONS /api/user-permissions/ HTTP/1.1
> Host: localhost:8000
> Accept: */*
> Access-Control-Request-Headers: authorization,content-type
> Access-Control-Request-Method: POST
> DNS: 1
> Origin: http://localhost:4200
> Referer: http://localhost:4200
> User-Agent: Mozilla/5.0
>
< HTTP/1.1 200 OK
< Date: Wed, 20 Feb 2019 02:47:39 GMT
< Server: WSGIServer/0.2 CPython/3.7.1
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Vary: Origin
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: http://localhost:4200
< Access-Control-Allow-Headers: accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with
< Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT
< Access-Control-Max-Age: 86400
<
* Connection #0 to host localhost left intact
Any ideas how this happen? Thx.
Sample Code:
// The method of the component that invokes the methods of PermissionService.
/** Update selected user's permissions. */
updatePermissions() {
const diff = this.diffPermissions();
const toBeCreated = diff[0];
const toBeDeleted = diff[1];
this.isLoading = true;
zip(
this.permissionService.createUserPermissions(toBeCreated),
this.permissionService.deleteUserPermissions(toBeDeleted),
).pipe(
map(() => true),
catchError((err: HttpErrorResponse) => {
alert(err.message);
return observableOf(false);
}),
).subscribe(succeed => {
this.isLoading = false;
});
}
// The methods of PermissionService that issue the HTTP requests.
createUserPermission(req: UserPermissionRequest) {
return this.http.post(`${environment.API_URL}/user-permissions/`, req);
}
createUserPermissions(reqs: UserPermissionRequest[]) {
// TODO(youchen): Evaluate the performance cost.
return forkJoin(reqs.map(req => this.createUserPermission(req)));
}
deleteUserPermission(permissionId: number) {
return this.http.delete(`${environment.API_URL}/user-permissions/${permissionId}/`);
}
deleteUserPermissions(permissionIds: number[]) {
// TODO(youchen): Evaluate the performance cost.
return forkJoin(permissionIds.map(id => this.deleteUserPermission(id)));
}
Found the cause: zip() with no parameters
In my case, I'm using zip to combine creations and deletions, see:
const createRequests = [c1, c2];
const deleteRequests = [d1, d2];
zip(
this.service.create(createRequests),
this.service.delete(deleteRequests),
)....
---
service.ts
create(reqs: CreateRequest[]) {
return zip(...reqs.map(req => this.createSingle(req));
}
delete(reqs: DeleteRequest[]) {
return zip(...reqs.map(req => this.deleteSingle(req));
}
But if one of the createRequests and deleteRequests is an empty list, this logic will go wrong. For example, if createRequests is empty while deleteRequests isn't, all HTTP requests fired by this.service.delete(deleteRequests) will be canceled due to an empty zip() is returned by this.service.create(createRequests).
Solution:
The solution to this problem is that we check the length of the reqs and return other observable instaned.
Fixed code:
create(reqs: CreateRequest[]) {
if (reqs.length === 0) return of([]);
return zip(...reqs.map(req => this.createSingle(req));
}
delete(reqs: DeleteRequest[]) {
if (reqs.length === 0) return of([]);
return zip(...reqs.map(req => this.deleteSingle(req));
}

POST url encoded form to Amazon API Gateway

I'm creating a webhook to receive notifications from a 3rd-party service, they sent the data in the body of a POST with content type application/x-www-form-urlencoded.
But it generates the same error:
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#456fe137; line: 1, column: 6]"}
I could reproduce the error with the following curl call:
% curl -v -X POST -d 'name=Ignacio&city=Tehuixtla' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.63...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.63) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Length: 180
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:29:40 GMT
< x-amzn-RequestId: cd4d9232-c5ba-11e5-a158-b9b39f0b0599
< X-Cache: Error from cloudfront
< Via: 1.1 1915b8b49d2fbff532431a79650103eb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: cxU2_b5DzIw4M_n3hJBFXTu9AVRBL3GpbQqUId9IxgS004DfLYqYmg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#d92973b; line: 1, column: 6]"}
If I wrap the body with double-quotes it works fine:
% curl -v -X POST -d '"name=Ignacio&city=Tehuixtla"' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.19...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.19) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 6
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:33:20 GMT
< x-amzn-RequestId: 50610606-c5bb-11e5-b140-5d837ffe26ed
< X-Cache: Miss from cloudfront
< Via: 1.1 a670cda0e28541e40881b95b60c672b7.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: mCLKL4eOnpUMd15IXQZw0RStJHw9Vdf3ivdCl37dcmno2JFOfxw0Vg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
"true"%
The lamba has only one line:
context.succeed('true');
How can I make the api gateway do not treat the body as json?
I tried the documentation about template mapping with no success, I even tried to convert it to a static template, with no variables at all! In all cases the error happens before getting to my code.
Try to set mapping template as following:
{
"body" : $input.json('$')
}
This would convert you string into json and pass to lambda.
From amazon docs:
$input.json(x) function evaluates a JSONPath expression and returns the results as a JSON string.
This is not entirely related, but if you are new to Amazon API Gateway, one additional step I did not know was required was to (re) deploy your API after adding the mapping template as others have suggested (in the case you had previously deployed your API). This cost me a bunch of debugging time as I did not understand why I was continuing to get this error even after making the suggestions posted here.
If using the AWS Console,
navigate to any pane within your API
Select Actions menu at the top
Select Deploy API from the menu, choose the relevant stage and confirm
The mapping template to make form data work is pretty complicated. Here is a gist: https://gist.github.com/ryanray/668022ad2432e38493df
Also, you can see this post I wrote that has an example of how to integrate with Slack(their hooks send a POST as form data to API Gateway): http://www.ryanray.me/serverless-slack-integrations
In the API Gateway, select the POST method for your resource, select Integration Request and create a new Mapping Template for application/x-www-form-urlencoded:
#set($body = $input.path('$'))
#set($jsonString = $util.urlencode($body))
#set($json = $util.parsejson($jsonString))
{
"body" : $json,
}
Alternatively, you can simply pass the url encoded string:
#set($body = $input.path('$'))
{
"body" : "$body",
}
and url decode and parse the JSON in your lambda.