Golang app on elastic beanstalk seems to be receiving double encoded requests - amazon-web-services

I'm using gorilla/mux inside a Golang application to retrieve variables out of my routes, like so:
func main() {
router := mux.NewRouter()
router.HandleFunc("/items/{name}", itemHandler)
log.Fatal(http.ListenAndServe(":5000", router)
}
func itemHandler(w http.ResponsWriter, r *http.Request) {
name := mux.Vars(r)["name"]
fmt.Println("name is: ", name)
}
If I navigate to /items/super%20duper on my local VM the console output is name is: super duper, as I would expect. But when I run this on our Elastic Beanstalk instance and go to the same URL the console output is name is: super%20duper.
We tried changing the proxy_pass entry in the nginx config thinking maybe nginx was not passing the request URI exactly as received, but that had no effect.
If anyone else has seen the same issue I would love to know how you solved it.

You may use func QueryUnescape(s string) (string, error) from "net/url" package:
QueryUnescape:
QueryUnescape does the inverse transformation of QueryEscape,
converting %AB into the byte 0xAB and '+' into ' ' (space). It returns
an error if any % is not followed by two hexadecimal digits.
package main
import (
"fmt"
"net/url"
)
func main() {
s, err := url.QueryUnescape("super%20duper")
if err != nil {
panic(err)
}
fmt.Println(s) // super duper
}
output:
super duper

Related

Cloud Function robots.txt 404 page not found issue

I have deployed one simple app via go language in Cloud Function.
*A robots.txt file is also included when distributing the app.
In this regard, a simple app normally shows the image below.
But it shows 404 page not found even though the robots.txt file is normal.
Does anyone know if the robots.txt file itself can't be served by Cloud Function?
##Function.go
// Package p contains an HTTP Cloud Function.
package p
import (
"encoding/json"
"fmt"
"html"
"io"
"log"
"net/http"
)
// HelloWorld prints the JSON encoded "message" field in the body
// of the request or "Hello, World!" if there isn't one.
func HelloWorld(w http.ResponseWriter, r *http.Request) {
var d struct {
Message string `json:"message"`
}
if err := json.NewDecoder(r.Body).Decode(&d); err != nil {
switch err {
case io.EOF:
fmt.Fprint(w, "Hello World!")
return
default:
log.Printf("json.NewDecoder: %v", err)
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return
}
}
if d.Message == "" {
fmt.Fprint(w, "Hello World!")
return
}
fmt.Fprint(w, html.EscapeString(d.Message))
}
##go.mod
module example.com/cloudfunction
##robots.txt
User-agent: *
Disallow: /
User-agent: Mediapartners-Google
Allow: /
Thank you in advance to those who have responded.
Cloud Functions isn't a webserver and you can't serve file directly like that. You need to process them in Go.
For that you need to know the code structure of the functions. All the original files are stored in /workspace/serverless_function_source_code directory. So, you can simply serve them using the URL path like that
var functionSourceCodeDir = "/workspace/serverless_function_source_code"
func ServeFile(w http.ResponseWriter, r *http.Request) {
file := r.URL.Path
if file == "/" {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "you must provide a file pathname")
return
}
http.ServeFile(w, r, functionSourceCodeDir+file)
}
Cloud Functions is not a web server.
You cannot just add files and expect those will be served as you would you do it with an NGINX or Apache web server.
When you reach your CF endpoint the code you added will be executed and you'll get a result from that and in any case the files you add will be used by the code but not served.
I'd suggest to first understand what Cloud Functions is intended for and as an alternative you may want to go with App Engine Standard.
Another way to go is to use a workaround to handle the routes and that all stuff as guillaume shows in this article for Python + Flask:
from flask import Flask, request
#Define an internal Flask app
app = Flask("internal")
#Define the internal path, idiomatic Flask definition
#app.route('/user/<string:id>', methods=['GET', 'POST'])
def users(id):
print(id)
return id, 200
#Comply with Cloud Functions code structure for entry point
def my_function(request):
#Create a new app context for the internal app
internal_ctx = app.test_request_context(path=request.full_path,
method=request.method)
#Copy main request data from original request
#According to your context, parts can be missing. Adapt here!
internal_ctx.request.data = request.data
internal_ctx.request.headers = request.headers
#Activate the context
internal_ctx.push()
#Dispatch the request to the internal app and get the result
return_value = app.full_dispatch_request()
#Offload the context
internal_ctx.pop()
#Return the result of the internal app routing and processing
return return_value
And it is important what he mentions at the end:
However, keep in mind that’s a workaround, even a hack, and Cloud Functions aren’t designed for this purpose.
Here's a sample for Go:
package function
import (
"net/http"
)
var mux = newMux()
//F represents cloud function entry point
func F(w http.ResponseWriter, r *http.Request) {
mux.ServeHTTP(w, r)
}
func newMux() *http.ServeMux {
mux := http.NewServeMux()
mux.HandleFunc("/one", one)
mux.HandleFunc("/two", two)
mux.HandleFunc("/subroute/three", three)
return mux
}
func one(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("hello from one"))
}
func two(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("hello from two"))
}
func three(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("hello from three"))
}

Porting a golang server app to aws lambda + api gateway

I am trying to port a server running a small golang app to AWS Lambda. I am not very familiar with golang and to deploy to a server I have just followed the instructions in the repo.
It runs a server with net/http, the main.go is as follows:
func main() {
r := new(route.Router)
r.HandleFunc("/squares", squares.Random)
// ... more Handlers
log.Println("Listening on " + os.Getenv("PORT"))
err := http.ListenAndServe(":"+os.Getenv("PORT"), r)
if err != nil {
log.Fatal("ListenAndServe:", err)
}
}
Now I have found this drop-in replacement repo on Github for ListenAndServe, apex/gateway, but I think I am missing a fundamental step in making it work.
What I've done is download and import of the library
import (
...
"github.com/apex/gateway/v2"
)
then simply replace the function in main, zip and upload to aws lambda
func main() {
r := new(route.Router)
r.HandleFunc("/squares", squares.Random)
// ... more Handlers
// log.Println("Listening on 8080")
err := gateway.ListenAndServe(":8080", r)
if err != nil {
log.Fatal("ListenAndServe:", err)
}
}
then I set up an http API Gateway and link to the lambda function.
It doesn't work. I think I'm missing something but I can't figure out what. From the example on the apex/gateway repo, I don't see what I'm missing.
The app is Tinygraphs fwiw.
Thank you
Edit:
As per Adrians comment, when I go to the api link I get
{"message":"Not Found"}
Hey User I believe you are not understanding how API Gateway and Lambda works properly
You do not need to set up a route listening on 8080 that is effectively what API Gateway is doing and then forwarding the request to your code running on the lambda. Which is normally in this format:
package main
import (
"fmt"
"context"
"github.com/aws/aws-lambda-go/lambda"
)
type MyEvent struct {
Name string `json:"name"`
}
func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
return fmt.Sprintf("Hello %s!", name.Name ), nil
}
func main() {
lambda.Start(HandleRequest)
}
Was a pretty stupid mistake.
I had defined the api path as / instead of /{proxy+}, so it wasn't accepting a non-root url whereas the invoke url is like this:
https://XXXXXXXX.execute-api.us-east-1.amazonaws.com/squares/tinygraphs?theme=frogideas&numcolors=4&size=220&fmt=svg
Thank you everyone who looked at this.
(regarding the port, according to this, the port specified is just ignored by gateway.ListenAndServe)

API Gateway HTTP client request with IAM auth with Go

Hello StackOverflow AWS Gophers,
I'm implementing a CLI with the excellent cobra/viper packages from spf13. We have an Athena database fronted by an API Gateway endpoint, which authenticates with IAM.
That is, in order to interact with its endpoints by using Postman, I have to define AWS Signature as Authorization method, define the corresponding AWS id/secret and then in the Headers there will be X-Amz-Security-Token and others. Nothing unusual, works as expected.
Since I'm new to Go, I was a bit shocked to see that there are no examples to do this simple HTTP GET request with the aws-sdk-go itself... I'm trying to use the shared credentials provider (~/.aws/credentials), as demonstrated for the S3 client Go code snippets from re:Invent 2015:
req := request.New(nil)
How can I accomplish this seemingly easy feat in 2019 without having to resort to self-cooked net/http and therefore having to manually read ~/.aws/credentials or worse, go with os.Getenv and other ugly hacks?
Any Go code samples interacting as client would be super helpful. No Golang Lambda/server examples, please, there's plenty of those out there.
Unfortunately, it seems that the library has been updated since the accepted answer was written and the solution no longer is the same. After some trial and error, this appears to be the more current method of handling the signing (using https://pkg.go.dev/github.com/aws/aws-sdk-go-v2):
import (
"context"
"net/http"
"time"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/aws/signer/v4"
)
func main() {
// Context is not being used in this example.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
// Handle error.
}
credentials, err := cfg.Credentials.Retrieve(context.TODO())
if err != nil {
// Handle error.
}
// The signer requires a payload hash. This hash is for an empty payload.
hash := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
req, _ := http.NewRequest(http.MethodGet, "api-gw-url", nil)
signer := v4.NewSigner()
err = signer.SignHTTP(context.TODO(), credentials, req, hash, "execute-api", cfg.Region, time.Now())
if err != nil {
// Handle error.
}
// Use `req`
}
The solution below uses aws-sdk-go-v2
https://github.com/aws/aws-sdk-go-v2
// A AWS SDK session is created because the HTTP API is secured using a
// IAM authorizer. As such, we need AWS client credentials and a
// session to properly sign the request.
cfg, err := external.LoadDefaultAWSConfig(
external.WithSharedConfigProfile(profile),
)
if err != nil {
fmt.Println("unable to create an AWS session for the provided profile")
return
}
req, _ := http.NewRequest(http.MethodGet, "", nil)
req = req.WithContext(ctx)
signer := v4.NewSigner(cfg.Credentials)
_, err = signer.Sign(req, nil, "execute-api", cfg.Region, time.Now())
if err != nil {
fmt.Printf("failed to sign request: (%v)\n", err)
return
}
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Printf("failed to call remote service: (%v)\n", err)
return
}
defer res.Body.Close()
if res.StatusCode != 200 {
fmt.Printf("service returned a status not 200: (%d)\n", res.StatusCode)
return
}
The first argument to request.New is aws.Config, where you can send credentials.
https://github.com/aws/aws-sdk-go/blob/master/aws/request/request.go#L99
https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config
There are multiple ways to create credentials object: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
For example using static values:
creds:= credentials.NewStaticCredentials("AKID", "SECRET_KEY", "TOKEN")
req := request.New(aws.Config{Credentials: creds}, ...)
I'm pretty new to go myself (3rd day learning go) but from watching the video you posted with the S3 example and reading through the source code (for the s3 service and request module) here is my understanding (which I'm hoping helps).
If you look at the code for the s3.New() function aws-sdk-go/service/s3/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *S3 {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, .SigningName) }
As opposed to request.New() function aws-sdk-go/aws/request/request.go
func New(cfg aws.Config, clientInfo metadata.ClientInfo, handlers Handlers,
retryer Retryer, operation *Operation, params interface{}, data interface{}) *Request { ...
As you can see in the s3 scenario the *aws.Config struct is a pointer, and so is probably initialized / populated elsewhere. As opposed to the request function where the aws.Config is a parameter. So I am guessing the request module is probably a very low level module which doesn't get the shared credentials automatically.
Now, seeing as you will be interacting with API gateway I had a look at that service specifically to see if there was something similar. I looked at aws-sdk-go/service/apigateway/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *APIGateway {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, c.SigningName) }...
Which looks pretty much the same as the s3 client, so perhaps try using that and see how you go?

Trying to log into Amazon.com to pull data, but getting an Enable Cookies response (using Go)

I'm trying to use Go to log into my account on Amazon to automatically pull some information, but I'm having trouble logging in because it complains about cookies. Here's a sanitized version of the code I was using:
package main
import (
"bytes"
"io/ioutil"
"net/http"
"net/http/cookiejar"
"net/url"
"strconv"
)
func CheckThis(AmazonUsername string, AmazonPassword string) error {
var LogonURL string
// Set the url
LogonURL = "https://www.amazon.com/ap/signin"
// Craft some form data
form := url.Values{}
form.Add("appAction", "SIGNIN")
form.Add("email", AmazonUsername)
form.Add("password", AmazonPassword)
form.Add("create", "0")
form.Add("appActionToken", “$VALUE”)
form.Add("openid.pape.max_auth_age", "$VALUE==")
form.Add("openid.identity", "$VALUE=")
form.Add("openid.assoc_handle", "$VALUE")
form.Add("openid.mode", "$VALUE")
form.Add("openid.ns.pape", "$VALUE==")
form.Add("openid.claimed_id", "$VALUE=")
form.Add("pageId", "$VALUE")
form.Add("openid.ns", "$VALUE=")
// Amazon sells cookies
cookieJar, _ := cookiejar.New(nil)
// Create a new client with the cookiejar in the struct...
client := &http.Client{
Jar: cookieJar,
}
// Craft the request to send to the website with the form containing login info
req, _ := http.NewRequest("POST", LogonURL, bytes.NewBufferString(form.Encode()))
// Some more headers
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("Content-Length", strconv.Itoa(len(form.Encode())))
req.Header.Add("Accept-Language", "en-US,en;q=0.8")
req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
req.Header.Add("Connection", "keep-alive")
req.Header.Add("Host", "www.amazon.com")
req.Header.Add("Referer", "https://www.amazon.com/ap/signin")
req.Header.Add("Upgrade-Insecure-Requests", "1")
req.Header.Add("Origin", "https://www.amazon.com")
req.Header.Add("Cache-Control", "max-age=0")
// And we're off to the races...
resp, _ := client.Do(req)
// What was in the response?
charResponse, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
// Write response body to a text file with title of…
_ = WriteOutputToFile(string(charResponse), “response.html")
// All done!
return nil
}
The $VALUE entries are because I'm not sure if the strings are significant to my account so I removed them; these are values I pulled from the developer tools of a Chrome login session. I also removed err checks for brevity.
The reply page (opening response.html on my drive within Chrome) looks like this:
What am I missing in order to keep the cookie with the client req/resp for the sign-in and later pages?
Or am I missing something where the response page I save is trying to pull elements from Amazon when I render the HTML, and the cookie issue is because the browser would be missing cookie information when I'm trying to view the results from the Go application?
I'm pretty sure Amazon uses different data every time when you try to login so better to parse login form. Here is example
package main
import (
"bytes"
"io/ioutil"
"net/http"
"net/http/cookiejar"
"net/url"
"github.com/PuerkitoBio/goquery"
"log"
)
func checkError(err error) {
if err != nil {
log.Fatal(err)
}
}
func CheckThis(AmazonUsername string, AmazonPassword string) error {
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
res, err := client.Get("https://www.amazon.com/gp/sign-in.html/ref=ord_cart_unrec_signin")
checkError(err)
doc, err := goquery.NewDocumentFromResponse(res)
form := url.Values{}
doc.Find("form[name='signIn'] input").Each(func(i int, s *goquery.Selection) {
name, exist := s.Attr("name")
if exist {
value, exist := s.Attr("value")
if exist {
form.Add(name, value)
}
}
})
form.Set("email", AmazonUsername)
form.Set("password", AmazonPassword)
req, _ := http.NewRequest("POST", "https://www.amazon.com/ap/signin", bytes.NewBufferString(form.Encode()))
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
res, err = client.Do(req)
checkError(err)
defer res.Body.Close()
charResponse, _ := ioutil.ReadAll(res.Body)
ioutil.WriteFile("response.html", charResponse, 0777)
return nil
}
func main() {
CheckThis("", "")
}
Amazon's index is setting this (and much more) cookies on my browser:
I suppose you need to set them into cookiejar in order to simulate a browser (amazon excepts that from your request, so is telling you have no cookies enabled because there isn't a required security value).
I agree with #JimB, you should be using Amazon SDK. Is there any reason why you are not doing this way?
Best regards.

App engine Local Unit test different instances [GO]

I have an issue with testing separate methods, each test case is running on a different instance and address.
I'm looking for a way to set up the API address in order to preform the tests on the same API server.
I assuming that this warning is part of the issue.
WARNING 2015-11-04 18:15:25,003 devappserver2.py:779] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
This command will set the API server but I can't do the same for test...
dev_appserver.py . --api_port 55555
Using aetest.NewInstance you can make sure that all your unit tests share a single instance:
var inst aetest.Instance
func TestMain(m *testing.M) {
var err error
inst, err = aetest.NewInstance(nil)
if err != nil {
log.Fatalf("aetest.NewInstance: %v", err)
}
e := m.Run()
inst.Close()
os.Exit(e)
}
func TestMyTest(t *testing.T) {
req, err := inst.NewRequest("GET", "/foo/bar", nil)
// etc.
}