How to verify a JWT Token from AWS Cognito in Go? - amazon-web-services

How can I validate and get info from a JWT received from Amazon Cognito?
I have setup Google authentication in Cognito, and set the redirect uri to to hit API Gateway, I then receive a code which I POST to this endpoint:
https://docs.aws.amazon.com/cognito/latest/developerguide/token-endpoint.html
To receive the JWT token, in a RS256 format. I am now struggling to validate, and parse the token in Golang. I’ve tried to parse it using jwt-go, but it appears to support HMAC instead by default and read somewhere that they recommend using frontend validation instead. I tried a few other packages and had similar problems.
I came across this answer here: Go Language and Verify JWT but assume the code is outdated as that just says panic: unable to find key.
jwt.io can easily decode the key, and probably verify too. I’m not sure where the public/secret keys are as Amazon generated the token, but from what I understand I need to use a JWK URL to validate too? I’ve found a few AWS specific solutions, but they all seem to be hundreds of lines long. Surely it isn’t that complicated in Golang is it?

Public keys for Amazon Cognito
As you already guessed, you'll need the public key in order to verify the JWT token.
https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html#amazon-cognito-user-pools-using-tokens-step-2
Download and store the corresponding public JSON Web Key (JWK) for your user pool. It is available as part of a JSON Web Key Set (JWKS).
You can locate it at
https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json
Parse keys and verify token
That JSON file structure is documented in the web, so you could potentially parse that manually, generate the public keys, etc.
But it'd probably be easier to just use a library, for example this one:
https://github.com/lestrrat-go/jwx
And then jwt-go to deal with the JWT part: https://github.com/dgrijalva/jwt-go
You can then:
Download and parse the public keys JSON using the first library
keySet, err := jwk.Fetch(THE_COGNITO_URL_DESCRIBED_ABOVE)
When parsing the token with jwt-go, use the "kid" field from the JWT header to find the right key to use
token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodRS256); !ok {
return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
}
kid, ok := token.Header["kid"].(string)
if !ok {
return nil, errors.New("kid header not found")
}
keys := keySet.LookupKeyID(kid);
if !ok {
return nil, fmt.Errorf("key with specified kid is not present in jwks")
}
var publickey interface{}
err = keys.Raw(&publickey)
if err != nil {
return nil, fmt.Errorf("could not parse pubkey")
}
return publickey, nil

The type assertion in the code provided by eugenioy and Kevin Wydler did not work for me: *jwt.SigningMethodRS256 is not a type.
*jwt.SigningMethodRS256 was a type in the initial commit. From the second commit on (back in July 2014) it was abstracted and replaced by a global variable (see here).
This following code works for me:
func verify(tokenString string, keySet *jwk.Set) {
tkn, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
if token.Method.Alg() != "RSA256" { // jwa.RS256.String() works as well
return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
}
kid, ok := token.Header["kid"].(string)
if !ok {
return nil, errors.New("kid header not found")
}
keys := keySet.LookupKeyID(kid)
if len(keys) == 0 {
return nil, fmt.Errorf("key %v not found", kid)
}
var raw interface{}
return raw, keys[0].Raw(&raw)
})
}
Using the following dependency versions:
github.com/dgrijalva/jwt-go/v4 v4.0.0-preview1
github.com/lestrrat-go/jwx v1.0.4

This is what I did with only the latest (v1.0.8) github.com/lestrrat-go/jwx. Note that github.com/dgrijalva/jwt-go does not seem to be maintained anymore and people are forking it to make the updates they need.
package main
import (
...
"github.com/lestrrat-go/jwx/jwk"
"github.com/lestrrat-go/jwx/jwt"
)
...
keyset, err := jwk.Fetch("https://cognito-idp." + region + ".amazonaws.com/" + userPoolID + "/.well-known/jwks.json")
parsedToken, err := jwt.Parse(
bytes.NewReader(token), //token is a []byte
jwt.WithKeySet(keyset),
jwt.WithValidate(true),
jwt.WithIssuer(...),
jwt.WithClaimValue("key", value),
)
//check err as usual
//here you can call methods on the parsedToken to get the claim values
...
Token claim methods

eugenioy's answer stopped working for me because of this refactor. I ended up fixing with something like this
token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodRS256); !ok {
return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
}
kid, ok := token.Header["kid"].(string)
if !ok {
return nil, errors.New("kid header not found")
}
keys := keySet.LookupKeyID(kid);
if len(keys) == 0 {
return nil, fmt.Errorf("key %v not found", kid)
}
// keys[0].Materialize() doesn't exist anymore
var raw interface{}
return raw, keys[0].Raw(&raw)
})

A newer method to achieve verification and access the token is to use Gin Cognito JWT Authentication Middleware:
package main
import (
jwtCognito "github.com/akhettar/gin-jwt-cognito"
"github.com/gin-gonic/gin"
"github.com/golang-jwt/jwt"
"log"
)
func main() {
r := gin.Default()
// Create the authentication middleware
mw, err := jwtCognito.AuthJWTMiddleware(<iss>, <user_pool_id>, <region>)
if err != nil {
panic(err)
}
r.GET("/someGet", mw.MiddlewareFunc(), func(c *gin.Context) {
// Get the token
tokenStr, _ := c.Get("JWT_TOKEN")
token := tokenStr.(*jwt.Token)
// Cast the claims
claims := token.Claims.(jwt.MapClaims)
log.Printf("userCognitoId=%v", claims["cognito:username"])
log.Printf("userName=%v", claims["name"])
c.Status(http.StatusOK)
})
// By default it serves on :8080
r.Run()
}

This is what worked for me:
import (
"errors"
"fmt"
"github.com/dgrijalva/jwt-go"
"github.com/gin-gonic/gin"
"github.com/lestrrat-go/jwx/jwk"
"net/http"
"os"
)
func verifyToken(token *jwt.Token) (interface{}, error) {
// make sure to replace this with your actual URL
// https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html#amazon-cognito-user-pools-using-tokens-step-2
jwksURL := "COGNITO_JWKS_URL"
set, err := jwk.FetchHTTP(jwksURL)
if err != nil {
return nil, err
}
keyID, ok := token.Header["kid"].(string)
if !ok {
return nil, errors.New("expecting JWT header to have string kid")
}
keys := set.LookupKeyID(keyID)
if len(keys) == 0 {
return nil, fmt.Errorf("key %v not found", keyID)
}
if key := set.LookupKeyID(keyID); len(key) == 1 {
return key[0].Materialize()
}
return nil, fmt.Errorf("unable to find key %q", keyID)
}
I am calling it like this (using AWS Lambda gin) in my case. If you are using a different way of managing requests, make sure to replace that with http.Request or any other framework that you might be using:
func JWTVerify() gin.HandlerFunc {
return func(c *gin.Context) {
tokenString := c.GetHeader("AccessToken")
_, err := jwt.Parse(tokenString, verifyToken)
if err != nil {
c.AbortWithStatus(http.StatusUnauthorized)
}
}
}
This is my go.mod:
module MY_MODULE_NAME
go 1.12
require (
github.com/aws/aws-lambda-go v1.20.0
github.com/aws/aws-sdk-go v1.36.0
github.com/awslabs/aws-lambda-go-api-proxy v0.9.0
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/gin-gonic/gin v1.6.3
github.com/google/uuid v1.1.2
github.com/lestrrat-go/jwx v0.9.2
github.com/onsi/ginkgo v1.14.2 // indirect
github.com/onsi/gomega v1.10.3 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
)

Here's an example using github.com/golang-jwt/jwt, (formally known as github.com/dgrijalva/jwt-go,) and a JWKs like the one AWS Cognito provides.
It'll refresh the AWS Cognito JWKs once every hour, refresh when a JWT signed with an unknown kid comes in, and have a global rate limit of 1 HTTP request to refresh the JWKs every 5 minutes.
package main
import (
"fmt"
"log"
"time"
"github.com/golang-jwt/jwt/v4"
"github.com/MicahParks/keyfunc"
)
func main() {
// Get the JWKS URL from your AWS region and userPoolId.
//
// See the AWS docs here:
// https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html
regionID := "" // TODO Get the region ID for your AWS Cognito instance.
userPoolID := "" // TODO Get the user pool ID of your AWS Cognito instance.
jwksURL := fmt.Sprintf("https://cognito-idp.%s.amazonaws.com/%s/.well-known/jwks.json", regionID, userPoolID)
// Create the keyfunc options. Use an error handler that logs. Refresh the JWKS when a JWT signed by an unknown KID
// is found or at the specified interval. Rate limit these refreshes. Timeout the initial JWKS refresh request after
// 10 seconds. This timeout is also used to create the initial context.Context for keyfunc.Get.
options := keyfunc.Options{
RefreshErrorHandler: func(err error) {
log.Printf("There was an error with the jwt.Keyfunc\nError: %s", err.Error())
},
RefreshInterval: time.Hour,
RefreshRateLimit: time.Minute * 5,
RefreshTimeout: time.Second * 10,
RefreshUnknownKID: true,
}
// Create the JWKS from the resource at the given URL.
jwks, err := keyfunc.Get(jwksURL, options)
if err != nil {
log.Fatalf("Failed to create JWKS from resource at the given URL.\nError: %s", err.Error())
}
// Get a JWT to parse.
jwtB64 := "eyJraWQiOiJmNTVkOWE0ZSIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJLZXNoYSIsImF1ZCI6IlRhc2h1YW4iLCJpc3MiOiJqd2tzLXNlcnZpY2UuYXBwc3BvdC5jb20iLCJleHAiOjE2MTkwMjUyMTEsImlhdCI6MTYxOTAyNTE3NywianRpIjoiMWY3MTgwNzAtZTBiOC00OGNmLTlmMDItMGE1M2ZiZWNhYWQwIn0.vetsI8W0c4Z-bs2YCVcPb9HsBm1BrMhxTBSQto1koG_lV-2nHwksz8vMuk7J7Q1sMa7WUkXxgthqu9RGVgtGO2xor6Ub0WBhZfIlFeaRGd6ZZKiapb-ASNK7EyRIeX20htRf9MzFGwpWjtrS5NIGvn1a7_x9WcXU9hlnkXaAWBTUJ2H73UbjDdVtlKFZGWM5VGANY4VG7gSMaJqCIKMxRPn2jnYbvPIYz81sjjbd-sc2-ePRjso7Rk6s382YdOm-lDUDl2APE-gqkLWdOJcj68fc6EBIociradX_ADytj-JYEI6v0-zI-8jSckYIGTUF5wjamcDfF5qyKpjsmdrZJA"
// Parse the JWT.
token, err := jwt.Parse(jwtB64, jwks.Keyfunc)
if err != nil {
log.Fatalf("Failed to parse the JWT.\nError: %s", err.Error())
}
// Check if the token is valid.
if !token.Valid {
log.Fatalf("The token is not valid.")
}
log.Println("The token is valid.")
// End the background refresh goroutine when it's no longer needed.
jwks.EndBackground()
}

Related

How to use go-vcr with githubv4 ? - getting httpClient Transport

Looking at the set up for go-vcr
// Start our recorder
r, err := recorder.New("fixtures/etcd")
if err != nil {
log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it
// Create an etcd configuration using our transport
cfg := client.Config{
Endpoints: []string{"http://127.0.0.1:2379"},
HeaderTimeoutPerRequest: time.Second,
Transport: r, // Inject as transport!
}
Attempting to use this library using the githubv4 library seems at though it needs a way to handle Oauth
import "golang.org/x/oauth2"
func main() {
src := oauth2.StaticTokenSource(
&oauth2.Token{AccessToken: os.Getenv("GITHUB_TOKEN")},
)
httpClient := oauth2.NewClient(context.Background(), src)
client := githubv4.NewClient(httpClient)
// Use client...
}
I'm not sure how to get the recorder 'r' into the oauth2 client. If at all possible.
Has anyone been successful with this? I've tried passing in a httpClient with the 'r' recorder but it ends up as a 401 - looks like this default client can't do the Oauth dance.
I'd like to use the GraphQL API but can fall back to the REST API if is is easier but I just want to make sure this isn't really possible. Has anyone else been successful with this?
This issue resolved this question for me.
https://github.com/dnaeon/go-vcr/issues/59
Example below
package example_test
import (
"context"
"github.com/dnaeon/go-vcr/cassette"
"github.com/dnaeon/go-vcr/recorder"
"github.com/google/go-github/v33/github"
"github.com/stretchr/testify/require"
"golang.org/x/oauth2"
"net/http"
"path"
"testing"
)
func TestGithub(t *testing.T) {
//custom http.Transport, since github uses oauth2 authentication
ts := oauth2.StaticTokenSource(
&oauth2.Token{AccessToken: "YOUR_GITHUB_TOKEN"},
)
tr := &oauth2.Transport{
Base: http.DefaultTransport,
Source: oauth2.ReuseTokenSource(nil, ts),
}
// Start our recorder
vcrRecorder, err := recorder.NewAsMode(path.Join("testdata", "fixtures", t.Name()), recorder.ModeRecording, tr)
require.NoError(t, err)
defer vcrRecorder.Stop() // NEWLY ADDED CODE HERE
// Filter out dynamic & sensitive data/headers
// Your test code will continue to see the real access token and
// it is redacted before the recorded interactions are saved
// =====> commenting out this section has no impact on missing recording
vcrRecorder.AddSaveFilter(func(i *cassette.Interaction) error {
delete(i.Request.Headers, "Authorization")
delete(i.Request.Headers, "User-Agent")
i.Request.Headers["Authorization"] = []string{"Basic UExBQ0VIT0xERVI6UExBQ0VIT0xERVI="} //PLACEHOLDER:PLACEHOLDER
return nil
})
// custom http.client
httpClient := &http.Client{
Transport: vcrRecorder,
}
ghClient := github.NewClient(httpClient)
// =====> actual test, should create cassettes, but does not.
_, _, err = ghClient.Users.Get(context.Background(), "")
require.NoError(t, err)
}

Mocking errors with client-go fake client

I'm using client-go (the k8s client for go) to programmatically retrieve and update some secrets from my cluster. While doing this, I'm facing the need of unit-testing my code, and after some investigation I stumbled upon client-go's fake client. However, I haven't been able to mock errors yet. I've followed the instructions from this issue, but without any success.
Here you have my business logic:
func (g goClientRefresher) RefreshNamespace(ctx context.Context, namespace string) (err error, warnings bool) {
client := g.kubeClient.CoreV1().Secrets(namespace)
secrets, err := client.List(ctx, metav1.ListOptions{LabelSelector: "mutated-by=confidant"})
if err != nil {
return fmt.Errorf("unable to fetch secrets from cluster: %w", err), false
}
for _, secret := range secrets.Items {
// business logic here
}
return nil, warnings
}
And the test:
func TestWhenItsNotPossibleToFetchTheSecrets_ThenAnErrorIsReturned(t *testing.T) {
kubeClient := getKubeClient()
kubeClient.CoreV1().(*fakecorev1.FakeCoreV1).
PrependReactor("list", "secret", func(action testingk8s.Action) (handled bool, ret runtime.Object, err error) {
return true, &v1.SecretList{}, errors.New("error listing secrets")
})
r := getRefresher(kubeClient)
err, warnings := r.RefreshNamespace(context.Background(), "target-ns")
require.Error(t, err, "an error should have been raised")
}
However, when I run the test I'm getting a nil error. Am I doing something wrong?
I've finally found the error... it is in the resource name of the reactor function, I had secret and it should be the plural secrets instead... :facepalm:. So this is the correct version of the code:
func TestWhenItsNotPossibleToFetchTheSecrets_ThenAnErrorIsReturned(t *testing.T) {
kubeClient := getKubeClient()
kubeClient.CoreV1().(*fakecorev1.FakeCoreV1).
PrependReactor("list", "secrets", func(action testingk8s.Action) (handled bool, ret runtime.Object, err error) {
return true, &v1.SecretList{}, errors.New("error listing secrets")
})
// ...
}

Retrieving secret from AWS secrets manager causes issue when decoding JWT

I'm retrieving a secret from AWS Secrets Manager that's used to decode a JWT on a webserver. The program retrieves the secret correctly and I confirm its identical to the one used to encode the jwts. However, the jwt-go library is unable to decode the incoming token citing "signature invalid" despite the secrets being identical. Notably, If I hard paste the secret instead of using the secrets manager the server decodes the jwt fine. This leads me to believe there is some weird encoding issue going on between how golang handles strings and how the aws secrets manager does.
Secret in question: KZQVGKt0WglphtNyME8912pa9RlnSZ1s8Xcdqe5OnQ
Secret Retrieval Script:
func GetTokenSecret(secretName string) (string, error) {
region := "us-east-1"
//Create a Secrets Manager client
svc := secretsmanager.New(session.New(),
aws.NewConfig().WithRegion(region))
input := &secretsmanager.GetSecretValueInput{
SecretId: aws.String(secretName),
VersionStage: aws.String("AWSCURRENT"), // VersionStage defaults to AWSCURRENT if unspecified
}
// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
result, err := svc.GetSecretValue(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case secretsmanager.ErrCodeDecryptionFailure:
// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
log.Println(secretsmanager.ErrCodeDecryptionFailure, aerr.Error())
case secretsmanager.ErrCodeInternalServiceError:
// An error occurred on the server side.
log.Println(secretsmanager.ErrCodeInternalServiceError, aerr.Error())
case secretsmanager.ErrCodeInvalidParameterException:
// You provided an invalid value for a parameter.
log.Println(secretsmanager.ErrCodeInvalidParameterException, aerr.Error())
case secretsmanager.ErrCodeInvalidRequestException:
// You provided a parameter value that is not valid for the current state of the resource.
log.Println(secretsmanager.ErrCodeInvalidRequestException, aerr.Error())
case secretsmanager.ErrCodeResourceNotFoundException:
// We can't find the resource that you asked for.
log.Println(secretsmanager.ErrCodeResourceNotFoundException, aerr.Error())
}
} else {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
log.Println(err.Error())
}
return "", err
}
// Decrypts secret using the associated KMS CMK.
// Depending on whether the secret is a string or binary, one of these fields will be populated.
var secretString, decodedBinarySecret string
if result.SecretString != nil {
secretString = *result.SecretString
} else {
decodedBinarySecretBytes := make([]byte, base64.StdEncoding.DecodedLen(len(result.SecretBinary)))
len, err := base64.StdEncoding.Decode(decodedBinarySecretBytes, result.SecretBinary)
if err != nil {
log.Println("Base64 Decode Error:", err)
return "", err
}
decodedBinarySecret = string(decodedBinarySecretBytes[:len])
return decodedBinarySecret, nil
}
return secretString, nil
// Your code goes here.
}
JWT Decode:
func (s *Server) parseJWT(encodedToken string) (Student, error) {
token, err := jwt.ParseWithClaims(encodedToken, &userClaims{}, func(token *jwt.Token) (interface{}, error) {
if _, isvalid := token.Method.(*jwt.SigningMethodHMAC); !isvalid {
return nil, fmt.Errorf("Invalid token %s", token.Header["alg"])
}
return []byte(s.tokenSecret), nil
})
if err != nil {
//ERROR THROWN HERE
log.Println("Error Parsing JWT: ", err)
return Student{}, err
}

Unable to perform query on AWS athena using Golang SDK

I am new to AWS and Golang, and I am trying to create a lambda function, which will trigger AWS Athena query and email the result using AWS SES service. Even after searching for an hour, I couldn't find a working example of lambda function (in Golang) to perform a query on Athena and getting the output of the query.
While searching, I found code for the same in Java, Python and Node Js, but not in Golang.
Even the Go-SDK page redirects to Java example. But unfortunately, I don't even understand Java.
I have also looked into this AWS SDK for Go API Reference page. But I don't understand what is the flow of the program and which operation to select.
I have tried to create the program for this, this may be completely wrong, and I don't know what to do next. Below is the code -
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/athena"
)
func main() {
// Create a new session in the us-west-2 region.
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
// Create an Athena session.
client := athena.New(sess)
// Example sending a request using the StartQueryExecutionRequest method.
query := "SELECT * FROM table1 ;"
params := query
req, resp := client.StartQueryExecutionRequest(params)
err1 := req.Send()
if err1 == nil { // resp is now filled
fmt.Println(resp)
}
}
Appreciate if someone can help me to perform an Athena query and to get its result in Golang(Preferably) or can share some resource. Once I get it, I can then send an email using AWS SES.
Use this to get started.
// run as: go run main.go
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "textqldb.textqltable"
const outputBucket = "s3://bucket-name-here/"
func main() {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
return
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
query := "select * from " + table
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("query error: %v\n", err)
return
}
fmt.Println(resp)
}
#Everton's code is executing a query on Athena, and its responses are getting saved on S3 bucket and not getting returned. So, I have added the code to execute the Athena query and get the response back. Hope this may help others.
// run as: go run main.go
package main
import (
"context"
"fmt"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "<Database_Name>.<Table_Name>"
const outputBucket = "s3://bucket-name-here/"
// Execute the query and return the query ID
func executeQuery(query string) *string {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
fmt.Println("Response is: ", resp, " Error is:", err)
if err != nil {
fmt.Printf("Query Error: %v\n", err)
}
fmt.Println("Query Execution Response ID:", resp.QueryExecutionId)
return resp.QueryExecutionId
}
// Takes queryId as input and returns its response
func getQueryResults(QueryID *string) (*athena.GetQueryResultsResponse, error) {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
panic("config error")
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
params1 := &athena.GetQueryResultsInput{
QueryExecutionId: QueryID,
}
req := client.GetQueryResultsRequest(params1)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("Query Response Error: %v\n", err)
return nil, err
}
return resp, nil
}
func main() {
query := "select * from " + table
// Execute an Athena Query
QueryID := executeQuery(query)
// Get the response of the query
// Wait for some time for query completion
time.Sleep(15 * time.Second) // Otherwise create a loop and try for every x seconds
Resp, err := getQueryResults(QueryID)
if err != nil {
fmt.Printf("Error getting Query Response: %v\n", err)
} else {
fmt.Println(" \nRows:", Resp.ResultSet.Rows)
}
}

Use Golang to login to private site and pull info

I try to use golang to login in a private area of a website and pull some info, but i don't quite seem to get it right.
I manage to fetch the login page to get the csrf token, then i post the csrf token together with the login info to the login page and i login just fine. If i stop at this point, i can see the page where i am redirected. However, any subsequent calls from this point on will redirect me back to login.
The code
package main
import (
"github.com/PuerkitoBio/goquery"
"io"
_ "io/ioutil"
"log"
"net/http"
"net/url"
_ "strings"
"sync"
)
type Jar struct {
sync.Mutex
cookies map[string][]*http.Cookie
}
func NewJar() *Jar {
jar := new(Jar)
jar.cookies = make(map[string][]*http.Cookie)
return jar
}
func (jar *Jar) SetCookies(u *url.URL, cookies []*http.Cookie) {
jar.Lock()
jar.cookies[u.Host] = cookies
jar.Unlock()
}
func (jar *Jar) Cookies(u *url.URL) []*http.Cookie {
return jar.cookies[u.Host]
}
func NewJarClient() *http.Client {
return &http.Client{
Jar: NewJar(),
}
}
func fetch(w http.ResponseWriter, r *http.Request) {
// create the client
client := NewJarClient()
// get the csrf token
req, _ := http.NewRequest("GET", "http://www.domain.com/login", nil)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
doc, err := goquery.NewDocumentFromResponse(resp)
if err != nil {
log.Fatal(err)
}
csrfToken := ""
if val, ok := doc.Find(`head meta[name="csrf-token-value"]`).Attr("content"); ok {
csrfToken = val
}
// post on the login form.
resp, _ = client.PostForm("http://www.domain.com/login", url.Values{
"UserLogin[email]": {"the email"},
"UserLogin[password]": {"the password"},
"csrf_token": {csrfToken},
})
doc, err = goquery.NewDocumentFromResponse(resp)
if err != nil {
log.Fatal(err)
}
// if i stop here then i can see just fine the dashboard where i am redirected after login.
// but if i continue and request a 3rd page, then i get the login page again,
// sign that i lose the cookies and i am redirected back
// html, _ := doc.Html()
// io.WriteString(w, html)
// return
// from this point on, any request will give me the login page once again.
// i am not sure why since the cookies should be set and sent on all requests
req, _ = http.NewRequest("GET", "http://www.domain.com/dashboard", nil)
resp, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
doc, err = goquery.NewDocumentFromResponse(resp)
if err != nil {
log.Fatal(err)
}
html, _ := doc.Html()
io.WriteString(w, html)
}
func main() {
http.HandleFunc("/", fetch)
http.ListenAndServe("127.0.0.1:49721", nil)
}
Any idea what i am missing here ?
Ok, the issue is the cookie jar implementation, more specifically the SetCookies function, which right now is:
func (jar *Jar) SetCookies(u *url.URL, cookies []*http.Cookie) {
jar.Lock()
jar.cookies[u.Host] = cookies
jar.Unlock()
}
And this is wrong because new cookies instead of being added to the existing ones they will simply be added as new discarding the old ones.
It seems the right way to do this is:
func (jar *Jar) SetCookies(u *url.URL, cookies []*http.Cookie) {
jar.Lock()
if _, ok := jar.cookies[u.Host]; ok {
for _, c := range cookies {
jar.cookies[u.Host] = append(jar.cookies[u.Host], c)
}
} else {
jar.cookies[u.Host] = cookies
}
jar.Unlock()
}