Hello StackOverflow AWS Gophers,
I'm implementing a CLI with the excellent cobra/viper packages from spf13. We have an Athena database fronted by an API Gateway endpoint, which authenticates with IAM.
That is, in order to interact with its endpoints by using Postman, I have to define AWS Signature as Authorization method, define the corresponding AWS id/secret and then in the Headers there will be X-Amz-Security-Token and others. Nothing unusual, works as expected.
Since I'm new to Go, I was a bit shocked to see that there are no examples to do this simple HTTP GET request with the aws-sdk-go itself... I'm trying to use the shared credentials provider (~/.aws/credentials), as demonstrated for the S3 client Go code snippets from re:Invent 2015:
req := request.New(nil)
How can I accomplish this seemingly easy feat in 2019 without having to resort to self-cooked net/http and therefore having to manually read ~/.aws/credentials or worse, go with os.Getenv and other ugly hacks?
Any Go code samples interacting as client would be super helpful. No Golang Lambda/server examples, please, there's plenty of those out there.
Unfortunately, it seems that the library has been updated since the accepted answer was written and the solution no longer is the same. After some trial and error, this appears to be the more current method of handling the signing (using https://pkg.go.dev/github.com/aws/aws-sdk-go-v2):
import (
"context"
"net/http"
"time"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/aws/signer/v4"
)
func main() {
// Context is not being used in this example.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
// Handle error.
}
credentials, err := cfg.Credentials.Retrieve(context.TODO())
if err != nil {
// Handle error.
}
// The signer requires a payload hash. This hash is for an empty payload.
hash := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
req, _ := http.NewRequest(http.MethodGet, "api-gw-url", nil)
signer := v4.NewSigner()
err = signer.SignHTTP(context.TODO(), credentials, req, hash, "execute-api", cfg.Region, time.Now())
if err != nil {
// Handle error.
}
// Use `req`
}
The solution below uses aws-sdk-go-v2
https://github.com/aws/aws-sdk-go-v2
// A AWS SDK session is created because the HTTP API is secured using a
// IAM authorizer. As such, we need AWS client credentials and a
// session to properly sign the request.
cfg, err := external.LoadDefaultAWSConfig(
external.WithSharedConfigProfile(profile),
)
if err != nil {
fmt.Println("unable to create an AWS session for the provided profile")
return
}
req, _ := http.NewRequest(http.MethodGet, "", nil)
req = req.WithContext(ctx)
signer := v4.NewSigner(cfg.Credentials)
_, err = signer.Sign(req, nil, "execute-api", cfg.Region, time.Now())
if err != nil {
fmt.Printf("failed to sign request: (%v)\n", err)
return
}
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Printf("failed to call remote service: (%v)\n", err)
return
}
defer res.Body.Close()
if res.StatusCode != 200 {
fmt.Printf("service returned a status not 200: (%d)\n", res.StatusCode)
return
}
The first argument to request.New is aws.Config, where you can send credentials.
https://github.com/aws/aws-sdk-go/blob/master/aws/request/request.go#L99
https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config
There are multiple ways to create credentials object: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
For example using static values:
creds:= credentials.NewStaticCredentials("AKID", "SECRET_KEY", "TOKEN")
req := request.New(aws.Config{Credentials: creds}, ...)
I'm pretty new to go myself (3rd day learning go) but from watching the video you posted with the S3 example and reading through the source code (for the s3 service and request module) here is my understanding (which I'm hoping helps).
If you look at the code for the s3.New() function aws-sdk-go/service/s3/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *S3 {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, .SigningName) }
As opposed to request.New() function aws-sdk-go/aws/request/request.go
func New(cfg aws.Config, clientInfo metadata.ClientInfo, handlers Handlers,
retryer Retryer, operation *Operation, params interface{}, data interface{}) *Request { ...
As you can see in the s3 scenario the *aws.Config struct is a pointer, and so is probably initialized / populated elsewhere. As opposed to the request function where the aws.Config is a parameter. So I am guessing the request module is probably a very low level module which doesn't get the shared credentials automatically.
Now, seeing as you will be interacting with API gateway I had a look at that service specifically to see if there was something similar. I looked at aws-sdk-go/service/apigateway/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *APIGateway {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, c.SigningName) }...
Which looks pretty much the same as the s3 client, so perhaps try using that and see how you go?
Related
This is the code written in the handler, which gets the token required to call the data service.
m2m, err := h.getM2MToken(ctx)
if err != nil {
return lc.SetResponse(&events.APIGatewayV2HTTPResponse{
StatusCode: http.StatusInternalServerError,
Body: "Internal Server Error (m2m)",
})
}
//Get the bearer token
userToken, err := h.getBearer(req.Headers)
if err != nil {
xray.AddError(ctx, err)
return lc.SetResponse(&events.APIGatewayV2HTTPResponse{
StatusCode: http.StatusInternalServerError,
Body: "Internal Server Error (bearer)",
})
}
My suggestion is to first try abstracting the inputs that you sent to a method
Like instead of this
userToken, err := h.getBearer(req.Headers)
You can pass specify interfaces like
type userTokenInput struct {}
uti := userTokenInput{}
userToken, err := h.getBearer(uti)
The above helps you to have control over input which makes testing easier
For network calls try using some mock HTTP client which can return expected
data you can follow this for mock HTTP client https://www.thegreatcodeadventure.com/mocking-http-requests-in-golang/
If the service does not work without a token, you will have to provide one.
If the calls you will be doing should not be seen on the real target system for whatever reason, you will need a different target system for testing.
Ask the provider if they have a test installation you can use.
Consider testing against a mock.
I am trying to port a server running a small golang app to AWS Lambda. I am not very familiar with golang and to deploy to a server I have just followed the instructions in the repo.
It runs a server with net/http, the main.go is as follows:
func main() {
r := new(route.Router)
r.HandleFunc("/squares", squares.Random)
// ... more Handlers
log.Println("Listening on " + os.Getenv("PORT"))
err := http.ListenAndServe(":"+os.Getenv("PORT"), r)
if err != nil {
log.Fatal("ListenAndServe:", err)
}
}
Now I have found this drop-in replacement repo on Github for ListenAndServe, apex/gateway, but I think I am missing a fundamental step in making it work.
What I've done is download and import of the library
import (
...
"github.com/apex/gateway/v2"
)
then simply replace the function in main, zip and upload to aws lambda
func main() {
r := new(route.Router)
r.HandleFunc("/squares", squares.Random)
// ... more Handlers
// log.Println("Listening on 8080")
err := gateway.ListenAndServe(":8080", r)
if err != nil {
log.Fatal("ListenAndServe:", err)
}
}
then I set up an http API Gateway and link to the lambda function.
It doesn't work. I think I'm missing something but I can't figure out what. From the example on the apex/gateway repo, I don't see what I'm missing.
The app is Tinygraphs fwiw.
Thank you
Edit:
As per Adrians comment, when I go to the api link I get
{"message":"Not Found"}
Hey User I believe you are not understanding how API Gateway and Lambda works properly
You do not need to set up a route listening on 8080 that is effectively what API Gateway is doing and then forwarding the request to your code running on the lambda. Which is normally in this format:
package main
import (
"fmt"
"context"
"github.com/aws/aws-lambda-go/lambda"
)
type MyEvent struct {
Name string `json:"name"`
}
func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
return fmt.Sprintf("Hello %s!", name.Name ), nil
}
func main() {
lambda.Start(HandleRequest)
}
Was a pretty stupid mistake.
I had defined the api path as / instead of /{proxy+}, so it wasn't accepting a non-root url whereas the invoke url is like this:
https://XXXXXXXX.execute-api.us-east-1.amazonaws.com/squares/tinygraphs?theme=frogideas&numcolors=4&size=220&fmt=svg
Thank you everyone who looked at this.
(regarding the port, according to this, the port specified is just ignored by gateway.ListenAndServe)
RingCentral has the ability to upload custom Hold Music via the UI. Can this be done via the API, as user and as an admin for other users? Searching the API Reference for hold music didn't turn up an API.
Here's some info on this functionality:
User: https://support.ringcentral.com/s/article/1798
Admin: https://support.ringcentral.com/s/article/8360
Here's what the UI looks like:
Try ringcentral API for creating custom user greeting and set greeting type as HoldMusic
https://developers.ringcentral.com/api-reference/Rule-Management/createCustomUserGreeting
Create custom user greeting API
There are two steps to this:
Get the Answering Rule ID of the rule you want to change. This can be a standard rule such as business-hours-rule, or it can be a custom rule that you have created. You can call the Get Call Handling Rules API to get a list of current standard and custom rules.
Call the Update Greeting API with type set to HoldMusic and answeringRuleId set to the id you wish to update, e.g. business-hours-rule
There are actually several ways to call the Update Greeting API:
multipart/form-data with individual string parts. In this approach, separate parts are sent with names type, answeringRuleId, and binary
multipart/form-data with JSON metadata part. In this approach, a JSON MIME part named json is sent with a payload like: {"type": "HoldMusic", "answeringRule": { "id": "12345678" }}
multipart/mixed
I tend to prefer the first approach (multipart/form-data with individual string parts) because it's easy to use with tools like cURL and many HTTP clients.
Here's an example using Go:
package main
import(
"log"
"net/http"
"net/url"
"os"
"github.com/grokify/gotilla/mime/multipartutil"
"github.com/grokify/oauth2more/ringcentral"
)
func main() {
// Get the Client (*http.Client):
client, err = ringcentral.NewClientPassword(
ringcentral.ApplicationCredentials{
ClientID: os.Getenv("RINGCENTRAL_CLIENT_ID"),
ClientSecret: os.Getenv("RINGCENTRAL_CLIENT_SECRET"),
ServerURL: os.Getenv("RINGCENTRAL_SERVER_URL")},
ringcentral.PasswordCredentials{
Username: os.Getenv("RINGCENTRAL_USERNAME"),
Extension: os.Getenv("RINGCENTRAL_EXTENSION"),
Password: os.Getenv("RINGCENTRAL_PASSWORD")})
if err!=nil {
log.Fatal(err)
}
// Create the HTTP Request (*http.Request)
params := url.Values{}
params.Set("type", "HoldMusic")
params.Set("answeringRuleId", "business-hours-rule")
req, err := multipartutil.NewRequest(
http.MethodPost,
"https://platform.ringcenral.com/restapi/v1.0/account/~/extension/~/greeting",
params,
[]multipartutil.FileInfo{
{
MIMEPartName: "binary",
Filepath: "mygreeting.wav",
},
},
)
// Send the request
resp, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
fmt.Printf("STATUS: %v\n", resp.StatusCode)
}
I'm trying to invoke a mutation from lambda (specifically using golang). I used AWS_IAM as the authentication method of my AppSync API. I also give appsync:GraphQL permission to my lambda.
However, after looking at the docs here: https://docs.aws.amazon.com/sdk-for-go/api/service/appsync/
I can't find any documentation on how to invoke the appsync from the library. Can anyone point me to the right direction here?
P.S. I don't want to query or subscribe or anything else from lambda. It's just a mutation
Thanks!
------UPDATE-------
Thanks to #thomasmichaelwallace for informing me to use https://godoc.org/github.com/machinebox/graphql
Now the problem is how can I sign the request from that package using aws v4?
I found a way of using plain http.Request and AWS v4 signing. (Thanks to #thomasmichaelwallace for pointing this method out)
client := new(http.Client)
// construct the query
query := AppSyncPublish{
Query: `
mutation ($userid: ID!) {
publishMessage(
userid: $userid
){
userid
}
}
`,
Variables: PublishInput{
UserID: "wow",
},
}
b, err := json.Marshal(&query)
if err != nil {
fmt.Println(err)
}
// construct the request object
req, err := http.NewRequest("POST", os.Getenv("APPSYNC_URL"), bytes.NewReader(b))
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", "application/json")
// get aws credential
config := aws.Config{
Region: aws.String(os.Getenv("AWS_REGION")),
}
sess := session.Must(session.NewSession(&config))
//sign the request
signer := v4.NewSigner(sess.Config.Credentials)
signer.Sign(req, bytes.NewReader(b), "appsync", "ap-southeast-1", time.Now())
//FIRE!!
response, _ := client.Do(req)
//print the response
buf := new(bytes.Buffer)
buf.ReadFrom(response.Body)
newStr := buf.String()
fmt.Printf(newStr)
The problem is that that API/library is designed to help you create/update app-sync instances.
If you want to actually invoke them then you need to POST to the GraphQL endpoint.
The easiest way for testing is to sign-in to the AWS AppSync Console, press the 'Queries' button in the sidebar and then enter and run your mutation.
I'm not great with go, but from what I can see there are client libraries for GraphQL in golang (e.g. https://godoc.org/github.com/machinebox/graphql).
If you are using IAM then you'll need to sign your request with a v4 signature (see this article for details: https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
I have 5 end points which have methods such as GET, POST, and DELETE to test. I wrote test cases using the go's in built testing package. I'm worried that I'm missing some cases which are not striking to my mind.I have posted in code review for my test case to be reviewed but I didn't get any response. I have also followed this post Testing HTTP routes in golang. All these test cases are checking for the response codes.
The problem is that, most of my test cases follow similar pattern where I post data in different formats and checking the response codes. I strongly feel like I'm missing something that will break my API when I push it to prod. I need some insight on testing these routes so that I can be confident to push the api to prod.
main_test.go
func TestSigHandler(t *testing.T){
test_cases := []string{"2021205"}
// GET Testing
for _, method := range test_cases{
usersUrl = fmt.Sprintf("%s/1/sig/id/%s", server.URL, method) //Grab the address for the API endpoint
request, err := http.NewRequest("GET", usersUrl, nil)
res, err := http.DefaultClient.Do(request)
if err != nil {
t.Error(err) //Something is wrong while sending request
}
if res.StatusCode != 200 {
t.Errorf("Something went wrong : ", res.StatusCode) //Uh-oh this means our test failed
}
}
// POST Testing
sig := []byte( `{
"raw": "a new sig"
}`)
usersUrl = fmt.Sprintf("%s/1/sig/id/2021205", server.URL) //Grab the address for the API endpoint
request, err := http.NewRequest("POST", usersUrl, bytes.NewBuffer(sig))
if err != nil{
t.Error(err)
}
request.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(request)
if err != nil {
t.Error(err) //Something is wrong while sending request
}
if res.StatusCode != 200 {
t.Errorf(" Something Went Wrong: ", res.StatusCode) //Uh-oh this means our test failed
}
// DELETE Testing
sigs_delete_cases := []string{ "1000345"}
for _, sig_to_be_deleted := range sigs_delete_cases{
usersUrl = fmt.Sprintf("%s/1/sig/id/%s", server.URL, sig_to_be_deleted) //Grab the address for the API endpoint
request, err := http.NewRequest("DELETE", usersUrl, nil)
res, err := http.DefaultClient.Do(request)
if err != nil {
t.Error(err) //Something is wrong while sending request
}
if res.StatusCode != 200 {
t.Errorf("Tried to delete a reserved Id : ", res.StatusCode) //Uh-oh this means our test failed
}
}
}
I like to do this way:
Establish Continuous Integration. If your project is Open Source, you may use services like Travis CI - it has very easy installation. This helps you to see how changes affect code.
Set code test coverage. It allows you to see what source code lines are covered with tests and what are not and where very possible bugs will emerge. Of course, code coverage tool is not a panacea. And if line was checked it doesn't mean it is absolutely ok, and it will not fail with other input. But it helps much to maintain good code and look for bugs. For open source you may use coveralls.io. There's a special goveralls plugin for it.
To help the problem above you may use so-called Fuzzy testing - exploratory tests with random input to find a root cause. There're standard https://golang.org/pkg/testing/quick/ and non-standard packages https://github.com/dvyukov/go-fuzz.
Then I experiment with tests, they are both positive and negative. I try check situation with errors, timeouts, incorrect replies.
For my tests I've used as usual client http so httptest package.