I have something that looks like so:
func (client *MyCustomClient) CheckURL(url string, json_response *MyCustomResponseStruct) bool {
r, err = http.Get(url)
if err != nil {
return false
}
defer r.Body.Close()
.... do stuff with json_response
And in my test, I have the following:
func TestCheckURL(t *test.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=UTF-8")
fmt.Fprintln(w, `{"status": "success"}`)
}))
defer ts.Close()
json_response := new(MyCustomResponseStruct)
client := NewMyCustomClient() // returns instance of MyCustomClient
done := client.CheckURL("test.com", json_response)
However, it does not appear as if the HTTP test server is working and that it actually goes out to test.com, as evidenced by the log output:
Get http:/test.com: dial tcp X.Y.Z.A: i/o timeout
My question is how to properly use the httptest package to mock out this request... I read through the docs and this helpful SO Answer but I'm still stuck.
Your client only calls the URL you provided as the first argument to the CheckURL method. Give your client the URL of your test server:
done := client.CheckURL(ts.URL, json_response)
Related
I've an API written using Gin that uses GORM for ORM. The API works perfectly fine when using a real DB and accessing the API URL from the web browser. But I can't get a mocked unit test to pass:
func TestRespForGetUsersHandlerWithSomeUsers(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal("can't create mock db")
}
defer db.Close()
sqlmock.NewRows(
[]string{"id", "name", "username"},
).
AddRow(1, "Abhishek Kumar", "abhishek")
w := httptest.NewRecord()
c, _ := gin.CreateTestContext(w)
postgresDB := postgres.New(postgres.Config{Conn: db})
gormDB, err := gorm.Open(postgresDB, &gorm.Config{})
if err != nil {
t.Fatal("can't create gormDB")
}
api.GetUsersWrapper(gormDB)(c)
if w.Code != http.StatusOK {
t.Errorf("Expected status code to be %d but got %d", http.StatusOK, w.Code)
}
var got []models.User
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
t.Fatalf("Can't unmarshal response body: %s", err)
}
if len(got) != 1 {
t.Errorf("Expected response to be 1 item but got %d items", len(got))
}
}
The last if-statement gets triggered. The length of got is actually 0. However, if I try calling the API endpoint associated with GetUsersWrapper from a browser (while the server is using a real DB), everything works as expected.
I suspect that either sqlmock.NewRows is not creating the rows such that it'll be visible to gormDB or I'm not testing the response from GetUsersWrapper properly. How can I unit test a gin API correctly?
If you're willing to try an alternate approach, you can unit test your gin APIs using keploy. It's open-source and also supports GORM.
https://github.com/keploy/keploy
This is the code written in the handler, which gets the token required to call the data service.
m2m, err := h.getM2MToken(ctx)
if err != nil {
return lc.SetResponse(&events.APIGatewayV2HTTPResponse{
StatusCode: http.StatusInternalServerError,
Body: "Internal Server Error (m2m)",
})
}
//Get the bearer token
userToken, err := h.getBearer(req.Headers)
if err != nil {
xray.AddError(ctx, err)
return lc.SetResponse(&events.APIGatewayV2HTTPResponse{
StatusCode: http.StatusInternalServerError,
Body: "Internal Server Error (bearer)",
})
}
My suggestion is to first try abstracting the inputs that you sent to a method
Like instead of this
userToken, err := h.getBearer(req.Headers)
You can pass specify interfaces like
type userTokenInput struct {}
uti := userTokenInput{}
userToken, err := h.getBearer(uti)
The above helps you to have control over input which makes testing easier
For network calls try using some mock HTTP client which can return expected
data you can follow this for mock HTTP client https://www.thegreatcodeadventure.com/mocking-http-requests-in-golang/
If the service does not work without a token, you will have to provide one.
If the calls you will be doing should not be seen on the real target system for whatever reason, you will need a different target system for testing.
Ask the provider if they have a test installation you can use.
Consider testing against a mock.
I had a simple function which connects to mongoDB and create a new document.
Now how do I mock the methods of the imported mongo package while unit testing.
Ive tried to mock GinContext by monkeypatching.
But unable to proceed with mocking the actual mongoClient as the package is imported.
func CreateUser(c GinContext) {
var userdetail UserDetails
binderr := c.ShouldBindJSON(&userdetail)
fmt.Println(binderr)
if binderr != nil {
c.JSON(500, gin.H{
"message": "Input payload not matching",
"error": binderr,
})
return
}
//-- Client if of type *mongo.Client.
//-- How do I mock the Client.Database, Client.Database.Connection
collection := Client.Database("demo").Collection("users")
ctx, err1 := context.WithTimeout(context.Background(), 10*time.Second)
if err1 != nil {
}
response, err2 := collection.InsertOne(ctx, userdetail)
if err2 != nil {
log.Println("Some error inserting the document")
}
fmt.Println(response.InsertedID)
c.JSON(200, gin.H{
"message": "User created successfully",
})
}
Expected: I should be able to mock or stub Client and provide dummy functionality. Just like in nodeJS we do
spyOn(Client,'Database').and.return(Something)
Every time I'm wondering "how to mock a method", this is mostly related to my code architecture. Not being able to test easily some code means, most of time, that the code is poorly designed and/or too coupled to the used libraries/frameworks. Here, you want to mock Mongo connection only because your code is too tightly related to Mongo (in the CreateUser function). Refactoring could help you to test your code (without any Mongo connection).
I've experienced that using interfaces and dependency injection simplifies
the testing process in Go, and clarifies the architecture. Here is my attempt to help you test your application.
Code refactoring
First, define what you want to do with an interface. Here, you're inserting users, so let's do a UserInserter interface, with a single method for now (Insert, to insert a single user) :
type UserInserter interface {
Insert(ctx context.Context, userDetails UserDetails) (insertedID interface{}, err error)
}
In the code you have provided, you are only using the insertedID, so you probably only need it as output of this Insert method (and an optional error if something gone wrong). insertedID is defined as an interface{} here, but feel free to change to whatever you want.
Then, let's modify your CreateUser method and inject this UserInserter as a parameter :
func CreateUser(c *gin.Context, userInserter UserInserter) {
var userdetail UserDetails
binderr := c.ShouldBindJSON(&userdetail)
fmt.Println(binderr)
if binderr != nil {
c.JSON(500, gin.H{
"message": "Input payload not matching",
"error": binderr,
})
return
}
// this is the modified part
insertedID, err2 := userInserter.Insert(c, userdetail)
if err2 != nil {
log.Println("Some error inserting the document")
}
fmt.Println(insertedID)
c.JSON(200, gin.H{
"message": fmt.Sprintf("User %s created successfully", insertedID),
})
}
This method could be refactored but, to avoid any confusion, I will not touch it.
userInserter.Insert(c, userdetail) replaces here the Mongo dependency in this method by injecting userInserter.
You can now implement your UserInserter interface with the backend of your choice (Mongo in your case). Insertion into Mongo needs a Collection object (the collection we are inserting the user in), so let's add this as an attribute :
type MongoUserInserter struct {
collection *mongo.Collection
}
Implementation of Insert method follows (call InsertOne method on *mongo.Collection) :
func (i MongoUserInserter) Insert(ctx context.Context, userDetails UserDetails) (insertedID interface{}, err error) {
response, err := i.collection.InsertOne(ctx, userDetails)
return response.InsertedID, err
}
This implementation could be in a separated package and should be tested separately.
Once implemented, you can use MongoUserInserter in your main application, where Mongo is the backend. MongoUserInserter is initialized in the main function, and injected in the CreateUser method. Router setup have been separated (also for testing purpose) :
func setupRouter(userInserter UserInserter) *gin.Engine {
router := gin.Default()
router.POST("/createUser", func(c *gin.Context) {
CreateUser(c, userInserter)
})
return router
}
func main() {
client, _ := mongo.NewClient()
collection := client.Database("demo").Collection("users")
userInserter := MongoUserInserter{collection: collection}
router := setupRouter(userInserter)
router.Run(":8080")
}
Note that if some day you want to change the backend, you will only
need to change the userInserter in the main function!
Tests
From a tests perspective, it is now easier to test because we can create a fake UserInserter, like :
type FakeUserInserter struct{}
func (_ FakeUserInserter) Insert(ctx context.Context, userDetails UserDetails) (insertedID interface{}, err error) {
return userDetails.Name, nil
}
(I supposed here UserDetails have an attribute Name).
If you really want to mock this interface, you can take a look at GoMock. In this case though, I'm not sure using a mock framework is required.
And now we can test our CreateUser method with a simple HTTP testing framework (see https://github.com/gin-gonic/gin#testing), without needing a Mongo connection or mocking it.
import (
"bytes"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCreateUser(t *testing.T) {
userInserter := FakeUserInserter{}
router := setupRouter(userInserter)
w := httptest.NewRecorder()
body := []byte(`{"name": "toto"}`)
req, _ := http.NewRequest("POST", "/createUser", bytes.NewBuffer(body))
router.ServeHTTP(w, req)
assert.Equal(t, 200, w.Code)
assert.Equal(t, `{"message":"User toto created successfully"}`, w.Body.String())
}
Note that this does not exempt to also test Insert method of MongoUserInserter, but separately : here, this test covers CreateUser, not the Insert method.
Hello StackOverflow AWS Gophers,
I'm implementing a CLI with the excellent cobra/viper packages from spf13. We have an Athena database fronted by an API Gateway endpoint, which authenticates with IAM.
That is, in order to interact with its endpoints by using Postman, I have to define AWS Signature as Authorization method, define the corresponding AWS id/secret and then in the Headers there will be X-Amz-Security-Token and others. Nothing unusual, works as expected.
Since I'm new to Go, I was a bit shocked to see that there are no examples to do this simple HTTP GET request with the aws-sdk-go itself... I'm trying to use the shared credentials provider (~/.aws/credentials), as demonstrated for the S3 client Go code snippets from re:Invent 2015:
req := request.New(nil)
How can I accomplish this seemingly easy feat in 2019 without having to resort to self-cooked net/http and therefore having to manually read ~/.aws/credentials or worse, go with os.Getenv and other ugly hacks?
Any Go code samples interacting as client would be super helpful. No Golang Lambda/server examples, please, there's plenty of those out there.
Unfortunately, it seems that the library has been updated since the accepted answer was written and the solution no longer is the same. After some trial and error, this appears to be the more current method of handling the signing (using https://pkg.go.dev/github.com/aws/aws-sdk-go-v2):
import (
"context"
"net/http"
"time"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/aws/signer/v4"
)
func main() {
// Context is not being used in this example.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
// Handle error.
}
credentials, err := cfg.Credentials.Retrieve(context.TODO())
if err != nil {
// Handle error.
}
// The signer requires a payload hash. This hash is for an empty payload.
hash := "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
req, _ := http.NewRequest(http.MethodGet, "api-gw-url", nil)
signer := v4.NewSigner()
err = signer.SignHTTP(context.TODO(), credentials, req, hash, "execute-api", cfg.Region, time.Now())
if err != nil {
// Handle error.
}
// Use `req`
}
The solution below uses aws-sdk-go-v2
https://github.com/aws/aws-sdk-go-v2
// A AWS SDK session is created because the HTTP API is secured using a
// IAM authorizer. As such, we need AWS client credentials and a
// session to properly sign the request.
cfg, err := external.LoadDefaultAWSConfig(
external.WithSharedConfigProfile(profile),
)
if err != nil {
fmt.Println("unable to create an AWS session for the provided profile")
return
}
req, _ := http.NewRequest(http.MethodGet, "", nil)
req = req.WithContext(ctx)
signer := v4.NewSigner(cfg.Credentials)
_, err = signer.Sign(req, nil, "execute-api", cfg.Region, time.Now())
if err != nil {
fmt.Printf("failed to sign request: (%v)\n", err)
return
}
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Printf("failed to call remote service: (%v)\n", err)
return
}
defer res.Body.Close()
if res.StatusCode != 200 {
fmt.Printf("service returned a status not 200: (%d)\n", res.StatusCode)
return
}
The first argument to request.New is aws.Config, where you can send credentials.
https://github.com/aws/aws-sdk-go/blob/master/aws/request/request.go#L99
https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config
There are multiple ways to create credentials object: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
For example using static values:
creds:= credentials.NewStaticCredentials("AKID", "SECRET_KEY", "TOKEN")
req := request.New(aws.Config{Credentials: creds}, ...)
I'm pretty new to go myself (3rd day learning go) but from watching the video you posted with the S3 example and reading through the source code (for the s3 service and request module) here is my understanding (which I'm hoping helps).
If you look at the code for the s3.New() function aws-sdk-go/service/s3/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *S3 {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, .SigningName) }
As opposed to request.New() function aws-sdk-go/aws/request/request.go
func New(cfg aws.Config, clientInfo metadata.ClientInfo, handlers Handlers,
retryer Retryer, operation *Operation, params interface{}, data interface{}) *Request { ...
As you can see in the s3 scenario the *aws.Config struct is a pointer, and so is probably initialized / populated elsewhere. As opposed to the request function where the aws.Config is a parameter. So I am guessing the request module is probably a very low level module which doesn't get the shared credentials automatically.
Now, seeing as you will be interacting with API gateway I had a look at that service specifically to see if there was something similar. I looked at aws-sdk-go/service/apigateway/service.go
func New(p client.ConfigProvider, cfgs ...*aws.Config) *APIGateway {
c := p.ClientConfig(EndpointsID, cfgs...)
return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, c.SigningName) }...
Which looks pretty much the same as the s3 client, so perhaps try using that and see how you go?
I'm trying to use Go to log into my account on Amazon to automatically pull some information, but I'm having trouble logging in because it complains about cookies. Here's a sanitized version of the code I was using:
package main
import (
"bytes"
"io/ioutil"
"net/http"
"net/http/cookiejar"
"net/url"
"strconv"
)
func CheckThis(AmazonUsername string, AmazonPassword string) error {
var LogonURL string
// Set the url
LogonURL = "https://www.amazon.com/ap/signin"
// Craft some form data
form := url.Values{}
form.Add("appAction", "SIGNIN")
form.Add("email", AmazonUsername)
form.Add("password", AmazonPassword)
form.Add("create", "0")
form.Add("appActionToken", “$VALUE”)
form.Add("openid.pape.max_auth_age", "$VALUE==")
form.Add("openid.identity", "$VALUE=")
form.Add("openid.assoc_handle", "$VALUE")
form.Add("openid.mode", "$VALUE")
form.Add("openid.ns.pape", "$VALUE==")
form.Add("openid.claimed_id", "$VALUE=")
form.Add("pageId", "$VALUE")
form.Add("openid.ns", "$VALUE=")
// Amazon sells cookies
cookieJar, _ := cookiejar.New(nil)
// Create a new client with the cookiejar in the struct...
client := &http.Client{
Jar: cookieJar,
}
// Craft the request to send to the website with the form containing login info
req, _ := http.NewRequest("POST", LogonURL, bytes.NewBufferString(form.Encode()))
// Some more headers
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("Content-Length", strconv.Itoa(len(form.Encode())))
req.Header.Add("Accept-Language", "en-US,en;q=0.8")
req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
req.Header.Add("Connection", "keep-alive")
req.Header.Add("Host", "www.amazon.com")
req.Header.Add("Referer", "https://www.amazon.com/ap/signin")
req.Header.Add("Upgrade-Insecure-Requests", "1")
req.Header.Add("Origin", "https://www.amazon.com")
req.Header.Add("Cache-Control", "max-age=0")
// And we're off to the races...
resp, _ := client.Do(req)
// What was in the response?
charResponse, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
// Write response body to a text file with title of…
_ = WriteOutputToFile(string(charResponse), “response.html")
// All done!
return nil
}
The $VALUE entries are because I'm not sure if the strings are significant to my account so I removed them; these are values I pulled from the developer tools of a Chrome login session. I also removed err checks for brevity.
The reply page (opening response.html on my drive within Chrome) looks like this:
What am I missing in order to keep the cookie with the client req/resp for the sign-in and later pages?
Or am I missing something where the response page I save is trying to pull elements from Amazon when I render the HTML, and the cookie issue is because the browser would be missing cookie information when I'm trying to view the results from the Go application?
I'm pretty sure Amazon uses different data every time when you try to login so better to parse login form. Here is example
package main
import (
"bytes"
"io/ioutil"
"net/http"
"net/http/cookiejar"
"net/url"
"github.com/PuerkitoBio/goquery"
"log"
)
func checkError(err error) {
if err != nil {
log.Fatal(err)
}
}
func CheckThis(AmazonUsername string, AmazonPassword string) error {
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
res, err := client.Get("https://www.amazon.com/gp/sign-in.html/ref=ord_cart_unrec_signin")
checkError(err)
doc, err := goquery.NewDocumentFromResponse(res)
form := url.Values{}
doc.Find("form[name='signIn'] input").Each(func(i int, s *goquery.Selection) {
name, exist := s.Attr("name")
if exist {
value, exist := s.Attr("value")
if exist {
form.Add(name, value)
}
}
})
form.Set("email", AmazonUsername)
form.Set("password", AmazonPassword)
req, _ := http.NewRequest("POST", "https://www.amazon.com/ap/signin", bytes.NewBufferString(form.Encode()))
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
res, err = client.Do(req)
checkError(err)
defer res.Body.Close()
charResponse, _ := ioutil.ReadAll(res.Body)
ioutil.WriteFile("response.html", charResponse, 0777)
return nil
}
func main() {
CheckThis("", "")
}
Amazon's index is setting this (and much more) cookies on my browser:
I suppose you need to set them into cookiejar in order to simulate a browser (amazon excepts that from your request, so is telling you have no cookies enabled because there isn't a required security value).
I agree with #JimB, you should be using Amazon SDK. Is there any reason why you are not doing this way?
Best regards.