Google Cloud Platform presignURL using Go - google-cloud-platform

Trying to upload a picture to Google Cloud Platform, I always get the same err "<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>GOOG4-RSA-SHA256 20.................951Z".
I did add a service-account to the bucket with the role Storage Admin and Storage Object Admin as you can see on the pic
I have generated a Key(for the service account) and downloaded it as .json file, then I generate a presignURL using this code:
// key is the downloaded .json key file from the GCP service-account
// the return string is the presignedURL
func getPresignedURL(path, key string) (string, error) {
sakeyFile := filepath.Join(path, key)
saKey, err := ioutil.ReadFile(sakeyFile)
if err != nil {
log.Fatalln(err)
}
cfg, err := google.JWTConfigFromJSON(saKey)
if err != nil {
log.Fatalln(err)
}
bucket := "mybucket"
ctx := context.Background()
client, err := storage.NewClient(ctx)
if err != nil {
return "", fmt.Errorf("storage.NewClient: %v", err)
}
defer client.Close()
opts := &storage.SignedURLOptions{
Scheme: storage.SigningSchemeV4,
Method: "PUT",
Headers: []string{
"Content-Type:image/jpeg",
},
Expires: time.Now().Add(15 * time.Minute),
GoogleAccessID: cfg.Email,
PrivateKey: cfg.PrivateKey,
}
u, err := client.Bucket(bucket).SignedURL("mypic.jpeg", opts)
if err != nil {
return "", fmt.Errorf("Bucket(%q).SignedURL: %v", bucket, err)
}
return u, nil
}
The presignedURL looks good, something like this:
https://storage.googleapis.com/djedjepicbucket/mypic.jpeg?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=djedje%40picstorage-363707.iam.gserviceaccount.com%2F20220926%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220926T081951Z&X-Goog-Expires=899&X-Goog Signature=3f330715d7a38ea08f99134a16f464fb............5ad800a7665dfb1440034ab1f5ab045252336&X-Goog-SignedHeaders=content-type%3Bhost
Then I read a file(picture) from disk and upload it using the presignURL
// the uri is the presignedURL
func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
if err != nil {
return nil, err
}
_, err = io.Copy(body, file)
for key, val := range params {
_ = writer.WriteField(key, val)
}
err = writer.Close()
if err != nil {
return nil, err
}
req, err := http.NewRequest("PUT", uri, body)
req.Header.Set("Content-Type", "image/jpeg")
return req, err
}
Then I exec the request
// the previous func
request, err := newfileUploadRequest(purl, extraParams, "picture", filepath.Join(path, "download.jpeg"))
if err != nil {
log.Fatal(err)
}
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
log.Fatal(err)
} else {
body := &bytes.Buffer{}
_, err := body.ReadFrom(resp.Body)
if err != nil {
log.Fatal(err)
}
resp.Body.Close()
fmt.Println(resp.StatusCode)
fmt.Println(resp.Header)
fmt.Println(body)
}
Unfortunatly, I always get the same error back
403
map[Alt-Svc:[h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Content-Length:[884] Content-Type:[application/xml; charset=UTF-8] Date:[Mon, 26 Sep 2022 08:22:19 GMT] Server:[UploadServer] X-Guploader-Uploadid:[ADPyc......................ECL_4W]]
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>GOOG4-RSA-SHA256
20220926T081951Z
20220926/auto/storage/goog4_request
c5f36838af4......................8ffb56329c1eb27f</StringToSign><CanonicalRequest>PUT
/djedjepicbucket/mypic.jpeg
X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=djedje%40picstorage-363707.iam.gserviceaccount.com%2F20220926%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220926T081951Z&X-Goog-Expires=899&X-Goog-SignedHeaders=content-type%3Bhost
content-type:multipart/form-data; boundary=5be13cc........................dd6aef6823
host:storage.googleapis.com
content-type;host
UNSIGNED-PAYLOAD</CanonicalRequest></Error>
Actually I have tryied many other ways as well but I basically always get this(more or less) same err back, Does someone have an Idea what am I forgetting(I am on that for 2 days now...) ? Thank you
=============================
I edited the question, that code works perfectly

Found the answer, in the both getPresignedURL() and newfileUploadRequest() func, the Header must be set to "Content-Type:application/octet-stream"(or "Content-Type:image/jpeg" for instance if the picture need to be display using its URL), then the pic is uploaded without issue.

Related

How to create a multipart.File in Golang [duplicate]

How do I set the Request.FormFile when trying to test an endpoint?
Partial code:
func (a *EP) Endpoint(w http.ResponseWriter, r *http.Request) {
...
x, err := strconv.Atoi(r.FormValue("x"))
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
f, fh, err := r.FormFile("y")
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
...
}
How do I use the httptest lib to generate a post request that has value that I can get in FormFile?
You don't need to mock the complete FormFile struct as suggested by the other answer. The mime/multipart package implements a Writer type that lets you create a FormFile. From the docs
CreateFormFile is a convenience wrapper around CreatePart. It creates
a new form-data header with the provided field name and file name.
func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error)
Then, you can pass this io.Writer to httptest.NewRequest, which accepts a reader as an argument.
request := httptest.NewRequest("POST", "/", myReader)
To do this, you can either write the FormFile to an io.ReaderWriter buffer or use an io.Pipe. Here is a complete example that makes use of pipes:
func TestUploadImage(t *testing.T) {
// Set up a pipe to avoid buffering
pr, pw := io.Pipe()
// This writer is going to transform
// what we pass to it to multipart form data
// and write it to our io.Pipe
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
// We create the form data field 'fileupload'
// which returns another writer to write the actual file
part, err := writer.CreateFormFile("fileupload", "someimg.png")
if err != nil {
t.Error(err)
}
// https://yourbasic.org/golang/create-image/
img := createImage()
// Encode() takes an io.Writer.
// We pass the multipart field
// 'fileupload' that we defined
// earlier which, in turn, writes
// to our io.Pipe
err = png.Encode(part, img)
if err != nil {
t.Error(err)
}
}()
// We read from the pipe which receives data
// from the multipart writer, which, in turn,
// receives data from png.Encode().
// We have 3 chained writers!
request := httptest.NewRequest("POST", "/", pr)
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler := UploadFileHandler()
handler.ServeHTTP(response, request)
t.Log("It should respond with an HTTP status code of 200")
if response.Code != 200 {
t.Errorf("Expected %s, received %d", 200, response.Code)
}
t.Log("It should create a file named 'someimg.png' in uploads folder")
if _, err := os.Stat("./uploads/someimg.png"); os.IsNotExist(err) {
t.Error("Expected file ./uploads/someimg.png' to exist")
}
}
This function makes use of the image package to generate a file dynamically taking advantage of the fact that you can pass an io.Writer to png.Encode. In the same vein, you could pass your multipart Writer to generate the bytes in a CSV format (NewWriter in package "encoding/csv"), generating a file on the fly, without needing to read anything from your filesystem.
If you have a look at the implementation of the FormFile function you'll see that it reads the exposed MultipartForm field.
https://golang.org/src/net/http/request.go?s=39022:39107#L1249
// FormFile returns the first file for the provided form key.
1258 // FormFile calls ParseMultipartForm and ParseForm if necessary.
1259 func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
1260 if r.MultipartForm == multipartByReader {
1261 return nil, nil, errors.New("http: multipart handled by MultipartReader")
1262 }
1263 if r.MultipartForm == nil {
1264 err := r.ParseMultipartForm(defaultMaxMemory)
1265 if err != nil {
1266 return nil, nil, err
1267 }
1268 }
1269 if r.MultipartForm != nil && r.MultipartForm.File != nil {
1270 if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
1271 f, err := fhs[0].Open()
1272 return f, fhs[0], err
1273 }
1274 }
1275 return nil, nil, ErrMissingFile
1276 }
In your test you should be able to create a test instance of multipart.Form and assign it to your request object - https://golang.org/pkg/mime/multipart/#Form
type Form struct {
Value map[string][]string
File map[string][]*FileHeader
}
Of course this will require that you use a real filepath which isn't great from a testing perspective. To get around this you could define an interface to read FormFile from a request object and pass a mock implementation into your EP struct.
Here is a good post with a few examples on how to do this: https://husobee.github.io/golang/testing/unit-test/2015/06/08/golang-unit-testing.html
I combined these and other answers into an Echo example without pipes or goroutines:
func Test_submitFile(t *testing.T) {
path := "testfile.txt"
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("object", path)
assert.NoError(t, err)
sample, err := os.Open(path)
assert.NoError(t, err)
_, err = io.Copy(part, sample)
assert.NoError(t, err)
assert.NoError(t, writer.Close())
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set(echo.HeaderContentType, writer.FormDataContentType())
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/submit")
if assert.NoError(t, submitFile(c)) {
assert.Equal(t, 200, rec.Code)
assert.Contains(t, rec.Body.String(), path)
fi, err := os.Stat(expectedPath)
if os.IsNotExist(err) {
t.Fatal("Upload file does not exist", expectedPath)
}
assert.Equal(t, wantSize, fi.Size())
}
}
By combining the previous answers, this worked for me:
filePath := "file.jpg"
fieldName := "file"
body := new(bytes.Buffer)
mw := multipart.NewWriter(body)
file, err := os.Open(filePath)
if err != nil {
t.Fatal(err)
}
w, err := mw.CreateFormFile(fieldName, filePath)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(w, file); err != nil {
t.Fatal(err)
}
// close the writer before making the request
mw.Close()
req := httptest.NewRequest(http.MethodPost, "/upload", body)
req.Header.Add("Content-Type", mw.FormDataContentType())
res := httptest.NewRecorder()
// router is of type http.Handler
router.ServeHTTP(res, req)

Sending a POST Request to external API from GCP cloud function returns 500 but not when sent locally

I'm currently trying to send a POST request to an external API from a GCP Cloud Function. I've tested the function extensively locally and it fulfills the request every time and also works from Postman, but when I run the exact same code from within a cloud function, it returns a 500 from the external API every single time.
I'm genuinely at a loss as to why when sending the POST request from within the cloud function it fails every single time.
Does GCP add any headers that might interfere with an external API call or is there a configuration option within the cloud function settings that needs to be configured to allow an external POST request?
I've attempted to implement an http retry mechanism, but that did not work either.
Again, locally and from Postman, the exact same code is successful every time I run it.
Here is the code I use to generate and send the request:
package email
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"github.com/hashicorp/go-retryablehttp"
)
var FailedRequestErr = errors.New("failed request to moosend")
const (
successCode = 0
moosendHost = "api.moosend.com/v3"
dailyNewsletterMailingListID = "2e461f4c-99d1-4a8e-80ea-168b20bdaf5f"
mainEmail = "jason#functionalbits.io"
campaignNameBase = "Functional Bits Newsletter - Issue"
campaignSubjectBase = "Functional Bits Issue"
)
type CreatingADraftCampaignRequest struct {
Name string `json:"Name"`
Subject string `json:"Subject"`
SenderEmail string `json:"SenderEmail"`
ReplyToEmail string `json:"ReplyToEmail"`
IsAB string `json:"IsAB"`
ConfirmationToEmail string `json:"ConfirmationToEmail,omitempty"`
WebLocation string `json:"WebLocation,omitempty"`
MailingLists []MailingLists `json:"MailingLists,omitempty"`
SegmentID string `json:"SegmentID,omitempty"`
ABCampaignType string `json:"ABCampaignType,omitempty"`
TrackInGoogleAnalytics string `json:"TrackInGoogleAnalytics,omitempty"`
DontTrackLinkClicks string `json:"DontTrackLinkClicks,omitempty"`
SubjectB string `json:"SubjectB,omitempty"`
WebLocationB string `json:"WebLocationB,omitempty"`
SenderEmailB string `json:"SenderEmailB,omitempty"`
HoursToTest string `json:"HoursToTest,omitempty"`
ListPercentage string `json:"ListPercentage,omitempty"`
ABWinnerSelectionType string `json:"ABWinnerSelectionType,omitempty"`
}
type MailingLists struct {
MailingListID string `json:"MailingListId"`
SegmentID float64 `json:"SegmentId,omitempty"`
}
type CampaignResponse struct {
Code int32 `json:"Code"`
Err interface{} `json:"Error"`
Context interface{} `json:"Context"`
}
type MoosendAPI struct {
apiKey string
client *http.Client
}
func NewMoosendAPI(apiKey string) *MoosendAPI {
retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 5
standardClient := retryClient.StandardClient()
return &MoosendAPI{
apiKey: apiKey,
client: standardClient,
}
}
func (m *MoosendAPI) CreateDraftCampaign(issueNumber string, webLocation string) (*CampaignResponse, error) {
campaign := CreatingADraftCampaignRequest{
Name: fmt.Sprintf("%s %s", campaignNameBase, issueNumber),
Subject: fmt.Sprintf("%s %s", campaignSubjectBase, issueNumber),
IsAB: "false",
WebLocation: webLocation,
MailingLists: []MailingLists{{MailingListID: dailyNewsletterMailingListID}},
SenderEmail: mainEmail,
ReplyToEmail: mainEmail,
ConfirmationToEmail: mainEmail,
TrackInGoogleAnalytics: "true",
}
body, err := json.Marshal(&campaign)
if err != nil {
log.Println("error marshalling campaign request")
return nil, err
}
fullURL := fmt.Sprintf("https://%s/campaigns/create.json?apikey=%s", moosendHost, m.apiKey)
req, err := http.NewRequest(http.MethodPost, fullURL, bytes.NewBuffer(body))
if err != nil {
log.Println("request error")
return nil, err
}
req.Header.Add("Content-Type", "application/json")
req.Header.Add("Accept", "application/json")
log.Printf("request: %+v", req)
resp, err := m.client.Do(req)
if resp.StatusCode != http.StatusOK {
return nil, FailedRequestErr
}
if err != nil {
log.Println("error sending request")
return nil, err
}
log.Printf("response: %+v", resp)
defer resp.Body.Close()
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println("error reading response body")
return nil, err
}
var draftResponse CampaignResponse
if err := json.Unmarshal(respBody, &draftResponse); err != nil {
log.Println("error unmarshalling response")
log.Printf("%+v", draftResponse)
return nil, err
}
return &draftResponse, nil
}
func (m *MoosendAPI) SendCampaign(campaignID string) error {
fullURL := fmt.Sprintf("https://%s/campaigns/%s/send.json?apikey=%s", moosendHost, campaignID, m.apiKey)
req, err := http.NewRequest(http.MethodPost, fullURL, nil)
if err != nil {
log.Println("error creating request")
return err
}
req.Header.Add("Content-Type", "application/json")
req.Header.Add("Accept", "application/json")
resp, err := m.client.Do(req)
if err != nil {
log.Println("error sending request")
return err
}
defer resp.Body.Close()
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println("error reading response body")
return err
}
var sendResponse CampaignResponse
if err := json.Unmarshal(respBody, &sendResponse); err != nil {
log.Println("error unmarshalling response")
log.Printf("%+v", sendResponse)
return err
}
return nil
}
Then how it's run in the main function code:
package function
import (
"context"
"encoding/json"
"errors"
"log"
"os"
"github.com/Functional-Bits/emailer-service/internal/email"
"github.com/Functional-Bits/emailer-service/internal/publish"
)
func CampaignGenerator(ctx context.Context, m publish.PubSubMessage) error {
moosendAPIKey, ok := os.LookupEnv("MOOSEND_API_KEY")
if !ok {
log.Println("missing moosendAPIKey")
}
mAPI := email.NewMoosendAPI(moosendAPIKey)
var msg publish.IncomingMessage
if err := json.Unmarshal(m.Data, &msg); err != nil {
log.Println(err)
return err
}
log.Printf("received message: %+v", msg)
log.Printf("generating draft campaign for issue %s", msg.IssueNumber)
draftResponse, err := mAPI.CreateDraftCampaign(msg.IssueNumber, msg.FileURL)
if err != nil {
log.Println(err)
return err
}
log.Printf("draft response: %+v", draftResponse)
campaignID, ok := draftResponse.Context.(string)
if !ok {
log.Printf("response didn't contain an ID: %+v", draftResponse)
return errors.New("no campaign generated")
}
log.Printf("sending campgain %s", campaignID)
if err := mAPI.SendCampaign(campaignID); err != nil {
log.Println(err)
return err
}
log.Printf("campaign successfully sent for issue number %s", msg.IssueNumber)
return nil
}
When this code is run locally, It correctly makes the 2 calls and sends an email campaign. When run from the cloud function I get a 500 internal server error with no additional information as to why. Link to API docs.
I get the following response from the external API (from my cloud function logs)
response: &{
Status:500 Internal Server Error
StatusCode:500
Proto:HTTP/1.1
ProtoMajor:1
ProtoMinor:1
Header:map[Access-Control-Allow-Headers:[Content-Type, Accept, Cache-Control, X-Requested-With]
Access-Control-Allow-Methods:[GET, POST, OPTIONS, DELETE, PUT]
Access-Control-Allow-Origin:[*]
Cache-Control:[private]
Content-Length:[12750]
Content-Type:[text/html; charset=utf-8]
Date:[Sun, 12 Dec 2021 07:00:09 GMT]
Server:[Microsoft-IIS/10.0]
X-Aspnet-Version:[4.0.30319]
X-Powered-By:[ASP.NET]
X-Robots-Tag:[noindex, nofollow]
X-Server-Id:[1]]
Body:0xc0003f04c0
ContentLength:12750
TransferEncoding:[]
Close:false
Uncompressed:false
Trailer:map[]
Request:0xc000160b00
TLS:0xc000500630
}
The response causes an unmarshal error because no campaign ID is returned.

Unit Testing http.MultipartForm [duplicate]

How do I set the Request.FormFile when trying to test an endpoint?
Partial code:
func (a *EP) Endpoint(w http.ResponseWriter, r *http.Request) {
...
x, err := strconv.Atoi(r.FormValue("x"))
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
f, fh, err := r.FormFile("y")
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
...
}
How do I use the httptest lib to generate a post request that has value that I can get in FormFile?
You don't need to mock the complete FormFile struct as suggested by the other answer. The mime/multipart package implements a Writer type that lets you create a FormFile. From the docs
CreateFormFile is a convenience wrapper around CreatePart. It creates
a new form-data header with the provided field name and file name.
func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error)
Then, you can pass this io.Writer to httptest.NewRequest, which accepts a reader as an argument.
request := httptest.NewRequest("POST", "/", myReader)
To do this, you can either write the FormFile to an io.ReaderWriter buffer or use an io.Pipe. Here is a complete example that makes use of pipes:
func TestUploadImage(t *testing.T) {
// Set up a pipe to avoid buffering
pr, pw := io.Pipe()
// This writer is going to transform
// what we pass to it to multipart form data
// and write it to our io.Pipe
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
// We create the form data field 'fileupload'
// which returns another writer to write the actual file
part, err := writer.CreateFormFile("fileupload", "someimg.png")
if err != nil {
t.Error(err)
}
// https://yourbasic.org/golang/create-image/
img := createImage()
// Encode() takes an io.Writer.
// We pass the multipart field
// 'fileupload' that we defined
// earlier which, in turn, writes
// to our io.Pipe
err = png.Encode(part, img)
if err != nil {
t.Error(err)
}
}()
// We read from the pipe which receives data
// from the multipart writer, which, in turn,
// receives data from png.Encode().
// We have 3 chained writers!
request := httptest.NewRequest("POST", "/", pr)
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler := UploadFileHandler()
handler.ServeHTTP(response, request)
t.Log("It should respond with an HTTP status code of 200")
if response.Code != 200 {
t.Errorf("Expected %s, received %d", 200, response.Code)
}
t.Log("It should create a file named 'someimg.png' in uploads folder")
if _, err := os.Stat("./uploads/someimg.png"); os.IsNotExist(err) {
t.Error("Expected file ./uploads/someimg.png' to exist")
}
}
This function makes use of the image package to generate a file dynamically taking advantage of the fact that you can pass an io.Writer to png.Encode. In the same vein, you could pass your multipart Writer to generate the bytes in a CSV format (NewWriter in package "encoding/csv"), generating a file on the fly, without needing to read anything from your filesystem.
If you have a look at the implementation of the FormFile function you'll see that it reads the exposed MultipartForm field.
https://golang.org/src/net/http/request.go?s=39022:39107#L1249
// FormFile returns the first file for the provided form key.
1258 // FormFile calls ParseMultipartForm and ParseForm if necessary.
1259 func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
1260 if r.MultipartForm == multipartByReader {
1261 return nil, nil, errors.New("http: multipart handled by MultipartReader")
1262 }
1263 if r.MultipartForm == nil {
1264 err := r.ParseMultipartForm(defaultMaxMemory)
1265 if err != nil {
1266 return nil, nil, err
1267 }
1268 }
1269 if r.MultipartForm != nil && r.MultipartForm.File != nil {
1270 if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
1271 f, err := fhs[0].Open()
1272 return f, fhs[0], err
1273 }
1274 }
1275 return nil, nil, ErrMissingFile
1276 }
In your test you should be able to create a test instance of multipart.Form and assign it to your request object - https://golang.org/pkg/mime/multipart/#Form
type Form struct {
Value map[string][]string
File map[string][]*FileHeader
}
Of course this will require that you use a real filepath which isn't great from a testing perspective. To get around this you could define an interface to read FormFile from a request object and pass a mock implementation into your EP struct.
Here is a good post with a few examples on how to do this: https://husobee.github.io/golang/testing/unit-test/2015/06/08/golang-unit-testing.html
I combined these and other answers into an Echo example without pipes or goroutines:
func Test_submitFile(t *testing.T) {
path := "testfile.txt"
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("object", path)
assert.NoError(t, err)
sample, err := os.Open(path)
assert.NoError(t, err)
_, err = io.Copy(part, sample)
assert.NoError(t, err)
assert.NoError(t, writer.Close())
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set(echo.HeaderContentType, writer.FormDataContentType())
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/submit")
if assert.NoError(t, submitFile(c)) {
assert.Equal(t, 200, rec.Code)
assert.Contains(t, rec.Body.String(), path)
fi, err := os.Stat(expectedPath)
if os.IsNotExist(err) {
t.Fatal("Upload file does not exist", expectedPath)
}
assert.Equal(t, wantSize, fi.Size())
}
}
By combining the previous answers, this worked for me:
filePath := "file.jpg"
fieldName := "file"
body := new(bytes.Buffer)
mw := multipart.NewWriter(body)
file, err := os.Open(filePath)
if err != nil {
t.Fatal(err)
}
w, err := mw.CreateFormFile(fieldName, filePath)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(w, file); err != nil {
t.Fatal(err)
}
// close the writer before making the request
mw.Close()
req := httptest.NewRequest(http.MethodPost, "/upload", body)
req.Header.Add("Content-Type", mw.FormDataContentType())
res := httptest.NewRecorder()
// router is of type http.Handler
router.ServeHTTP(res, req)

golang: GET request for presigned url of S3 object with SSE fails with 403 error

I'm getting 403 Forbidden HTTP response when I make a GET request to a presigned url generated by the AWS Presign method in Go.
The error message is:
The request signature we calculated does not match the signature you provided. Check your key and signing method
X-Amz-SignedHeaders is: host;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5
I'm writing the object to S3 like this:
type DocumentStore struct {
bucketName string
bucketEncryptionKeyAlias string
aws *session.Session
}
func (s *DocumentStore) PutDocument(ctx context.Context, envelope []byte, metadata map[string]string) (PutDocumentResult, error) {
uploader := s3manager.NewUploader(s.aws)
var objectKey = uuid.New().String()
if _, err := uploader.UploadWithContext(ctx, &s3manager.UploadInput{
Bucket: aws.String(s.bucketName),
Key: aws.String(objectKey),
ContentType: aws.String("application/octet-stream"),
Body: bytes.NewReader(envelope),
ServerSideEncryption: aws.String(s3.ServerSideEncryptionAwsKms),
SSEKMSKeyId: aws.String(s.bucketEncryptionKeyAlias),
Metadata: aws.StringMap(metadata),
}); err != nil {
return PutDocumentResult{}, fmt.Errorf("put document failed on upload: %v", err.Error())
}
return PutDocumentResult{
BucketName: s.bucketName,
ObjectKey: objectKey,
}, nil
}
I'm signing the url like this:
func (s *DocumentStore) NewSignedGetURL(ctx context.Context, objectKey string, ttl time.Duration) (string, error) {
svc := s3.New(s.aws)
req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String(s.bucketName),
Key: aws.String(objectKey),
SSECustomerKey: aws.String(s.bucketEncryptionKeyAlias),
SSECustomerAlgorithm: aws.String(s3.ServerSideEncryptionAwsKms),
})
url, err := req.Presign(ttl)
if err != nil {
return "", fmt.Errorf("failed to presign GetObjectRequest for key %q: %v", objectKey, err)
}
return url, nil
}
And I'm calling the methods like this:
result, err := target.PutDocument(context.TODO(), envelope, metadata)
if err != nil {
t.Errorf("PutDocument failed: %v", err)
return
}
getURL, err := target.NewSignedGetURL(context.TODO(), result.ObjectKey, time.Minute*5)
if err != nil {
t.Errorf("failed to sign url: %v", err)
return
}
req, _ := http.NewRequest("GET", getURL, nil)
req.Header.Add("x-amz-server-side-encryption-customer-algorithm", s3.ServerSideEncryptionAwsKms)
req.Header.Add("x-amz-server-side-encryption-customer-key", test.cfg.AWS.BucketKMSAlias)
req.Header.Add("x-amz-server-side-encryption-customer-key-md5", "")
resp, err := http.DefaultClient.Do(req.WithContext(context.TODO()))
if err != nil {
t.Errorf("failed to request object from signed url: %v", err)
return
}
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Errorf("failed to read object stream from S3: %v", err)
return
}
if resp.StatusCode != http.StatusOK {
t.Errorf("failed to get object. Http status: %s(%d)\n%s", resp.Status, resp.StatusCode, data)
return
}
I can read the download the file from the aws cli like this:
aws --profile dispatcher_stage --region us-east-1 s3 cp s3://[bucket-name]/0c/09179312-e283-431c-ab71-6a0c437177fe . --sse aws:kms --sse-kms-key-id alias/[key-alias-name]
What am I missing?
I figured it out: the GetObject request doesn't need the SSE parameters as long as the user has Decrypt permission on the KMS key. Here's the relevant changes:
I'm now signing the url like this:
func (s *DocumentStore) NewSignedGetURL(ctx context.Context, objectKey string, ttl time.Duration) (string, error) {
svc := s3.New(s.aws)
req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String(s.bucketName),
Key: aws.String(objectKey),
})
url, err := req.Presign(ttl)
if err != nil {
return "", fmt.Errorf("failed to presign GetObjectRequest for key %q: %v", objectKey, err)
}
return url, nil
}
And I'm downloading the object like this:
getURL, err := target.NewSignedGetURL(context.TODO(), result.ObjectKey, time.Minute*5)
if err != nil {
t.Errorf("failed to sign url: %v", err)
return
}
req, _ := http.NewRequest("GET", getURL, nil)
req.Header.Add("host", req.Host)
resp, err := http.DefaultClient.Do(req.WithContext(context.TODO()))
if err != nil {
t.Errorf("failed to request object from signed url: %v", err)
return
}
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Errorf("failed to read object stream from S3: %v", err)
return
}

How to test an HTTP function which takes folder as an input?

I have an HTTP handler function (POST) which allows a user to upload a folder from a web browser application. The folder is passed from JavaScript code as an array of files in a folder and on the backend (Go API) it is accepted as a []*multipart.FileHeader. I am struggling in writing a Go unit test for this function. How can I pass a folder as input from a test function? I need help in creating the httpRequest in the right format.
I have tried to use / set values for an array of FileHeader, but some attributes are not allowed to be imported. So there must be a different way of testing this handler that I am not aware of.
Handler Function for folder upload:
func FolderUpload(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
// SOME LOGIC
files := r.MultipartForm.File["multiplefiles"] // files is of the type []*multipart.FileHeader
// SOME LOGIC TO PARSE THE FILE NAMES TO RECREATE THE SAME TREE STRUCTURE ON THE SERVER-SIDE AND STORE THEM AS A FOLDER
Unit Test function for the same handler:
func TestFolderUpload(t *testing.T) {
// FolderPreCondition()
request, err := http.NewRequest("POST", uri, body) //Q: HOW TO CREATE THE BODY ACCEPTABLE BY THE ABOVE HANDLER FUNC?
// SOME ASSERTION LOGIC
}
You should write your file to request:
func newFileUploadRequest(url string, paramName, path string) (*http.Request, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile(paramName, filepath.Base(path))
if err != nil {
return nil, err
}
_, err = io.Copy(part, file)
if err != nil {
return nil, err
}
err = writer.Close()
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", url, body)
if err != nil {
return nil, err
}
req.Header.Add("Content-Type", writer.FormDataContentType())
return req, err
}
then use it:
req, err := newFileUploadRequest("http://localhost:1234/upload", "multiplefiles", path)
client := &http.Client{}
resp, err := client.Do(req)
It works for me, hope it helps you)