Getting SerializationException while trying to PutLogEvents on cloudwatch using golang - amazon-web-services

I am trying to achieve following using my program:
Create log-group on aws cloudwatch
Create log-stream under above log-group
Put log-events under above log-stream
All this using go lang
package main
import (
"time"
"fmt"
"github.com/jcxplorer/cwlogger"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/aws"
)
func main() {
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
svc := cloudwatchlogs.New(sess)
logGroupName := "my-log-group";
logStreamName := "my-log-stream";
logGroupInput := cloudwatchlogs.CreateLogGroupInput{LogGroupName: &logGroupName}
svc.CreateLogGroup(&logGroupInput);
logStreamInput := cloudwatchlogs.CreateLogStreamInput{LogGroupName: &logGroupName, LogStreamName: &logStreamName}
svc.CreateLogStream(&logStreamInput)
logevents := make([]*cloudwatchlogs.InputLogEvent, 1)
logevents = append(logevents, &cloudwatchlogs.InputLogEvent{
Message: aws.String("Simple log message"),
Timestamp: aws.Int64(111),
})
p := cloudwatchlogs.PutLogEventsInput{LogEvents: logevents, LogGroupName: &logGroupName, LogStreamName: &logStreamName}
resp, err := svc.PutLogEvents(&p)
if err != nil {
panic(err)
}
fmt.Print("Next Token: {}", resp.NextSequenceToken)
}
Now when I run above code, it successfully creates log-group and log-stream and I can verify that in aws cloudwatch. But for some reason PutLogEvents fails with following error:
panic: SerializationException:
status code: 400, request id: 0685efcc-47e3-11e9-b528-81f33ec2f468
I am not sure what may be wrong here. Any suggestion or direction will be really helpful.
Thanks in advance.

Reason for SerializationException was:logevents := make([]*cloudwatchlogs.InputLogEvent, 1)
followed by append which created first empty entry in slice. I replaced code with
logevents := make([]*cloudwatchlogs.InputLogEvent, 0) and it got resolved.
Additionally while debugging to find why logs were not getting populated I figured out that timestamp value used is not valid one. According to aws documentation timestamp for each event can't be older than 14 days and can't be more than 2hr in future.
Here is link: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html
Hope it will be helpful to someone facing similar issue in future.

Related

AWS Go SDK not finding the credentials file at C:/###/.aws/credentials

I am using Amazon Kinesis and the Go SDK for AWS, but I'm getting an error.
This is my code:
package main
import (
"math/rand"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
_kinesis "github.com/aws/aws-sdk-go/service/kinesis"
)
func main() {
session, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
})
handleErr(err)
kinesis := _kinesis.New(session)
laugh := strings.Builder{}
laughingSounds := []string{"haha", "hoho", "hehe", "hehehe", "*snicker*"}
for i := 0; i < 10; i++ {
laugh.WriteString(laughingSounds[rand.Intn(len(laughingSounds))])
}
_, err = kinesis.PutRecord(&_kinesis.PutRecordInput{
Data: []byte(laugh.String()),
PartitionKey: aws.String("laughs"),
StreamName: aws.String("laughs"),
})
handleErr(err)
}
func handleErr(err error) {
if err != nil {
panic(err)
}
}
When I run this, I get an error:
panic: UnrecognizedClientException: The security token included in the request is invalid.
status code: 400, request id: dc139793-cd38-fb30-86a3-f92b6410e1c7
goroutine 1 [running]:
main.handleErr(...)
C:/Users/####/----/main.go:5
main.main()
C:/Users/####/----/main.go:34 +0x3ac
exit status 2
I have run aws configure:
$ aws configure
AWS Access Key ID [None]: ####
AWS Secret Access Key [None]: ####
Default region name [None]: us-east-1
Default output format [None]:
and the C:/users/####/.aws/credentials file is created with the correct configuration. But my program still wouldn't execute successfully.
I tried to set an environment variable, but to no avail.
$ $env:aws_access_key_id="####"
Version info:
$ pwsh -v
PowerShell 7.2.2
$ aws -v
aws-cli/2.4.27 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
OS: Windows 11
I found the answer in GoDoc, I just had to change a config setting and use NewSessionWithOptions:
session, err := session.NewSessionWithOptions(session.Options{
Config: aws.Config{
Region: aws.String("<your region>"),
}, SharedConfigState: session.SharedConfigEnable,
})
Be sure to run aws configure with the correct credentials (or even just manually create the ~/.aws/credentials file), or this won't work.

Amazon SQS:: Got an error while trying to create queue: NoCredentialProviders: no valid providers in chain. Deprecated

I am trying to create Amazon SQS from my local machine and facing errors like
Got an error while trying to create queue: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
What I did:
Step-1:
I have set up my credentials in .aws/credentials file
[default]
aws_access_key_id = TestAccessKey
aws_secret_access_key = TestSecretAccessKey
Step-2:
My code in go lang like below
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sqs"
)
func CreateQueue(sess *session.Session, queueName string) (*sqs.CreateQueueOutput, error) {
sqsClient := sqs.New(sess)
result, err := sqsClient.CreateQueue(&sqs.CreateQueueInput{
QueueName: &queueName,
Attributes: map[string]*string{
"DelaySeconds": aws.String("0"),
"VisibilityTimeout": aws.String("60"),
},
})
if err != nil {
return nil, err
}
return result, nil
}
func main() {
sess, err := session.NewSessionWithOptions(session.Options{
Profile: "default",
Config: aws.Config{
Region: aws.String("us-east-1"),
},
})
if err != nil {
fmt.Printf("Failed to initialize new session: %v", err)
return
}
queueName := "my-new-queue"
createRes, err := CreateQueue(sess, queueName)
if err != nil {
fmt.Printf("Got an error while trying to create queue: %v", err)
return
}
fmt.Println("Created a new queue with url: " + *createRes.QueueUrl)
}
Step-3:
Try updating profile in Shared Credential File (.aws/credentials) as below:
[default]
aws_access_key_id=TestAccessKey
aws_secret_access_key=TestSecretAccessKey
region=us-east-1
I just added region at the end.
To verify if your CLI session has correct AWS credentials configured run this command aws sts get-caller-identity This command will show which profile is used. If this works fine then you can run any simple AWS commands something like S3.getBuckets() to verify if the dev environment is setup correctly.
These 2 solutions should give you enough input to figure out what's wrong.

Image upload failed using AWS SDK V2 in GO

I am using AWS S3 service to upload images. Yesterday I updated the SDK v1 to v2 and found that the image upload is failing with the following error:
operation error S3: PutObject, https response error StatusCode: 403, RequestID: XXXXXXXXXXX, HostID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
UPDATED:
I have aws credentials on my home folder in linux in .aws folder in the following format:
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
Here is the code:
package main
import (
"context"
"fmt"
"io"
"net/http"
"github.com/aws/aws-sdk-go-v2/aws"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func main() {
fileName := "test123.jpg"
filePath := "/BUCKET_NAME/uploads/aman/2021/6/25/"
res, err := http.Get("https://images.app.goo.gl/mpQ5nXYXjdUMKGgW7")
if err != nil || res.StatusCode != 200 {
// handle errors
}
defer res.Body.Close()
UploadFileInS3Bucket(res.Body, fileName, filePath)
}
func UploadFileInS3Bucket(file io.Reader, fileName, filePath string) {
cfg, err := awsconfig.LoadDefaultConfig(context.TODO(),
awsconfig.WithRegion("REGION"),
)
client := s3.NewFromConfig(cfg)
uploader := manager.NewUploader(client)
uploadResp, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
Bucket: aws.String(filePath),
Key: aws.String(fileName),
Body: file,
ContentType: aws.String("image"),
})
fmt.Println(uploadResp)
fmt.Println(err)
}
I did not change any credentials/buckets/regions in my code.However if I run the code with SDK v1 then it works fine & images are uploading.
What is going wrong with the SDK v2 ?
After spending a couple of days, I came to know that SDK V2 takes following format for Bucket & Key field:
fileName := "uploads/2021/6/25/test123.jpg"
filePath := "BUCKET_NAME"
Basically for these fields there is vice versa behaviour in SDK V1 & V2. Above is the V2. Below is the V1:
fileName := "test123.jpg"
filePath := "/BUCKET_NAME/uploads/2021/6/25/"

How to unittest prometheus GaugeVec

I am new to golang so it's kind of hard to understand what would be the best way to test prometheus.GuageVec.
Here is my try 1. where i am trying to get metrics channel. But this request is timing out?
cb, _ := GaugeVecs[name]
ch := make(chan prometheus.Metric)
cb.Collect(ch)
metric := <-ch
fmt.Printf("%+v", metric)
The second try that i did was
cb, _ := GaugeVecs[name]
err := testutil.CollectAndCompare(cb, strings.NewReader(`expected outcome`), "name")
assert.Nil(t, err)
^^^ this one got error that parsing expected metrics failed: text format parsing error in line 1: unexpected end of input stream.
How can i test the output of metric?

x509: certificate signed by unknown authority using AWS IoT

When trying to publish a message to a topic using the AWS IoT SDK for go I get the following error: "x509: certificate signed by unknown authority".
I am on windows and all I did was install different root CA's (literally via doubleclick) and a device certificate generated by AWS IoT Console.
I feel like I should somehow specify the path to the certificate but unlike the Python SDk the one for go does not mention that anywhere. I also added my credentials via the AWS Cli so that should not be the issue.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/iotdataplane"
)
func main() {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("eu-central-1"), Endpoint: aws.String("xxxxxxxxxx.iot.eu-central-1.amazonaws.com")},
)
if err != nil {
log.Fatal(err)
}
iotDataSvc := iotdataplane.New(sess)
input := &iotdataplane.PublishInput{
Payload: []byte(`{
'state': {
'desired':{
'humidity':10,
'temp':10
}
}
}`),
Topic: aws.String("/update"),
Qos: aws.Int64(0),
}
resp, err := iotDataSvc.Publish(input)
if err != nil {
log.Fatal(err)
}
fmt.Println(resp)
}
Found the mistake: xxxxxxxxxx.iot.eu-central-1.amazonaws.com needs to be xxxxxxxxxx-ats.iot.eu-central-1.amazonaws.com.