I am following this guide monitoring_opencensus_metrics_quickstart-go. Besides, I also tried the way from this answer.
Code here:
package main
import (
"context"
"fmt"
"log"
"os"
"path"
"time"
"google.golang.org/api/option"
"contrib.go.opencensus.io/exporter/stackdriver"
"go.opencensus.io/stats"
"go.opencensus.io/stats/view"
"golang.org/x/exp/rand"
)
var (
// The task latency in milliseconds.
latencyMs = stats.Float64("task_latency", "The task latency in milliseconds", "ms")
)
func main() {
ctx := context.Background()
v := &view.View{
Name: "task_latency_distribution",
Measure: latencyMs,
Description: "The distribution of the task latencies",
Aggregation: view.Distribution(0, 100, 200, 400, 1000, 2000, 4000),
}
if err := view.Register(v); err != nil {
log.Fatalf("Failed to register the view: %v", err)
}
exporter, err := stackdriver.NewExporter(stackdriver.Options{
ProjectID: os.Getenv("GOOGLE_CLOUD_PROJECT"),
MonitoringClientOptions: []option.ClientOption{
option.WithCredentialsFile(path.Join("./.gcp/stackdriver-monitor-admin.json")),
},
})
if err != nil {
log.Fatal(err)
}
view.RegisterExporter(exporter)
view.SetReportingPeriod(60 * time.Second)
// Flush must be called before main() exits to ensure metrics are recorded.
defer exporter.Flush()
if err := exporter.StartMetricsExporter(); err != nil {
log.Fatalf("Error starting metric exporter: %v", err)
}
defer exporter.StopMetricsExporter()
// Record 100 fake latency values between 0 and 5 seconds.
for i := 0; i < 100; i++ {
ms := float64(5*time.Second/time.Millisecond) * rand.Float64()
fmt.Printf("Latency %d: %f\n", i, ms)
stats.Record(ctx, latencyMs.M(ms))
time.Sleep(1 * time.Second)
}
fmt.Println("Done recording metrics")
}
I run above code locally, NOT in GCE, GAE and GKE environments.
At the metrics explorer web UI, here is the metric query condition:
Resource Type: Consumed API
Metric: custom.googleapis.com/opencensus/task_latency_distribution
Full query:
fetch consumed_api
| metric 'custom.googleapis.com/opencensus/task_latency_distribution'
| align delta(1m)
| every 1m
| group_by [],
[value_task_latency_distribution_aggregate:
aggregate(value.task_latency_distribution)]
The service account has Monitoring Admin role.
But got No data is available for the selected time frame.
It works for me but I've tended to do things slightly differently:
export GOOGLE_APPLICATION_CREDENTIALS=path/to/creds instead of MonitoringClientOptions{};
Previously (!?) I had problems with distributions with a 0 bucket; try removing that initial (0) bucket and try again;
Drop the resource.type from the metrics explorer; from APIs Explorer, if any, this should be global
Google APIs Explorer is an excellent way to diagnose Stackdriver API challenges. You can use it to list metrics and to list timeseries (replace your-project-id and update the 2 interval values):
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/list?apix=true&apix_params=%7B%22name%22%3A%22projects%2Fyour-project-id%22%2C%22filter%22%3A%22metric.type%3D%5C%22custom.googleapis.com%2Fopencensus%2Ftask_latency_distribution%5C%22%22%7D
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list?apix=true&apix_params=%7B%22name%22%3A%22projects%2Fyour-project-id%22%2C%22aggregation.alignmentPeriod%22%3A%22%2B300s%22%2C%22aggregation.crossSeriesReducer%22%3A%22REDUCE_MEAN%22%2C%22aggregation.perSeriesAligner%22%3A%22ALIGN_DELTA%22%2C%22filter%22%3A%22metric.type%3D%5C%22custom.googleapis.com%2Fopencensus%2Ftask_latency_distribution%5C%22%22%2C%22interval.endTime%22%3A%222020-09-08T23%3A59%3A59Z%22%2C%22interval.startTime%22%3A%222020-09-08T00%3A00%3A00Z%22%7D
Using Chrome's Developer Console, you can find one of Stackdriver's calls to the API to more easily reproduce it, e.g.
filter: metric.type="custom.googleapis.com/opencensus/task_latency_distribution"
aggregation.crossSeriesReducer: REDUCE_MEAN
aggregation.alignmentPeriod: +60s
aggregation.perSeriesAligner: ALIGN_DELTA
secondaryAggregation.crossSeriesReducer: REDUCE_NONE
interval.startTime: 2020-09-08T23:59:59Z
interval.endTime: 2020-09-08T00:00:00Z
Related
I am at the moment trying to set up a connection to AWS Neptune via go, but its not working. I am able to connect to AWS itself, but when I try to connect to Neptune DB it says "no successful connections could be made: dial tcp 172.31.4.48:8182: i/o timeout". I am using the Gremlingo module like in this code
package main
import (
"fmt"
"net/http"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/neptune"
"github.com/gin-gonic/gin"
gremlingo "github.com/apache/tinkerpop/gremlin-go/v3/driver"
)
func main() {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-id1"),
Credentials: credentials.NewStaticCredentials("AWS-id key", "aws secret id key", ""),
})
if err != nil {
fmt.Println("Couldn't create new session")
return
}
neptune.New(sess)
driverRemoteConnection, err := gremlingo.NewDriverRemoteConnection("wss://database-1-instance-1.asdasdasd.us-east-1.neptune.amazonaws.com:8182/gremlin",
func(settings *gremlingo.DriverRemoteConnectionSettings) {
settings.TraversalSource = "g"
})
if err != nil {
fmt.Println(err)
return
}
//Cleanup
defer driverRemoteConnection.Close()
//Creating graph traversal
g := gremlingo.Traversal_().WithRemote(driverRemoteConnection)
// Perform traversal
results, err := g.V().Limit(2).ToList()
if err != nil {
fmt.Println(err)
return
}
// print results
for _, r := range results {
fmt.Println(r.GetString())
}
}
I wasn't quite sure what the problem was so I tried to connect to the cluster itself and as it didn't work I tried to connect to the Writer.
Thank you very much for your help.
Best regards
Amazon Neptune runs inside of a VPC and does not expose a public endpoint. Code designed to send queries must have access tho that VPC. This could be as simple as the code running on an EC2 instance in the same VPC, but there are many other ways that access to a VPC can be granted, such as Load Balancers, VPC Peering, Direct Connect, and many others.
An easy way to check if your code can access the database, is to send an HTTP request to the /status API from the same point of origin and see if it works.
I've the AWS instance type(e.g. c5.18xlarge etc.) in my config files. During runtime, I would like to fetch the number of vCPUs(e.g. 72) from here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html and do some computations and launch instances based on the number of vCPUs.
I can store this data from the webpage in a map and refer the map, but is there a way to fetch this information from AWS using AWS go client?
You can use DescribeInstanceTypes
For example:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ec2"
)
func main() {
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("eu-west-1")},
)
svc := ec2.New(sess)
resp, _ := svc.DescribeInstanceTypes(&ec2.DescribeInstanceTypesInput{})
for _, v := range resp.InstanceTypes {
fmt.Println(*v.InstanceType, v.VCpuInfo)
}
}
I have golang based lambda function which does some work and logs the information during execution.
I'm using zapcore Golang logger as below
func NewLogger(logLevel string) (*zap.SugaredLogger, error) {
encoderConfig := zapcore.EncoderConfig{
TimeKey: "Time",
LevelKey: "Level",
NameKey: "Name",
CallerKey: "Caller",
MessageKey: "Msg",
StacktraceKey: "St",
EncodeLevel: zapcore.CapitalColorLevelEncoder,
EncodeTime: zapcore.ISO8601TimeEncoder,
EncodeDuration: zapcore.StringDurationEncoder,
EncodeCaller: zapcore.ShortCallerEncoder,
}
consoleEncoder := zapcore.NewConsoleEncoder(encoderConfig)
consoleOut := zapcore.Lock(os.Stdout)
var level zap.AtomicLevel
err := level.UnmarshalText(([]byte(logLevel)))
if err != nil {
level.UnmarshalText([]byte(defaultLogLevel))
}
core := zapcore.NewTee(zapcore.NewCore(
consoleEncoder,
consoleOut,
level,
))
logger := zap.New(core)
Logger = logger.Sugar()
return Logger, nil
}
However when it generates the logs in CloudWatch, I see an extra character in the logs.
2022-06-30T21:52:43.310-07:00 2022-07-01T04:52:43.310Z [34mINFO[0m Process my event
Why the logs is getting generated with [34m and [0m string in it?
Do I really need to generate logs with timestamp because I see cloudwatch already adds timestamp to logs.
To disable colors you can remove ASCII Escape Codes:
- EncodeLevel: zapcore.CapitalColorLevelEncoder
+ EncodeLevel: zapcore.CapitalLevelEncoder
I don't think you need to repeat time information, if you need more metrics later you'll probably log as JSONs and there will be already time information when querying from CloudWatch.
I have created REST API using Go Iris framework. Now I want to deploy these API's on AWS with lambda function. I am using MySQL as database. Is it possible to deploy my Go executable file on AWS lambda or should I need to modify my code according to AWS lambda specifications? I am trying to find the solution, but not getting much information.
Here is one of my API end point.
package main
import (
"database/sql"
"github.com/kataras/iris"
"github.com/kataras/iris/middleware/logger"
"github.com/kataras/iris/middleware/recover"
)
type Reward struct {
Id int `json:"reward_id"`
LotteryID int `json:"lottery_id"`
RewardName string `json:"reward_name"`
Description string `json:"reward_description"`
Asset int `json:"reward_asset"`
AssetName string `json:"reward_asset_name"`
}
func dbConn() (db *sql.DB) {
dbDriver := "mysql"
dbUser := "xxx"
dbPass := "xxx"
dbName := "xxx"
db, err := sql.Open(xxxxxxxxx)
if err != nil {
panic(err.Error())
}
return db
}
func newApp() *iris.Application {
app := iris.New()
app.Logger().SetLevel("debug")
app.Use(recover.New())
app.Use(logger.New())
db := dbConn()
app.Get("/reward/{reward_id:int}", func(ctx iris.Context) {
id1 := ctx.Params().GetIntDefault("reward_id", 0)
stmtOut, err := db.Prepare("select id, lottery_id,reward_name,reward_description,reward_asset, reward_asset_name from rewards_table where id =?")
if err != nil {
panic(err.Error())
}
defer stmtOut.Close()
var id, lotteryId, rewardAsset int
var rewardName, rewardDescription, rewardAssetName string
err1 := stmtOut.QueryRow(id1).Scan(&id, &lotteryId, &rewardName, &rewardDescription, &rewardAsset, &rewardAssetName)
if err1 != nil {
panic(err.Error())
}
reward := Reward{
Id: id,
LotteryID: lotteryId,
RewardName: rewardName,
Description: rewardDescription,
Asset: rewardAsset,
AssetName: rewardAssetName,
}
ctx.JSON(&reward)
})
return app
}
func main() {
app := newApp()
app.Run(iris.Addr(":8080"), iris.WithoutServerError(iris.ErrServerClosed), iris.WithOptimizations)
}
I have few more API endpoints which do basic CRUD operations. I am thinking about using AWS lambda and AWS API Gateway.
should I need to modify my code according to AWS lambda specifications?
Yes. Your code for lambda will require to have a handler:
AWS Lambda function handler in Go
This is the entry point to your function.
Also it seems that your go program is a web server build on iris. If this is the case, you won't be able to use it anyway, as you can't invoke lambda from internet as you would a regular server.
Also lambda runs for max 15 minutes, thus its use as a server would be very limited.
AWS Config has a set of Managed Rules and I am trying to use the Golang AWS SDK to use the DescribeConfigRules API to retrieve the list of AWS Config Managed Rule Names and other details.
It seems like every request receives a response of 25 rules and a NextToken for the next set of results. What I am having trouble understanding is how do I use this NextToken to retrieve the next set of results?
Here is what I have so far.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
func main() {
//Create an aws session
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-aws-profile")})
// Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
configrulesoutput, err := configsvc.DescribeConfigRules(rules)
if err != nil {
log.Fatal(err)
}
for _, rule := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", *rule.ConfigRuleName)
}
}
The above code successfully prints the first 25 rules received in the response. However I am not sure how to use the NextToken received in the response to get the next set of results.
Sample Response.
ConfigRules: [
{
ConfigRuleArn: "ConfigRuleARN",
ConfigRuleId: "config-rule-ppwclr",
ConfigRuleName: "cloudtrail-enabled",
ConfigRuleState: "ACTIVE",
Description: "Checks whether AWS CloudTrail is enabled in your AWS account. Optionally, you can specify which S3 bucket, SNS topic, and Amazon CloudWatch Logs ARN to use.",
InputParameters: "{}",
MaximumExecutionFrequency: "TwentyFour_Hours",
Source: {
Owner: "AWS",
SourceIdentifier: "CLOUD_TRAIL_ENABLED"
}
},
{ Rule 2 }, ....{ Rule 25}
],
NextToken: "nexttoken"
}
Code extracts the rulenames from the response and output is as below.
Rule: cloudtrail-enabled
Rule: restricted-ssh
Rule: securityhub-access-keys-rotated
Rule: securityhub-autoscaling-group-elb-healthcheck-required
Rule: securityhub-cloud-trail-cloud-watch-logs-enabled
Rule: securityhub-cloud-trail-encryption-enabled
Rule: securityhub-cloud-trail-log-file-validation-enabled
Rule: securityhub-cloudtrail-enabled
Rule: securityhub-cmk-backing-key-rotation-enabled
Rule: securityhub-codebuild-project-envvar-awscred-check
Rule: securityhub-codebuild-project-source-repo-url-check
Rule: securityhub-ebs-snapshot-public-restorable-check
Rule: securityhub-ec2-managedinstance-patch-compliance
Rule: securityhub-ec2-security-group-attached-to-eni
Rule: securityhub-eip-attached
Rule: securityhub-elasticsearch-encrypted-at-rest
Rule: securityhub-elasticsearch-in-vpc-only
Rule: securityhub-iam-password-policy-ensure-expires
Rule: securityhub-iam-password-policy-lowercase-letter-check
Rule: securityhub-iam-password-policy-minimum-length-check
Rule: securityhub-iam-password-policy-number-check
Rule: securityhub-iam-password-policy-prevent-reuse-check
Rule: securityhub-iam-password-policy-symbol-check
Rule: securityhub-iam-password-policy-uppercase-letter-check
Rule: securityhub-iam-policy-no-statements-with-admin-access
End Goal: Using golang AWS SDK, extract the AWS Config Managed Rule details and put it in an excel format using Excelize to review which AWS Config rules we want enabled.
Thanks for your help in advance.
---New based on #Adrian's comment and doc reference---
As per doc
type DescribeConfigRulesInput struct {
// The names of the AWS Config rules for which you want details. If you do not
// specify any names, AWS Config returns details for all your rules.
ConfigRuleNames []*string `type:"list"`
// The nextToken string returned on a previous page that you use to get the
// next page of results in a paginated response.
NextToken *string `type:"string"`
// contains filtered or unexported fields }
So here is what I am trying. Specifying nil should give me back all rules. nextToken is blank string for the first call.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
nextToken := ""
rules.SetNextToken(nextToken)
getConfigRulesFunc(configsvc, rules)
//getConfigRulesFunc function
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for i, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
}
if *configrulesoutput.NextToken != "" {
ruleset := (*configservice.DescribeConfigRulesInput)(nil)
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Above code compiles fine but here the runtime error I believe because of nil.
configsvc type: *configservice.ConfigService
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13c7ed2]
goroutine 1 [running]:
github.com/aws/aws-sdk-go/service/configservice.(*DescribeConfigRulesInput).SetNextToken(...)
/Users/user/go/src/github.com/aws/aws-sdk-go/service/configservice/api.go:12230
main.main()
/Users/user/golang/awsgotest/awsgotest.go:26 +0x232
Ok, finally figured it out with the help of a very kind Alex Diehl via this ticket https://github.com/aws/aws-sdk-go/issues/3293 on the official aws-sdk-go repo.
I would still say the aws sdk for go definitely lacks simple examples for configservice at the least on recommended usage.
Here the code that works. This will also show how to use simple recursive function in go to use NextToken for pagination of api results that span multiple pages especially apis that do not have built in paginators.
Also note that DescribeConfigRules API does not list all AWS Managed Config Rules, only the Config rules enabled for your account.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
var i int = 0
func main() {
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-profile")})
if err != nil {
log.Fatal(err)
}
//Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
fmt.Printf("configsvc type: %T\n", configsvc)
rules := &configservice.DescribeConfigRulesInput{}
getConfigRulesFunc(configsvc, rules)
}
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for _, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
i = i + 1
}
if configrulesoutput.NextToken != nil {
fmt.Println("In if nexttoken is not empty")
fmt.Println("Print NextToken: ", *configrulesoutput.NextToken)
ruleset := &configservice.DescribeConfigRulesInput{}
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Code in Bold were the ones giving me grief on how to use the NextToken based on best practices atleast for the go sdk for aws.
FYI, you could have looked in the AWS Go guide as there is a section on pagination: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/making-requests.html#using-pagination-methods.