How to mock redis cluster server in Golang ?
Simple redis server can be mocked using 'github.com/alicebob/miniredis' package but it does not support cluster mocking.
Received the following error:
ERR unknown command cluster, with args beginning with: slots``
My program uses 'github.com/go-redis/redis' package for redis implementation.
For example - create mock server for the following client
redisCache := redis.NewClusterClient(redisConfig)
_, err := redisCache.Ping().Result()
if err != nil {
log.Fatalf("fatal error Not able to connect using redis client: %s", err)
}
Maybe using "github.com/alicebob/miniredis/v2" instead of "github.com/alicebob/miniredis" would solve the problem.
Here's an example.
package redis_cluster_test
import (
"log"
"testing"
"time"
"github.com/alicebob/miniredis/v2"
"github.com/go-redis/redis"
)
func TestRedisCluster(t *testing.T) {
mock, err := miniredis.Run()
if err != nil {
panic(err)
}
redisConfig := redis.ClusterOptions{
Addrs: []string{mock.Addr()},
ReadTimeout: 1 * time.Second,
WriteTimeout: 1 * time.Second,
PoolSize: 6500,
PoolTimeout: 30 * time.Second}
redisCache := redis.NewClusterClient(&redisConfig)
_, err = redisCache.Ping().Result()
if err != nil {
t.Errorf("fatal error Not able to connect using redis client: %s", err)
log.Fatalf("fatal error Not able to connect using redis client: %s", err)
}
}
Related
I am running the webserver in Go on an EC2:
package main
import (
"fmt"
"net/http"
"github.com/gorilla/websocket"
)
var upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(r *http.Request) bool { return true },
}
func echo(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
fmt.Println(err)
return
}
defer conn.Close()
for {
msgType, msg, err := conn.ReadMessage()
if err != nil {
break
}
conn.WriteMessage(msgType, msg)
}
}
func main() {
fmt.Println("listening")
http.HandleFunc("/echo", echo)
http.ListenAndServe(":8918", nil)
}
And sending a request to it using a Python script:
import websocket
ws = websocket.WebSocket()
ws.connect("ws://myEIP:8918/echo") // redacted elastic IP
ws.send("Hello, WebSocket!")
ws.close()
However this blocks for ~30 seconds before raising an exception:
Exception has occurred: TimeoutError
[Errno 60] Operation timed out
I have my NACL set up to allow everything on port 8918, on both inbound at outbound.
I have the security group set up to allow everything on port 8918 - also outbound, but I believe security group access is stateful, so outbound isn't necessary.
I have created an AWS EKS cluster. In order to obtain its kubeconfig, I usually run aws eks update-kubeconfig --name cluster-name --region us-west-2 using a shell.
However, I now wish to obtain the kubeconfig in Go without having to run anything in the shell (the goal being to create and then manipulate an EKS cluster in a Go test). I am able to describe an EKS cluster using this code:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/eks"
)
func main() {
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
}))
eksSvc := eks.New(sess, aws.NewConfig().WithRegion("us-west-2"))
clusterOutput, err := eksSvc.DescribeCluster(&eks.DescribeClusterInput{
Name: aws.String("cluster-name"),
})
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", clusterOutput)
}
After that, I currently have no idea about how could I get the kubeconfig of that cluster in order to then use it with the Go client for Kubernetes without having to use aws eks separately.
I have checked the AWS documentation, AWS CLI codebase, and eksctl codebase with no luck so far. The connection to an EKS cluster is only documented in this webpage and it uses a shell: https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection/
Any ideas?
The general flow goes something like this:
DescribeCluster (as you have done) and extract some necessary data
Using the necessary data, get a token using aws-iam-authenticator's package token
Using that token, create a Kubernetes clientset with the help of client-go.
package main
import (
"encoding/base64"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/eks"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"sigs.k8s.io/aws-iam-authenticator/pkg/token"
)
func newClientset(cluster *eks.Cluster) (*kubernetes.Clientset, error) {
log.Printf("%+v", cluster)
gen, err := token.NewGenerator(true, false)
if err != nil {
return nil, err
}
opts := &token.GetTokenOptions{
ClusterID: aws.StringValue(cluster.Name),
}
tok, err := gen.GetWithOptions(opts)
if err != nil {
return nil, err
}
ca, err := base64.StdEncoding.DecodeString(aws.StringValue(cluster.CertificateAuthority.Data))
if err != nil {
return nil, err
}
clientset, err := kubernetes.NewForConfig(
&rest.Config{
Host: aws.StringValue(cluster.Endpoint),
BearerToken: tok.Token,
TLSClientConfig: rest.TLSClientConfig{
CAData: ca,
},
},
)
if err != nil {
return nil, err
}
return clientset, nil
}
func main() {
name := "wonderful-outfit-1583362361"
region := "us-east-2"
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String(region),
}))
eksSvc := eks.New(sess)
input := &eks.DescribeClusterInput{
Name: aws.String(name),
}
result, err := eksSvc.DescribeCluster(input)
if err != nil {
log.Fatalf("Error calling DescribeCluster: %v", err)
}
clientset, err := newClientset(result.Cluster)
if err != nil {
log.Fatalf("Error creating clientset: %v", err)
}
nodes, err := clientset.CoreV1().Nodes().List(metav1.ListOptions{})
if err != nil {
log.Fatalf("Error getting EKS nodes: %v", err)
}
log.Printf("There are %d nodes associated with cluster %s", len(nodes.Items), name)
}
Here's my go.mod for versions:
module github.com/swoldemi/sandbox
go 1.14
require (
github.com/aws/aws-sdk-go v1.29.19
k8s.io/apimachinery v0.0.0-20190612125636-6a5db36e93ad
k8s.io/client-go v0.0.0-20190425172711-65184652c889
sigs.k8s.io/aws-iam-authenticator v0.5.0
)
Adding to Simon's great answer, you may want to also pass the session in the GetTokenOptions struct, like so:
opts := &token.GetTokenOptions{
ClusterID: aws.StringValue(cluster.Name),
Session: sess,
}
tok, err := gen.GetWithOptions(opts)
Otherwise the gen.GetWithOptions(opts) call will try to read your AWS credentials from local sources (e.g. ~/.aws/credentials) and may fail with this error:
NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I am new to AWS and Golang, and I am trying to create a lambda function, which will trigger AWS Athena query and email the result using AWS SES service. Even after searching for an hour, I couldn't find a working example of lambda function (in Golang) to perform a query on Athena and getting the output of the query.
While searching, I found code for the same in Java, Python and Node Js, but not in Golang.
Even the Go-SDK page redirects to Java example. But unfortunately, I don't even understand Java.
I have also looked into this AWS SDK for Go API Reference page. But I don't understand what is the flow of the program and which operation to select.
I have tried to create the program for this, this may be completely wrong, and I don't know what to do next. Below is the code -
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/athena"
)
func main() {
// Create a new session in the us-west-2 region.
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
// Create an Athena session.
client := athena.New(sess)
// Example sending a request using the StartQueryExecutionRequest method.
query := "SELECT * FROM table1 ;"
params := query
req, resp := client.StartQueryExecutionRequest(params)
err1 := req.Send()
if err1 == nil { // resp is now filled
fmt.Println(resp)
}
}
Appreciate if someone can help me to perform an Athena query and to get its result in Golang(Preferably) or can share some resource. Once I get it, I can then send an email using AWS SES.
Use this to get started.
// run as: go run main.go
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "textqldb.textqltable"
const outputBucket = "s3://bucket-name-here/"
func main() {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
return
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
query := "select * from " + table
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("query error: %v\n", err)
return
}
fmt.Println(resp)
}
#Everton's code is executing a query on Athena, and its responses are getting saved on S3 bucket and not getting returned. So, I have added the code to execute the Athena query and get the response back. Hope this may help others.
// run as: go run main.go
package main
import (
"context"
"fmt"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "<Database_Name>.<Table_Name>"
const outputBucket = "s3://bucket-name-here/"
// Execute the query and return the query ID
func executeQuery(query string) *string {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
fmt.Println("Response is: ", resp, " Error is:", err)
if err != nil {
fmt.Printf("Query Error: %v\n", err)
}
fmt.Println("Query Execution Response ID:", resp.QueryExecutionId)
return resp.QueryExecutionId
}
// Takes queryId as input and returns its response
func getQueryResults(QueryID *string) (*athena.GetQueryResultsResponse, error) {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
panic("config error")
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
params1 := &athena.GetQueryResultsInput{
QueryExecutionId: QueryID,
}
req := client.GetQueryResultsRequest(params1)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("Query Response Error: %v\n", err)
return nil, err
}
return resp, nil
}
func main() {
query := "select * from " + table
// Execute an Athena Query
QueryID := executeQuery(query)
// Get the response of the query
// Wait for some time for query completion
time.Sleep(15 * time.Second) // Otherwise create a loop and try for every x seconds
Resp, err := getQueryResults(QueryID)
if err != nil {
fmt.Printf("Error getting Query Response: %v\n", err)
} else {
fmt.Println(" \nRows:", Resp.ResultSet.Rows)
}
}
I am trying to make a simple demo using the fabric-sdk-go. I wonder anybod know why the code below ("client.Query") returns the error "failed to create transactor: Channel_Cfg_Cache - cache is closed"?
func initSdkClient() (*channel.Client){
sdk, err := fabsdk.New(config.FromFile("config.yaml"))
if err != nil {
logger.Fatalf("Failed to create new SDK: %s", err)
}
defer sdk.Close()
//prepare channel client context using client context
clientChannelContext := sdk.ChannelContext(channelID, fabsdk.WithUser("User1"), fabsdk.WithOrg(orgName))
// Channel client is used to query and execute transactions (Org1 is default org)
client, err := channel.New(clientChannelContext)
if err != nil {
logger.Fatalf("Failed to create new channel client: %s", err)
}
return client
}
func queryCC(client *channel.Client, targetEndpoints ...string) []byte {
response, err := client.Query(channel.Request{ChaincodeID: ccID, Fcn: "invoke", Args: defaultQueryArgs},
channel.WithRetry(retry.DefaultChannelOpts),
channel.WithTargetEndpoints(targetEndpoints...),
)
if err != nil {
***logger.Fatalf("Failed to query funds: %s", err)*** // error: failed to create transactor: Channel_Cfg_Cache - cache is closed
}
return response.Payload
}
func main() {
client := initSdkClient()
existingValue := queryCC(client)
logger.Info(existingValue)
logger.Info("hello, world\n")
}
Probably because you're closing the SDK instance at the end of initSdkClient function. Rather close the SDK at the end of main function.
Do something like
type Setup struct {
sdk *fabsdk.FabricSDK
client *channel.Client
}
func (setup *Setup) initSdkClient() *channel.Client {
sdk, err := fabsdk.New(config.FromFile("config.yaml"))
if err != nil {
fmt.Errorf("Failed to create new SDK: %s", err)
}
setup.sdk = sdk
//prepare channel client context using client context
clientChannelContext := setup.sdk.ChannelContext(channelID, fabsdk.WithUser("User1"), fabsdk.WithOrg(orgName))
// Channel client is used to query and execute transactions (Org1 is default org)
client, err := channel.New(clientChannelContext)
if err != nil {
fmt.Errorf("Failed to create new channel client: %s", err)
}
setup.client = client
return client
}
func (setup *Setup) queryCC(client *channel.Client, targetEndpoints ...string) []byte {
response, err := setup.client.Query(channel.Request{ChaincodeID: ccID, Fcn: "invoke", Args: defaultQueryArgs},
channel.WithRetry(retry.DefaultChannelOpts),
channel.WithTargetEndpoints(targetEndpoints...),
)
if err != nil {
fmt.Errorf("Failed to query funds: %s", err)
}
return response.Payload
}
func main() {
var setup Setup
client := setup.initSdkClient()
defer setup.sdk.Close()
existingValue := setup.queryCC(client)
}
I am trying to create websockets using golang with gorilla on aws (without docker) with load balancer, I have a certificate connected to the load balancer.
I managed to make the code work with http, but when I try to use the code over https it does not work.
What am I doing wrong?
When moving from http to https I changed the client request to wss instead of ws, and added the certificate to the load balancer.
Here is my code.
This is my main.go
http.HandleFunc("/wss", serveWs)
fmt.Printf("Listening on port %s\n\n", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
fmt.Printf(err.Error())
}
This is the serveWs:
func serveWs(w http.ResponseWriter, r *http.Request) {
upgrader.CheckOrigin = func(r *http.Request) bool {
// allow all connections by default
return true
}
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
fmt.Println(err)
if _, ok := err.(websocket.HandshakeError); !ok {
log.Println(err)
}
return
}
var lastMod time.Time
if n, err := strconv.ParseInt(r.FormValue("lastMod"), 16, 64); err != nil {
lastMod = time.Unix(0, n)
}
go writer(ws, lastMod, w, r)
reader(ws)
}
This is the request:
var conn = new WebSocket("wss://https://www.weldpad.com/wss?"
When I send a request from the client i get the folloing error:
failed: Error during WebSocket handshake: Unexpected response code: 400
Looking at the server log, I see.
"could not find upgrade header with token 'websocket"
This is my load balancer configuration:
Load Balancing Configuration
I believe it should be wss://www.weldpad.com/wss? not wss://https://www.weldpad.com/wss?