Unable to obtain kubeconfig of an AWS EKS cluster in Go code - amazon-web-services

I have created an AWS EKS cluster. In order to obtain its kubeconfig, I usually run aws eks update-kubeconfig --name cluster-name --region us-west-2 using a shell.
However, I now wish to obtain the kubeconfig in Go without having to run anything in the shell (the goal being to create and then manipulate an EKS cluster in a Go test). I am able to describe an EKS cluster using this code:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/eks"
)
func main() {
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
}))
eksSvc := eks.New(sess, aws.NewConfig().WithRegion("us-west-2"))
clusterOutput, err := eksSvc.DescribeCluster(&eks.DescribeClusterInput{
Name: aws.String("cluster-name"),
})
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", clusterOutput)
}
After that, I currently have no idea about how could I get the kubeconfig of that cluster in order to then use it with the Go client for Kubernetes without having to use aws eks separately.
I have checked the AWS documentation, AWS CLI codebase, and eksctl codebase with no luck so far. The connection to an EKS cluster is only documented in this webpage and it uses a shell: https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection/
Any ideas?

The general flow goes something like this:
DescribeCluster (as you have done) and extract some necessary data
Using the necessary data, get a token using aws-iam-authenticator's package token
Using that token, create a Kubernetes clientset with the help of client-go.
package main
import (
"encoding/base64"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/eks"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"sigs.k8s.io/aws-iam-authenticator/pkg/token"
)
func newClientset(cluster *eks.Cluster) (*kubernetes.Clientset, error) {
log.Printf("%+v", cluster)
gen, err := token.NewGenerator(true, false)
if err != nil {
return nil, err
}
opts := &token.GetTokenOptions{
ClusterID: aws.StringValue(cluster.Name),
}
tok, err := gen.GetWithOptions(opts)
if err != nil {
return nil, err
}
ca, err := base64.StdEncoding.DecodeString(aws.StringValue(cluster.CertificateAuthority.Data))
if err != nil {
return nil, err
}
clientset, err := kubernetes.NewForConfig(
&rest.Config{
Host: aws.StringValue(cluster.Endpoint),
BearerToken: tok.Token,
TLSClientConfig: rest.TLSClientConfig{
CAData: ca,
},
},
)
if err != nil {
return nil, err
}
return clientset, nil
}
func main() {
name := "wonderful-outfit-1583362361"
region := "us-east-2"
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String(region),
}))
eksSvc := eks.New(sess)
input := &eks.DescribeClusterInput{
Name: aws.String(name),
}
result, err := eksSvc.DescribeCluster(input)
if err != nil {
log.Fatalf("Error calling DescribeCluster: %v", err)
}
clientset, err := newClientset(result.Cluster)
if err != nil {
log.Fatalf("Error creating clientset: %v", err)
}
nodes, err := clientset.CoreV1().Nodes().List(metav1.ListOptions{})
if err != nil {
log.Fatalf("Error getting EKS nodes: %v", err)
}
log.Printf("There are %d nodes associated with cluster %s", len(nodes.Items), name)
}
Here's my go.mod for versions:
module github.com/swoldemi/sandbox
go 1.14
require (
github.com/aws/aws-sdk-go v1.29.19
k8s.io/apimachinery v0.0.0-20190612125636-6a5db36e93ad
k8s.io/client-go v0.0.0-20190425172711-65184652c889
sigs.k8s.io/aws-iam-authenticator v0.5.0
)

Adding to Simon's great answer, you may want to also pass the session in the GetTokenOptions struct, like so:
opts := &token.GetTokenOptions{
ClusterID: aws.StringValue(cluster.Name),
Session: sess,
}
tok, err := gen.GetWithOptions(opts)
Otherwise the gen.GetWithOptions(opts) call will try to read your AWS credentials from local sources (e.g. ~/.aws/credentials) and may fail with this error:
NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Related

How to fix error with AWS Lambda handler, DynamoDB Put req?

Trying to create a Lambda to interact with my DynamoDB.
This specific Lambda is to put/write an item to the DB:
package main
import (
"fmt"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
type Item struct {
Email string `json:"email"`
Password string `json:"password"`
Rname string `json:"rname"`
}
func Put() error {
// Create a session - London Region
session, err := session.NewSession(&aws.Config{
Region: aws.String("eu-west-2")},
)
if err != nil {
fmt.Println(err)
}
svc := dynamodb.New(session)
// Create instance of Item Struct
item := Item{
Email: "123#mail.com",
Password: "12345678",
Rname: "abcde",
}
// Marshall Item
av, err := dynamodbattribute.MarshalMap(item)
if err != nil {
fmt.Println("Got error marshalling map:")
fmt.Println(err)
}
// Create Item
input := &dynamodb.PutItemInput{
Item: av,
TableName: aws.String("accountsTable"),
}
_, err = svc.PutItem(input)
if err != nil {
fmt.Println("Got error calling PutItem:")
fmt.Println(err)
}
return err
}
func main() {
lambda.Start(Put())
}
However getting the error:
{
"errorMessage": "handler is nil",
"errorType": "errorString"
}
I have changed the handler in run time settings to main too so don't think that would be the issue.
Building with:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -a main.go
and putting the zip of the executable into AWS via console (no IAC's)
Any help would be greatly appreciated to resolve this error! Thanks.
You need to pass function handle not function result to lambda.Start
Please update your main function with👇
func main() {
lambda.Start(Put)
}

k8s how to update specific value in secret

I need to update a secret with specific value,(the secret contain additional data) my question is how I can update just the value and not all the secret data (I don't want to override the existing data). I mean if the secret have additional values I don’t want to override them just the entry foo
updSec := v1.Secret{
TypeMeta: metav1.TypeMeta{},
ObjectMeta: metav1.ObjectMeta{
Name: "d-values",
Namespace: "terv”,
},
Immutable: nil,
Data: nil,
StringData: nil,
Type: "Opaque",
}
updSec.Data[“foo”] = newVal
if err := r.Client.Update(ctx, &updSec); err != nil {
return ctrl.Result{}, err
}
The issue is that the secret is already exist and here im creating new object and not sure how to do it right ...I need for secret that called d-values just update the newVal for key foo
update
when trying the code in the answer after I run the
patch, err := yaml.Marshal(updSec)
the data looks like following
and the patch are failed with error, any idea if its related ?
if I try with the c.Client.Update it works but not with Patch but the Patch is the right way as if I've ties before is should keep them..
I don't think you can update a single key using the Update method, but you can certainly do that using Patch instead. Here's an example that uses a StrategicMergePatch; it will replace the key val2 in a secret with the value newval:
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"path/filepath"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
)
func main() {
var kubeconfig *string
var namespace *string
var secretname *string
namespace = flag.String("namespace", "", "namespace of secret")
secretname = flag.String("name", "", "name of secret")
if home := homedir.HomeDir(); home != "" {
kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
} else {
kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
}
flag.Parse()
if *namespace == "" {
panic(fmt.Errorf("you must specify a namespace"))
}
if *secretname == "" {
panic(fmt.Errorf("you must specify a secret name"))
}
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
panic(err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
panic(err)
}
secretClient := clientset.CoreV1().Secrets(*namespace)
ctx := context.TODO()
updSec := v1.Secret{
Data: map[string][]byte{
"val2": []byte("newval"),
},
}
payloadBytes, err := json.Marshal(updSec)
if err != nil {
panic(err)
}
if _, err = secretClient.Patch(ctx, *secretname,
types.StrategicMergePatchType, payloadBytes, metav1.PatchOptions{}); err != nil {
panic(err)
}
// Fetch updated secret
sec, err := secretClient.Get(ctx, *secretname, metav1.GetOptions{})
if err != nil {
panic(err)
}
secJson, err := json.MarshalIndent(sec, "", " ")
if err != nil {
panic(err)
}
fmt.Print(string(secJson))
}
For example, if I create a secret like this:
kubectl create secret generic \
--from-literal val1=key1 \
--from-literal val2=key2 example
And then run the above code like this:
go run main.go -namespace default -name example
The code will output the update secret. Looking at the data section, we see:
"data": {
"val1": "a2V5MQ==",
"val2": "bmV3dmFs"
},
And if we decode val2 we see:
$ kubectl get secret example -o json | jq '.data.val2|#base64d'
"newval"
Using the Operator SDK
If you're working with the Operator SDK, you can use Update if you're first reading the existing value, like this:
// Read the existing secret
secret := &corev1.Secret{}
if err := r.Get(ctx, req.NamespacedName, secret); err != nil {
panic(err)
}
// Check if it needs to be modified
val, ok := secret.Data["val2"]
// If yes, update the secret with a new value and then write
// the entire object back with Update
if !ok || !bytes.Equal(val, []byte("val2")) {
ctxlog.Info("needs update", "secret", secret)
secret.Data["val2"] = []byte("newval")
if err := r.Update(ctx, secret); err != nil {
panic(err)
}
}
You can use the Patch method if you only want to submit a partial update:
if !ok || !bytes.Equal(val, []byte("val2")) {
ctxlog.Info("needs update", "secret", secret)
newVal := corev1.Secret{
Data: map[string][]byte{
"val2": []byte("newval"),
},
}
patch, err := json.Marshal(newVal)
if err != nil {
panic(err)
}
if err := r.Client.Patch(ctx, secret, client.RawPatch(types.StrategicMergePatchType, patch)); err != nil {
panic(err)
}
}
This is pretty much identical to the earlier example. There are examples of using the client.Patch method in the docs, but I'll be honest, I don't find the example very clear.

Is there an Amazon API to get Nice Region names rather than us-west1?

I'm using the following API to retrieve a list of amazon regions.
However, it basically returns the regions as "us-west1, us-west2" etc. Is there a way to get the region name from the API with output such as "US West (N. California)", "US West (Oregon)" ?
// Get a list of regions from our default region
svc := ec2.NewFromConfig(cfg)
result, err := svc.DescribeRegions(context.TODO(), &ec2.DescribeRegionsInput{})
if err != nil {
return nil, err
}
var regions []portaineree.Pair
for _, region := range result.Regions {
fmt.Println("region.Name=", *region.RegionName)
// do something with region...
}
You can use the SSM Agent to both get the list of regions, and pull out the long name for each region:
package main
import (
"log"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ssm"
)
func main() {
// Build a AWS SSM Agent
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
// We're requesting global data, the region doesn't matter
svc := ssm.New(sess, &aws.Config{Region: aws.String("us-east-1")})
var nextToken *string
for {
// Request all regions, paginating the results if needed
var input = &ssm.GetParametersByPathInput{
Path: aws.String("/aws/service/global-infrastructure/regions"),
NextToken: nextToken,
}
var output, err = svc.GetParametersByPath(input)
if err != nil {
log.Fatal(err)
}
// For each region, get the "longName" for the region
for _, element := range output.Parameters {
region := (*element.Name)[strings.LastIndex(*element.Name, "/")+1:]
var regionInfo, err = svc.GetParameter(&ssm.GetParameterInput{
Name: aws.String("/aws/service/global-infrastructure/regions/" + region + "/longName"),
})
if err != nil {
log.Fatal(err)
}
regionDesc := *regionInfo.Parameter.Value
// Just output the region and region description
log.Println(region, " = ", regionDesc)
}
// Pull in the next page of regions if needed
nextToken = output.NextToken
if nextToken == nil {
break
}
}
}
This can be done pretty easily through CLI commands if that's an option
region=us-east-1
aws ssm get-parameter --name /aws/service/global-infrastructure/regions/$region/longName --query "Parameter.Value" --output text```

Unable to perform query on AWS athena using Golang SDK

I am new to AWS and Golang, and I am trying to create a lambda function, which will trigger AWS Athena query and email the result using AWS SES service. Even after searching for an hour, I couldn't find a working example of lambda function (in Golang) to perform a query on Athena and getting the output of the query.
While searching, I found code for the same in Java, Python and Node Js, but not in Golang.
Even the Go-SDK page redirects to Java example. But unfortunately, I don't even understand Java.
I have also looked into this AWS SDK for Go API Reference page. But I don't understand what is the flow of the program and which operation to select.
I have tried to create the program for this, this may be completely wrong, and I don't know what to do next. Below is the code -
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/athena"
)
func main() {
// Create a new session in the us-west-2 region.
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
// Create an Athena session.
client := athena.New(sess)
// Example sending a request using the StartQueryExecutionRequest method.
query := "SELECT * FROM table1 ;"
params := query
req, resp := client.StartQueryExecutionRequest(params)
err1 := req.Send()
if err1 == nil { // resp is now filled
fmt.Println(resp)
}
}
Appreciate if someone can help me to perform an Athena query and to get its result in Golang(Preferably) or can share some resource. Once I get it, I can then send an email using AWS SES.
Use this to get started.
// run as: go run main.go
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "textqldb.textqltable"
const outputBucket = "s3://bucket-name-here/"
func main() {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
return
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
query := "select * from " + table
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("query error: %v\n", err)
return
}
fmt.Println(resp)
}
#Everton's code is executing a query on Athena, and its responses are getting saved on S3 bucket and not getting returned. So, I have added the code to execute the Athena query and get the response back. Hope this may help others.
// run as: go run main.go
package main
import (
"context"
"fmt"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/athena"
)
const table = "<Database_Name>.<Table_Name>"
const outputBucket = "s3://bucket-name-here/"
// Execute the query and return the query ID
func executeQuery(query string) *string {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
fmt.Printf("config error: %v\n", err)
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
resultConf := &athena.ResultConfiguration{
OutputLocation: aws.String(outputBucket),
}
params := &athena.StartQueryExecutionInput{
QueryString: aws.String(query),
ResultConfiguration: resultConf,
}
req := client.StartQueryExecutionRequest(params)
resp, err := req.Send(context.TODO())
fmt.Println("Response is: ", resp, " Error is:", err)
if err != nil {
fmt.Printf("Query Error: %v\n", err)
}
fmt.Println("Query Execution Response ID:", resp.QueryExecutionId)
return resp.QueryExecutionId
}
// Takes queryId as input and returns its response
func getQueryResults(QueryID *string) (*athena.GetQueryResultsResponse, error) {
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
panic("config error")
}
cfg.Region = endpoints.UsEast2RegionID
client := athena.New(cfg)
params1 := &athena.GetQueryResultsInput{
QueryExecutionId: QueryID,
}
req := client.GetQueryResultsRequest(params1)
resp, err := req.Send(context.TODO())
if err != nil {
fmt.Printf("Query Response Error: %v\n", err)
return nil, err
}
return resp, nil
}
func main() {
query := "select * from " + table
// Execute an Athena Query
QueryID := executeQuery(query)
// Get the response of the query
// Wait for some time for query completion
time.Sleep(15 * time.Second) // Otherwise create a loop and try for every x seconds
Resp, err := getQueryResults(QueryID)
if err != nil {
fmt.Printf("Error getting Query Response: %v\n", err)
} else {
fmt.Println(" \nRows:", Resp.ResultSet.Rows)
}
}

storagegateway.ListGateways() returning "Invalid resource" with status code: 400

I am having trouble using the go sdk to list all of the storage gateways in a region using the following code:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/storagegateway"
)
func newSess() (sess *session.Session) {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-2"),
})
if err != nil {
fmt.Printf("Error creating session: %s\n", err)
return
}
return sess
}
func main() {
sgw := storagegateway.New(newSess())
input := &storagegateway.ListGatewaysInput{
Limit: aws.Int64(5),
Marker: aws.String("1"),
}
result, err := sgw.ListGateways(input)
if err != nil {
fmt.Printf("Error listing gateways %s\n", err)
return
}
fmt.Println(result)
}
My environment is the following:
Go version: 1.9.2 darwin/amd64
Operating system: macOS 10.13.2
aws-sdk-go: v1.12.63
The SDK is using the default profile specified in ~/.aws/config as it should and the credentials are correct as well. Furthermore, I am able to list the gateways using the awscli alone.
This could very well be something simple that I have overlooked as I am no expert. That said, any help would be greatly appreciated.
The example code seems to work as expected for me by simply removing the Marker: aws.String("1") line.