When using the Kubernetes Fake Client to write unit tests, I noticed that it fails to create two identical objects which have their ObjectMeta.GenerateName field set to some string. A real cluster accepts this specification and generates a unique name for each object.
Running the following test code:
package main
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes/fake"
)
func TestFake(t *testing.T) {
ctx := context.Background()
client := fake.NewSimpleClientset()
_, err := client.CoreV1().Secrets("default").Create(ctx, &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
GenerateName: "generated",
},
StringData: map[string]string{"foo": "bar"},
}, metav1.CreateOptions{})
assert.NoError(t, err)
_, err = client.CoreV1().Secrets("default").Create(ctx, &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
GenerateName: "generated",
},
StringData: map[string]string{"foo": "bar"},
}, metav1.CreateOptions{})
assert.NoError(t, err)
}
fails with
--- FAIL: TestFake (0.00s)
/Users/mihaitodor/Projects/kubernetes/main_test.go:44:
Error Trace: main_test.go:44
Error: Received unexpected error:
secrets "" already exists
Test: TestFake
FAIL
FAIL kubernetes 0.401s
FAIL
According to this GitHub issue comment:
the fake clientset doesn't attempt to duplicate server-side behavior
like validation, name generation, uid assignment, etc. if you want to
test things like that, you can add reactors to mock that behavior.
To add the required reactor, we can insert the following code before creating the corev1.Secret objects:
client.PrependReactor(
"create", "*",
func(action k8sTesting.Action) (handled bool, ret runtime.Object, err error) {
ret = action.(k8sTesting.CreateAction).GetObject()
meta, ok := ret.(metav1.Object)
if !ok {
return
}
if meta.GetName() == "" && meta.GetGenerateName() != "" {
meta.SetName(names.SimpleNameGenerator.GenerateName(meta.GetGenerateName()))
}
return
},
)
There are a few gotchas in there:
The Clientset contains an embedded Fake structure which has the PrependReactor method we need to call for this use case (there are a few others). This code here is invoked when creating such objects.
The PrependReactor method has 3 parameters: verb, resource and reaction. For verb, resource, I couldn't find any named constants, so, in this case, "create" and "secrets" (strange that it's not "secret") seem to be the correct values for them if we want to be super-specific, but setting resource to "*" should be acceptable in this case.
The reaction parameter is of type ReactionFunc, which takes an Action as a parameter and returns handled, ret and err. After some digging, I noticed that the action parameter will be cast to CreateAction, which has the GetObject() method that returns a runtime.Object instance, which can be cast to metav1.Object. This interface allows us to get and set the various metadata fields of the underlying object. After setting the object Name field as needed, we have to return handled = false, ret = mutatedObject and err = nil to instruct the calling code to execute the remaining reactors.
Digging through the apiserver code, I noticed that the ObjectMeta.Name field is generated from the ObjectMeta.GenerateName field using the names.SimpleNameGenerator.GenerateName utility.
Related
hope someone can shed some light on this issue I've been trying to wrap my head around.
I've got this table in Dynamo, let's call it people and in this table, I've got the attributes of id as our partition key, name, lastName, and status.
I'd like to be able to just update either a single attribute or all of them save for the ID.
Now, this is how I've gone about it. I've created the following struct:
type PersonUpdate struct {
FirstName string `json:"firstName,omitempty"`
LastName string `json:"lastName,omitempty"`
Status string `json:"status,omitempty"`
}
And the request coming from the server would be to just update the person's last name, so our request body would look as follows:
{
"lastName": "bob"
}
After we bind our request to our struct the object that would be sent to dynamo looks as as such:
{
"firstName": "",
"lastName": "bob",
"status": "",
}
Now when it's time to write to dynamo as you can see, only one attribute should be updated while the rest which are empty/null should be ignored.
The code written to perform this action can be condensed to the following actions:
// Marshal our object
_, err := dynamodbattribute.MarshalMap(person)
if err != nil{
fmt.Println("some error marshaling")
}
// Create our update input
input := &dynamodb.UpdateItemInput{
key: map[string]*dynamodb.AttributeValue{
"id":{
S: aws.String("1234"),
},
},
TableName: aws.String("people"),
ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
":fn": {
S: aws.String(person.FirstName),
},
":ln":{
S: aws.String(person.LastName),
},
":st":{
S: aws.String(person.Status),
},
},
UpdateExpression: aws.String("set firstName = :fn, lastName = :ln, status = :st"),
ReturnValues: aws.String("UPDATED_NEW"),
}
// Send update request to Dynamo
_, err := service.UpdateItem(input)
Now, the update goes through with no issues, the problem is that the values for firstName and status which are null are getting passed as well. I've tried to go through their documentation but left a bit more confused. I know for a fact the Java SDK allows you to pass a flag called UPDATE_SKIP_NULL_ATTRIBUTES which allows you to skip those empty values and only update those which have data. I don't know what would be the equivalent in Go, any assistance/guidance would be great.
Update:
Tried to use the following model:
type UserModel struct {
FirstName string `json:"firstName,omitempty" dynamodbav:"firstName,omitempty"`
LastName string `json:"lastName,omitempty" dynamodbav:"lastName,omitempty"`
}
following the suggestion given by #jarmod and #fedonev. Logically and using the docs it makes sense why this should work, unfortunately, it didn't
Decided to switch the SDK from V1 to V2 and see if maybe updating it would help, again though I'm in the same hole. This is what my update function looks like.
update :=expression.Set(expression.Name("firstName"),expression.Value(user.FirstName))
update.Set(expression.Name("lastName"), expression.Value(user.LastName))
expr, err := expression.NewBuilder().WithUpdate(update).Build()
if err != nil {
"log error here..."
}
_, err = svc.UpdateItem(context.TODO(), &dynamodb.UpdateItemInput{
TableName: aws.String("people"),
Key: map[string]types.AttributeValue{"id": id},
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
UpdateExpression: expr.Update(),
ReturnValues: types.ReturnValueUpdatedNew,
})
if err != nil {
"log error here..."
}
I ended up solving my problem by writing a query-and-create function, which pretty much queries by the ID that we'd like to update takes in the JSON payload, and make a diff of the query against what we need to update and any empty fields get replaced by the query results. So far it solves my problem but I still would like to know how to go about doing an update using its intended function.
As #jarmod's comment says, apply the dynamodbav:",omitempty" tag to skip zero-value fields when marshaling the struct:
type PersonUpdate struct {
FirstName string `json:"firstName,omitempty" dynamodbav:",omitempty"`
LastName string `json:"lastName,omitempty" dynamodbav:",omitempty"`
Status string `json:"status,omitempty" dynamodbav:",omitempty"`
}
[Edit: add usage] MarshalMap will now ignore zero-value fields, respecting the tags. Iterate over the map to construct the update expression:
av, _ := attributevalue.MarshalMap(person)
update := expression.UpdateBuilder{}
for k, v := range av {
update = update.Set(expression.Name(k), expression.Value(v))
}
Build the expression and pass its outputs to UpdateItem, as in the OP.
I'm putting and reading files to S3 using the AWS golang v2 SDK. Locally I am using local stack and thus need to set the param S3ForcePathStyle. But, I can't find where to set this parameter in the config.
This is what my config looks like:
conf, err = config.LoadDefaultConfig(
context.TODO(),
config.WithRegion("us-east-1"),
config.WithEndpointResolver(
aws.EndpointResolverFunc(func(service, region string) (aws.Endpoint, error) {
return aws.Endpoint{
PartitionID: "aws",
URL: "http://localstack:4566",
SigningRegion: "us-east-1",
}, nil
}),
),
)
Where can I pass in S3ForcePathStyle = true?
Seems I was looking in the wrong place. The documentation here explains that in aws-sdk-go-v2 they moved the service-specific configuration flags to the individual service client option types. Ironically to improve discoverability.
I should set the UsePathStyle like this:
client := s3.NewFromConfig(conf, func(o *s3.Options) {
o.UsePathStyle = true
})
AWS Config has a set of Managed Rules and I am trying to use the Golang AWS SDK to use the DescribeConfigRules API to retrieve the list of AWS Config Managed Rule Names and other details.
It seems like every request receives a response of 25 rules and a NextToken for the next set of results. What I am having trouble understanding is how do I use this NextToken to retrieve the next set of results?
Here is what I have so far.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
func main() {
//Create an aws session
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-aws-profile")})
// Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
configrulesoutput, err := configsvc.DescribeConfigRules(rules)
if err != nil {
log.Fatal(err)
}
for _, rule := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", *rule.ConfigRuleName)
}
}
The above code successfully prints the first 25 rules received in the response. However I am not sure how to use the NextToken received in the response to get the next set of results.
Sample Response.
ConfigRules: [
{
ConfigRuleArn: "ConfigRuleARN",
ConfigRuleId: "config-rule-ppwclr",
ConfigRuleName: "cloudtrail-enabled",
ConfigRuleState: "ACTIVE",
Description: "Checks whether AWS CloudTrail is enabled in your AWS account. Optionally, you can specify which S3 bucket, SNS topic, and Amazon CloudWatch Logs ARN to use.",
InputParameters: "{}",
MaximumExecutionFrequency: "TwentyFour_Hours",
Source: {
Owner: "AWS",
SourceIdentifier: "CLOUD_TRAIL_ENABLED"
}
},
{ Rule 2 }, ....{ Rule 25}
],
NextToken: "nexttoken"
}
Code extracts the rulenames from the response and output is as below.
Rule: cloudtrail-enabled
Rule: restricted-ssh
Rule: securityhub-access-keys-rotated
Rule: securityhub-autoscaling-group-elb-healthcheck-required
Rule: securityhub-cloud-trail-cloud-watch-logs-enabled
Rule: securityhub-cloud-trail-encryption-enabled
Rule: securityhub-cloud-trail-log-file-validation-enabled
Rule: securityhub-cloudtrail-enabled
Rule: securityhub-cmk-backing-key-rotation-enabled
Rule: securityhub-codebuild-project-envvar-awscred-check
Rule: securityhub-codebuild-project-source-repo-url-check
Rule: securityhub-ebs-snapshot-public-restorable-check
Rule: securityhub-ec2-managedinstance-patch-compliance
Rule: securityhub-ec2-security-group-attached-to-eni
Rule: securityhub-eip-attached
Rule: securityhub-elasticsearch-encrypted-at-rest
Rule: securityhub-elasticsearch-in-vpc-only
Rule: securityhub-iam-password-policy-ensure-expires
Rule: securityhub-iam-password-policy-lowercase-letter-check
Rule: securityhub-iam-password-policy-minimum-length-check
Rule: securityhub-iam-password-policy-number-check
Rule: securityhub-iam-password-policy-prevent-reuse-check
Rule: securityhub-iam-password-policy-symbol-check
Rule: securityhub-iam-password-policy-uppercase-letter-check
Rule: securityhub-iam-policy-no-statements-with-admin-access
End Goal: Using golang AWS SDK, extract the AWS Config Managed Rule details and put it in an excel format using Excelize to review which AWS Config rules we want enabled.
Thanks for your help in advance.
---New based on #Adrian's comment and doc reference---
As per doc
type DescribeConfigRulesInput struct {
// The names of the AWS Config rules for which you want details. If you do not
// specify any names, AWS Config returns details for all your rules.
ConfigRuleNames []*string `type:"list"`
// The nextToken string returned on a previous page that you use to get the
// next page of results in a paginated response.
NextToken *string `type:"string"`
// contains filtered or unexported fields }
So here is what I am trying. Specifying nil should give me back all rules. nextToken is blank string for the first call.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
nextToken := ""
rules.SetNextToken(nextToken)
getConfigRulesFunc(configsvc, rules)
//getConfigRulesFunc function
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for i, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
}
if *configrulesoutput.NextToken != "" {
ruleset := (*configservice.DescribeConfigRulesInput)(nil)
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Above code compiles fine but here the runtime error I believe because of nil.
configsvc type: *configservice.ConfigService
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13c7ed2]
goroutine 1 [running]:
github.com/aws/aws-sdk-go/service/configservice.(*DescribeConfigRulesInput).SetNextToken(...)
/Users/user/go/src/github.com/aws/aws-sdk-go/service/configservice/api.go:12230
main.main()
/Users/user/golang/awsgotest/awsgotest.go:26 +0x232
Ok, finally figured it out with the help of a very kind Alex Diehl via this ticket https://github.com/aws/aws-sdk-go/issues/3293 on the official aws-sdk-go repo.
I would still say the aws sdk for go definitely lacks simple examples for configservice at the least on recommended usage.
Here the code that works. This will also show how to use simple recursive function in go to use NextToken for pagination of api results that span multiple pages especially apis that do not have built in paginators.
Also note that DescribeConfigRules API does not list all AWS Managed Config Rules, only the Config rules enabled for your account.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
var i int = 0
func main() {
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-profile")})
if err != nil {
log.Fatal(err)
}
//Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
fmt.Printf("configsvc type: %T\n", configsvc)
rules := &configservice.DescribeConfigRulesInput{}
getConfigRulesFunc(configsvc, rules)
}
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for _, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
i = i + 1
}
if configrulesoutput.NextToken != nil {
fmt.Println("In if nexttoken is not empty")
fmt.Println("Print NextToken: ", *configrulesoutput.NextToken)
ruleset := &configservice.DescribeConfigRulesInput{}
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Code in Bold were the ones giving me grief on how to use the NextToken based on best practices atleast for the go sdk for aws.
FYI, you could have looked in the AWS Go guide as there is a section on pagination: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/making-requests.html#using-pagination-methods.
Not sure what is going on, this code worked once yesterday. Now no matter what value I use, AWS is returning a error that it already exists, but that's impossible.
2020/04/17 19:10:30 error ResourceExistsException: The operation failed because the secret /gog1/RandomSiteName3 already exists.
_, err = PutParam("/gog1/RandomSiteName3", "test", true, EventGuid)
if err != nil {
log.Printf("error writing secret: %v ", err)
return
}
func PutParam(paramName string, paramValue string, encrypt bool, guid string) (output string, err error) {
svc := secretsmanager.New(AWSSession)
input := &secretsmanager.CreateSecretInput{
// ClientRequestToken: aws.String(guid),
// Description: aws.String("My test database secret created with the CLI"),
Name: aws.String(paramName),
SecretString: aws.String(paramValue),
}
fmt.Printf("putting secret key: %v", paramName)
_, err = svc.CreateSecret(input)
if err != nil {
return "", err
}
return
}
It was due to an s3 trigger firing in a loop:
NOTE: If writing to the bucket that triggers the notification, this
could cause an execution loop. For example, if the bucket triggers a
Lambda function each time an object is uploaded, and the function
uploads an object to the bucket, then the function indirectly triggers
itself. To avoid this, use two buckets, or configure the trigger to
only apply to a prefix used for incoming objects.
I am new to Go and have written a function that uses the AWS Secrets Manager to fetch a secret:
//Helper function to get secret from AWS Secret Manager
func getAWSSecrets() (secretMap map[string]string, err error) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
// Create a new instance of the SecretsManager client with session
svc := secretsmanager.New(sess)
//Get secret config values
req, resp := svc.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
...
}
I need to create a unit test for the function, and to do so I need to mock the AWS Secrets Manager. I discovered a Secrets Manager Interface that AWS was created to help with unit testing. In the example displayed, the AWS Secrets Manager is passed into the function being tested, making it easy to pass in the mock service. Is this the only way to successfully unit test the function? Or can the service be mocked in the function I have above?
As the comments say, make the function call a method, and take advantage of the interface AWS is providing.
I would create a service, like this one:
package service
type SecretService struct {
AwsProvider aws.SecretsManagerAPI
}
func NewSecretService() (*SecretService, err) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
return &SecretService{
AwsProvider: secretsmanager.New(sess),
}, nil
}
func (s *SecretService) GetAWSSecrets() {
req, resp := s.AwsProvider.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
// More magic below ...
}
That way in the tests, I could pass any mock into SecretService, like this:
func TestSecretService_GetAWSSecrets(t *testing.T) {
service := &service.SecretService{
AwsProvider: <use your mock here>
}
}
One caveat, I guess is that the mock has to implement all methods of SecretsManagerAPI, which I think it's a lot of work for this simple scenario... in any case you can create your own interface inside the service package with only the subset of methods you'll use, let's say you're only going to use GetSecretValueRequest, and CreateSecret:
package service
type SecretProvider interface {
CreateSecret(*secretsmanager.CreateSecretInput) (*secretsmanager.CreateSecretOutput, error)
GetSecretValueRequest(*secretsmanager.GetSecretValueInput) (*request.Request, *secretsmanager.GetSecretValueOutput)
}
Change the service:
type SecretService struct {
AwsProvider SecretProvider
}
Now your mock only has to implement SecretProvider interface methods only. And of course, AWS SecretsManager implicitly impleements SecretProvider.