CDK custom resource response for flyway lambda - amazon-web-services

I have taken a Flyway lambda from this repo and I am trying to implement a custom resource that invokes the lambda whenever the migration source folder changes. It works fine but occasionally the response object exceeds the 4096 byte limit and the stack fails to deploy. Below is my custom resource code:
flyway_resource = cr.AwsCustomResource(
self,
f"FlywayCustomResource-{construct_id}",
log_retention=RetentionDays.ONE_WEEK,
on_update=cr.AwsSdkCall(
service="Lambda",
action="invoke",
physical_resource_id=cr.PhysicalResourceId.of("FlywayTrigger"),
parameters={
"FunctionName": flyway_lambda_function.function_name,
"InvocationType": "RequestResponse",
"Payload": "{" +
"\"flywayRequest\":{\"flywayMethod\": \"migrate\"}," +
" \"assetHash\": \"" +
# The hash of this asset (i.e., the local migration folder) changes when the content
# of the folder changes and hence an UPDATED life cycle message is produced.
# (i.e., the params of the function have changed)
assets.Asset(self, "hashId", path=migrations_source).asset_hash +
"\"}"
},
# This would avoid lambda telling output size too big when erroring, we must use the right selector(s)
# output_paths=['info', 'errorMessage', 'errorType']
),
policy=cr.AwsCustomResourcePolicy.from_statements([
iam.PolicyStatement(
actions=["lambda:InvokeFunction"],
resources=[flyway_lambda_function.function_arn]
)
])
)
I know that I can use the output_paths parameter to limit the fields in the response that will be actually selected, but I am not quite sure I have the correct field names in the list. Here's a successful example response of the lambda:
For failure I have names such as 'errorMessage', 'errorType' in the root level (the same level that 'info' exists). I have two questions:
Is the commented output_paths parameter correct? I want to verify that the values of this parameter must match the keys in the json response of the lambda. I have a hard time understanding this in the docs. (If I uncomment it works but it might be because the the specified paths don't exist and the response becomes empty and hence does not exceed the limit anymore)
Assuming that 1) is correct how do I verify the response of the custom resource. In the case that the response in erroneous I would like for the deployment to fail. I know that I can do cr.get_response_field so with cr.get_response_field('info') or cr.get_response_field('errorMessage') perhaps I could check which of these keys exists in another lambda backed custom resource. Is there a better way?

Related

limiting array size in security rules cloud firestore using request.resorce.data?

I have the two following security rules the later checks if the document premiumUitill value in DB is bigger that current time meaning the premium is valid.
the issue here is with the first rule I want to disable array size so it won't pass 50 of length and I am pushing using arrayUninon(data) should I check for the size of resource.datarather than request.resorce.data ? in my testing request.resource.data.arr.size() < 50 works but it doesn't make sense to check the incoming data since the incoming has the payload only is something with the arrayUnion() that makes it work ?
await updateDoc(docRef, {
arr: arrayUnion(payload),
}).catch((error) => {
errorHandeling(error, 'An error has happened', reduxDispatch, SetSnackBarMsg);
});
&& request.resource.data.arr.size() < 50
&& resource.data.premiumUntill > request.time
In Cloud Firestore security rules, resource refers to the existing document in the database, and request.resource refers to the document as it exists in the request (during a write, i.e. a set or update).
From the documentation on data validation:
The resource variable refers to the requested document, and resource.data is a map of all of the fields and values stored in the document. For more information on the resource variable, see the reference documentation.
When writing data, you may want to compare incoming data to existing data. In this case, if your ruleset allows the pending write, the request.resource variable contains the future state of the document. For update operations that only modify a subset of the document fields, the request.resource variable will contain the pending document state after the operation. You can check the field values in request.resource to prevent unwanted or inconsistent data updates:
service cloud.firestore {
match /databases/{database}/documents {
// Make sure all cities have a positive population and
// the name is not changed
match /cities/{city} {
allow update: if request.resource.data.population > 0
&& request.resource.data.name == resource.data.name;
}
}
}
Additionally you may watch this video and also may have a look at this stackoverflow

Google Cloud platform terraform/terragrunt googleapi: Error 409: Requested entity already exist

I am having a strange issue when trying to push code out to our gcp repo. It fails with the following error "googleapi: Error 409: Requested entity already exists, alreadyExists" and it is referring to a project that already exists() This only occurs after i either remove another project that's no longer needed or add .bck to the terragrunt.hcl files. These projects have no dependancies on each other whatsoever.
terraform {
source = "../../../modules//project/"
}
include {
path = find_in_parent_folders("org.hcl")
}
dependency "folder" {
config_path = "../"
# Configure mock outputs for the terraform commands that are returned when there are no
outputs available (e.g the
# module hasn't been applied yet.
mock_outputs_allowed_terraform_commands = ["plan", "validate"]
mock_outputs = {
folder_id = "folder-not-created-yet"
}
}
inputs = {
project_name = "<pimsstest-project>"
folder_id = dependency.folder.outputs.folder_created # Test folder id
is_service_project = true
code push will fail with the structure in VS code is like this:
But it succeeds when like this
Some background to add. Pimsstest used to exist in a production folder under org and i moved it to test via vs code with a simple cut and paste and re push of code. I then removed the project from the console as it still existed in production. I cannot work out why the removal of another project will flag up this existing error message on pimsstest. It doesn't make any sense to me.
Across GCP a project ID can exist once only. Upon removal, it might not be instantly available again (it will always have status "scheduled for removal" - and you should receive an email, with the details of the scheduled operation). What the error message actually is trying to tell may be:
Error 409: Requested entity already STILL exist.
In theory (even if it's unlikely, when the name is unique enough), any other customer could snatch the name in the meanwhile - in which case the error message could be literally understood.

AWS Systems Manager `GetParametersByPath` API returns outdated results

I am trying to utilize SSM's GetParametersByPath API and I am getting outdated results which looks like a ~3 seconds caching by Path parameter.
I am testing the following scenario:
Get parameters by path recursively
Put a parameter under the tested path
Get parameters by path again using same arguments as in (1)
I am getting the same response in step (3) as in step (1) regardless of the changes in step (2).
I am thinking that this is related to caching because the following 2 scenarios work as expected.
Correct behavior scenario 1:
Get parameters by path recursively
Put a parameter under the tested path
Sleep for 3 seconds
Get parameters by path again using same parameters as in (1)
Correct behavior scenario 2:
Put a parameter under the tested path
Get parameters by path recursively
This behavior is consistent across different SDKs which I tried: .Net, Python (boto3) and CLI, so this is not an SKD issue.
Here is a code snippet in Python with boto3 that replicates incorrect behavior:
import boto3
client = boto3.client('ssm')
first = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(first['Parameters'][0]['Version'] if first['Parameters'] else None)
put_response = client.put_parameter(
Name='/param-store-test/1/2',
Value='test',
Type='SecureString',
KeyId='alias/aws/ssm',
Overwrite=True,
Tier='Standard')
print("v{}".format(put_response['Version']))
second = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(second['Parameters'][0]['Version'] if second['Parameters'] else None)
This code gives me the following output when run for the first time:
None
v1
None
And when run for the second time:
1
v2
1
You can see the pattern - the first request is being cached.
According to API docs: Request results are returned on a best-effort basis.
So is this behavior considered to be correct?
Does this mean that I don't have any way to get all parameters by path in a reliable way?
I have noticed that get_parameters_by_path() does not work as I expected.
You might think that (simplified code):
put_parameter('/abc/def', "123")
get_parameters_by_path('/abc/def')
would return "123", whereas it seems to return [] nothing.
However,
get_parameter('/abc/def')
correctly returns "123".
Also,
get_parameters_by_path('/abc')
will return '/abc/def' => "123"
So it seems to work, but not quite as one might expect.

Can't access SQS message attributes using boto3

I'm trying to pass and then retrieve a message with attributes into AWS SQS.
Even though I can see attributes of a message through Management Console, I can't get them using boto3, always get None. Changing "AttributeNames" doesn't make a difference. Message body can be retrieved OK.
import boto3
sqs = boto3.resource('sqs', region_name = "us-west-2")
queue = sqs.get_queue_by_name(QueueName='test')
queue.send_message(MessageBody = "LastEvaluatedKey",
MessageAttributes ={
'class_number':{
"StringValue":"Value value ",
"DataType":"String"
}
}
)
messages = queue.receive_messages(
MaxNumberOfMessages=1,
AttributeNames=['All']
)
for msg in messages:
print(msg.message_attributes) # returns None
print(msg.body) # returns correct value
Attributes (system generated) and Message Attributes (user defined) are two different kinds of things provided by the back-end API.
You're looking for message attributes, but you're asking the code to fetch attributes, instead.
AttributeNames=['All']`
It seems like you need to be fetching message attributes...
MessageAttributeNames=['All']
...when calling queue.receive_messages(). Or, the specific message attribute names that you want, if known, could be used instead of 'All' (which -- had I designed this API -- would have been called '*', but I digress).
Admittedly, this is only an intuitive guess on my part based on familiarity with the underlying API, but it seems to be consistent with http://boto3.readthedocs.io/en/latest/guide/sqs.html.

How to increase deploy timeout limit at AWS Opsworks?

I would like to increase the deploy time, in a stack layer that hosts many apps (AWS Opsworks).
Currenlty I get the following error:
Eror
[2014-05-05T22:27:51+00:00] ERROR: Running exception handlers
[2014-05-05T22:27:51+00:00] ERROR: Exception handlers complete
[2014-05-05T22:27:51+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache/chef-stacktrace.out
[2014-05-05T22:27:51+00:00] ERROR: deploy[/srv/www/lakers_test] (opsworks_delayed_job::deploy line 65) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 600s:
Thanks in advance.
First of all, as mentioned in this ticket reporting a similar issue, the Opsworks guys recommend trying to speed up the call first (there's always room for optimization).
If that doesn't work, we can go down the rabbit hole: this gets called, which in turn calls Mixlib::ShellOut.new, which happens to have a timeout option that you can pass in the initializer!
Now you can use an Opsworks custom cookbook to overwrite the initial method, and pass the corresponding timeout option. Opsworks merges the contents of its base cookbooks with the contents of your custom cookbook - therefore you only need to add & edit one single file to your custom cookbook: opsworks_commons/libraries/shellout.rb:
module OpsWorks
module ShellOut
extend self
# This would be your new default timeout.
DEFAULT_OPTIONS = { timeout: 900 }
def shellout(command, options = {})
cmd = Mixlib::ShellOut.new(command, DEFAULT_OPTIONS.merge(options))
cmd.run_command
cmd.error!
[cmd.stderr, cmd.stdout].join("\n")
end
end
end
Notice how the only additions are just DEFAULT_OPTIONS and merging these options in the Mixlib::ShellOut.new call.
An improvement to this method would be changing this timeout option via a chef attribute, that you could in turn update via your custom JSON in the Opsworks interface. This means passing the timeout attribute in the initial Opsworks::ShellOut.shellout call - not in the method definition. But this depends on how the shellout method actually gets called...