I have an S3 lifecycle policy to delete objects after 3 days, and I am using a prefix. My problem is that the policy works for all but one sub-directory. For example, lets say my bucket looks like this:
s3://my-bucket/myPrefix/env=dev/
s3://my-bucket/myPrefix/env=stg/
s3://my-bucket/myPrefix/env=prod/
When I check the stg and prod directories, there are no objects older than 3 days. However, when I check the dev directory, there are objects a lot older than that.
Note - There is a huge difference between the volume of data in dev compared to the other 2. Dev holds a lot more logs than the others.
My initial thought was that it was taking longer for Eventual Consistency to show what was deleted and what wasn't, but that theory is gone considering the time that has passed.
The issue seems related to the amount of data in this location under the prefix compared to the others, but I'm not sure what I can do to resolve this. Should I have another policy specific to this location, or is there a somewhere I can check to see what is causing the failure? I did not see anything in Cloudtrail for this event.
Here is my policy:
{
"Rules": [
{
"Expiration": {
"Days": 3
},
"ID": "Delete Object When Stale",
"Prefix": "myPrefix/",
"Status": "Enabled"
}
]
}
Related
So I've created a logging alert policy on google cloud that monitors the project's logs and sends an alert if it finds a log that matches a certain query. This is all good and fine, but whenever it does send an email alert, it's barebones. I am unable to include anything useful in the email alert such as the actual message, the user must instead click on "View incident" and go to the specified timeframe of when the alert happened.
Is there no way to include the message? As far as I can tell viewing the gcp Using Markdown and variables in documentation templates doc on this.
I'm only really able to use ${resource.label.x} which isn't really all that useful because it already includes most of that stuff by default in the alert.
Could I have something like ${jsonPayload.message}? It didn't work when I tried it.
Probably (!) not.
To be clear, the alerting policies track metrics (not logs) and you've created a log-based metric that you're using as the basis for an alert.
There's information loss between the underlying log (that contains e.g. jsonPayload) and the metric that's produced from it (which probably does not). You can create Log-based metrics labels using expressions that include the underlying log entry fields.
However, per the example in Google's docs, you'd want to consider a limited (enum) type for these values (e.g. HTTP status although that may be too broad too) rather than a potentially infinite jsonPayload.
It is possible. Suppose you need to pass "jsonPayload.message" present in your GCP log to documentation section in your policy. You need to use "label_extractor" feature to extract your log message.
I will share a policy creation JSON file template wherein you can pass "jsonPayload.message" in the documentation section in your policy.
policy_json = {
"display_name": "<policy_name>",
"documentation": {
"content": "I have the extracted the log message:${log.extracted_label.msg}",
"mime_type": "text/markdown"
},
"user_labels": {},
"conditions": [
{
"display_name": "<condition_name>",
"condition_matched_log": {
"filter": "<filter_condition>",
"label_extractors": {
"msg": "EXTRACT(jsonPayload.message)"
}
}
}
],
"alert_strategy": {
"notification_rate_limit": {
"period": "300s"
},
"auto_close": "604800s"
},
"combiner": "OR",
"enabled": True,
"notification_channels": [
"<notification_channel>"
]
}
POST METHOD URL
https://cloudresourcemanager.googleapis.com/v1/projects/project-name:setIamPolicy
Request:
{
"resource": "projects/project-name",
"policy": {
"bindings": [
{
"role": "roles/resourcemanager.organizationAdmin",
"members": [
"user:test12345678#domain.com"
]
}
],
"etag": "BwWWja0YfJA=",
"version": 3
}
}
Response:
{
"error": {
"code": 409,
"message": "There were concurrent policy changes. Please retry the
whole read-modify-write with exponential backoff.",
"status": "ABORTED" }
}
Documentation recommends using the read-modify-write pattern to update policy for a resource.
Reading the current policy by calling getIamPolicy().
Editing the returned policy, either by using a text editor or programmatically, to add or remove any desired members and their role grants.
Writing the updated policy by calling setIamPolicy().
Looks like in your case the policy you're trying to set and the policy that is currently active on the resource have diverged. One of the ways this can happen is if you did:
getIamPolicy() > policy.json
addIamPolicyBinding() or removeIamPolicyBinding()
setIamPolicy() policy.json
The policy version on the resource after #2, is out of sync with what #3 is trying to set, and so it throws an exception. To confirm you can compare the etag field in the policy your strying to set with the etag currently on the resource. There should be a mismatch.
This means that more than one change was performed at the same time. You should try to perform only one request to change policies at the same time.
Implementing Exponential backoff should help you with this error. It is as simple as handle your request retry with a time magnitude of n+1 + random_number_milliseconds seconds and retry the request
I was able to fix this issue by removing
"etag": "BwWWja0YfJA=",
"version": 3
from the template when using gcloud projects set-iam-policy command. It will ask you to overwrite the existing policy before committing the changes
Is there any info when and why ES may trigger automatically purge of documents marked for deletion?
Where can be found logs with possible info about trigger?
The service in question is actually AWS ES, but I do not think it is related to the topic.. may be I'm wrong?
The version in question is ElasticSearch 5.1
when the merge happens, the marked document will be purged.
there are some merge policies that indicate when the merge process triggered. for example, the number of segment's files is more than 300 or the marked document is more than 15% of a segment.
there is some information here for elasticsearch 1.4:
https://www.elastic.co/guide/en/elasticsearch/reference/1.4/index-modules-merge.html
It seems that the developers don't want to clarify the policies anymore.
this is an example of merge policy settings:
"merge": {
"scheduler": {
"max_thread_count": "1",
"auto_throttle": "true",
"max_merge_count": "6"
},
"policy": {
"reclaim_deletes_weight": "2.0",
"floor_segment": "2mb",
"max_merge_at_once_explicit": "30",
"max_merge_at_once": "10",
"max_merged_segment": "5gb",
"expunge_deletes_allowed": "10.0",
"segments_per_tier": "10.0",
"deletes_pct_allowed": "33.0"
}
for logging a merge process I think you should change logs level to INFO or DEBUG. (log4j settings)
I've created a lambda function to read a file (input.csv) from s3 bucket and make some changes into it and save that file(output.csv) in same bucket.
Note: i have not deleted input.csv file in bucket.
The lambda function is triggered with object-created(All) event. But the function is called continuously like infinite number of times as input file is present in bucket.
Is is supposed to happen like this ? or Is it fault?
This is your fault :)
You have set up a recursive trigger - each time you update the file, you're actually writing a new copy of it, which triggers the event, etc.
This was a key warning in the initial demo when Lambda was released (an image is uploaded to S3, lambda is triggered to create a thumbnail - if that thumbnail is written to the same bucket, it will trigger again, etc)
As #chris has pointed out, you've triggered a recursive loop by having events triggered by an S3 PUT event, which in turns performs another PUT, calling the trigger again and again.
To avoid this problem, the simplest method is to use two S3 buckets - one for files to be placed prior to processing, and another for files to be placed post-processing.
If you don't want to use two S3 buckets, you can modify your trigger condition to include FilterRules (docs). This allows you to control the trigger such that it would only get executed when an object is placed in a certain "folder" in S3 (of course folders don't really exist in S3, they're just key prefixes).
Here's an example:
{
"LambdaFunctionConfigurations": [
{
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "Prefix",
"Value": "queue/"
}
]
}
},
"LambdaFunctionArn": <lambda_func_arn>,
"Id": "<lambda_func_name>:app.lambda_handler",
"Events": [
"s3:ObjectCreated:*"
]
}
]
}
I am new to terraform - I have created remote tfstate in s3, and now there are some manual changes too that are done in my AWS infrastructure. I need to import those manual changes into tfstate.
I used the import command for some resources, but for some resources such as IAM policy etc, there is no such import command.
Also some resources such as DB are changed with new parameters added, and I need to import them as well. When I try to import those changes it says:
Error importing: 1 error(s) occurred:
* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide
with an existing resource.
Please remove or rename this resource before continuing.
Any help would be appreciated. Thanks.
Before directly answering this question I think some context would help:
Behind the scenes, Terraform maintains a state file that contains a mapping from the resources in your configuration to the objects in the underlying provider API. When you create a new object with Terraform, the id of the object that was created is automatically saved in the state so that future commands can locate the referenced object for read, update, and delete operations.
terraform import, then, is a different way to create an entry in the state file. Rather than creating a new object and recording its id, instead the user provides an id on the command line. Terraform reads the object with that id and adds the result to the state file, after which it is indistinguishable in the state from a resource that Terraform created itself.
So with all of that said, let's address your questions one-by-one.
Importing Resources That Don't Support terraform import
Since each resource requires a small amount of validation and data-fetching code to do an import, not all resources are supported for import at this time.
Given what we know about what terraform import does from the above, in theory it's possible to skip Terraform's validation of the provided id and instead manually add the resource to the state. This is an advanced operation and must be done with care to avoid corrupting the state.
First, retrieve the state into a local file that you'll use for your local work:
terraform state pull >manual-import.tfstate
This will create a file manual-import.tfstate that you can open in a text editor. It uses JSON syntax, so though its internal structure is not documented as a stable format we can carefully edit it as long as we remain consistent with the expected structure.
It's simplest to locate an existing resource that is in the same module as where you want to import and duplicate and edit it. Let's assume we have a resources object like this:
"resources": {
"null_resource.foo": {
"type": "null_resource",
"depends_on": [],
"primary": {
"id": "5897853859325638329",
"attributes": {
"id": "5897853859325638329"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
}
},
Each attribute within this resources object corresponds to a resource in your configuration. The attribute name is the type and name of the resource. In this case, the resource type is null_resource and the attribute name is foo. In your case you might see something like aws_instance.server here.
The id attributes are, for many resources (but not all!), the main thing that needs to be populated. So we can duplicate this structure for a hypothetical IAM policy:
"resources": {
"null_resource.foo": {
"type": "null_resource",
"depends_on": [],
"primary": {
"id": "5897853859325638329",
"attributes": {
"id": "5897853859325638329"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
},
"aws_iam_policy.example": {
"type": "aws_iam_policy",
"depends_on": [],
"primary": {
"id": "?????",
"attributes": {
"id": "?????"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
}
},
The challenge at this step is to figure out what sort of id this resource requires. The only sure-fire way to know this is to read the code, which tells me that this resource expects the id to be the full ARN of the policy.
With that knowledge, we replace the two ????? sequences in the above example with the ARN of the policy we want to import.
After making manual changes to the state it's necessary to update the serial number at the top-level of the file. Terraform expects that any new change will have a higher serial number, so we can increment this number.
After completing the updates, we must upload the updated state file back into Terraform:
terraform state push manual-import.tfstate
Finally we can ask Terraform to refresh the state to make sure it worked:
terraform refresh
Again, this is a pretty risky process since the state file is Terraform's record of its relationship with the underlying system and it can be hard to recover if the content of this file is lost. It's often easier to simply replace a resource than to go to all of this effort, unless it's already serving a critical role in your infrastructure and there is no graceful migration strategy available.
Imports Colliding With Existing Resources
The error message given in your question is talking about an import "colliding" with an existing resource:
Error importing: 1 error(s) occurred:
* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide with an existing resource.
Please remove or rename this resource before continuing.
The meaning of this message is that when Terraform tried to write the new resource to the state file it found a resource entry already present for the name aws_security_group.Q8SgProdAdminSshInt. This suggests that either it was already imported or that a new security group was already created by Terraform itself.
You can inspect the attributes of the existing resource in state:
terraform state show aws_security_group.Q8SgProdAdminSshInt
Compare the data returned with the security group you were trying to import. If the ids match then there's nothing left to do, since the resource was already imported.
If the ids don't match then you need to figure out which of the two objects is the one you want to keep. If you'd like to keep the one that Terraform already has, you can manually delete the one you were trying to import.
If you'd like to keep the one you were trying to import instead, you can drop the unwanted one from the Terraform state to make way for the import to succeed:
terraform state rm aws_security_group.Q8SgProdAdminSshInt
Note that this just makes Terraform "forget" the resource; it will still exist in EC2, and will need to be deleted manually via the console, command line tools, or API. Be sure to note down its id before deleting it to ensure that you can find it in order to to clean it up.
For resources you have to use import function or manually add terraform state like block..
or if there is any change in config like you mentioned dB configs.. if the dB resource is managed by terraform remote state.. terraform refresh will help you..