I have set up a replication rule on my S3 bucket to populate a preprod bucket for testing purposes. This means I will want to be able to turn the replication on and off easily, and likely dump and refresh the replication bucket as necessary. I'm creating a script for this but am having a hard time finding a way to easily turn the replication rule on and off outside of using the AWS Console.
Is there an option beyond put-bucket-replication? That works but is basically restating the whole replication config each time, instead of just enabling or disabling the existing one.
It looks like the only solution is to pass different put-bucket-replications with the Status as disabled or enabled. Example of disabled below using python and boto3:
import boto3
client = boto3.client('s3')
##Enable
client.put_bucket_replication(Bucket='yoursourcebucketname', ReplicationConfiguration={
"Role": "arn:aws:iam::999999999:role/service-role/yourrolename",
"Rules": [
{
"Status": "Disabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::yourlandingbucket",
"Account": "838382828"
}
}
]
}
)
Related
The resource.labels.region field for the dataflow_step logs in stackdriver, points to global even though the specified regional endpoint is Europe-west2.
Any idea on what is it exactly pointing to?
Once you've supplied GCP Logs Viewer with the desired filtering option, as most simple query based on your inputs seeking for dataflow_step resource type:
resource.type="dataflow_step"
resource.labels.region="europe-west2"
You would probably observe query results retrieved from Cloud Dataflow REST API, consisting with logs entries formatted as a JSON outputs for all Dataflow Jobs that are residing within your GCP project in europe-west2 Regional endpoint:
{
"insertId": "insertId",
"jsonPayload": {
....
"message": "Message content",
....
},
"resource": {
"type": "dataflow_step",
"labels": {
"job_id": "job_id",
"region": "europe-west2",
"job_name": "job_name",
"project_id": "project_id",
"step_id": "step_id"
}
},
"timestamp": "timestamp",
"severity": "severity_level",
"labels": {
"compute.googleapis.com/resource_id": "resource_id",
"dataflow.googleapis.com/job_id": "job_id",
"compute.googleapis.com/resource_type": "resource_type",
"compute.googleapis.com/resource_name": "resource_name",
"dataflow.googleapis.com/region": "europe-west2",
"dataflow.googleapis.com/job_name": "job_name"
},
"logName": "logName",
"receiveTimestamp": "receiveTimestamp"
According to GCP logging service documentation each monitoring resource type derives particular labels from the nested service API, dataflow.googleapis.com corresponds to Dataflow service.
Therefore, if you run Dataflow Job defining the location for job's metadata region, GCP logging service will fetch up this regional endpoint from job description throughout dataflow.googleapis.com REST methods.
The resource.labels.region field on Dataflow Step logs should refer to the regional endpoint that the job is using. "Global" is not an expected value there.
I have been trying to use the update functionality of the AWS CLI to update codepipeline with the following command:
aws codepipeline update-pipeline --cli-input-json file://Pipelines/AWS/SomeName.json
And I keep getting the following error
Unknown parameter in pipeline.stages[0].actions[0]: "region", must be one of: name, actionTypeId, runOrder, configuration, outputArtifacts, inputArtifacts, roleArn
I have checked the documentation for AWS and I don´t think theres anything wrong with the way actions is set up, here is the snippet from JSON:
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"owner": "AWS",
"provider": "S3",
"version": "1"
},
"runOrder": 1,
"configuration": {
"PollForSourceChanges": "false",
"S3Bucket": "some-bucket-name",
"S3ObjectKey": "someEnvironment/someZip.zip"
},
"outputArtifacts": [
{
"name": "SourceArtifact"
}
],
"inputArtifacts": [],
"region": "eu-west-1"
},...
]
According to the documentation provided at https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-S3.html
everything seems to be correct. Removing the region parameter updates the pipeline correctly but I am unsure of the consequences that could have on the updates itself
Any help is appreciated.
Cheers
Sky
If you try to create the pipeline through the AWS console and choose S3 as a source you will notice the region option is not available (as shown in the screenshot below). I would say this is a current limitation to the service more then a chosen design and a gap in documentation (however, happy to be proven wrong).
However, you could try include the full S3 bucket Arn which would include the region. Or take comfort in that any action deployed (without a region specified) defaults to the same region that the codepipeline is in, as per the AWS documentation.
I am trying to figure out usage of an AD user, using AWS via AssumeRoleWithSAML, following this ink, https://aws.amazon.com/blogs/security/how-to-easily-identify-your-federated-users-by-using-aws-cloudtrail/.
However, i dont see AssumeRoleWithSAML event at all in my Cloudtrails, though i can clearly see activity from this user. I went all the way to early July in cloudtrail to look up AssumeRoleWithSaml and dont see any event.
Am i missing something? Bcos of this event not coming, i am not able to correlate what this user is doing in AWS.
Thanks
Amit
You are right, there should be an event with name AssumeRoleWithSAML in the CloudTrail logs.
You already referenced the correct AWS security blog post which describes how to "identify a SAML federated user". [1]
Let's go into detail.
The IAM docs [2] contain an example how the AssumeRoleWithSAML event should look like:
{
"eventVersion": "1.05",
"userIdentity": {
"type": "WebIdentityUser",
"principalId": "accounts.google.com:[id-of-application].apps.googleusercontent.com:[id-of-user]",
"userName": "[id of user]",
"identityProvider": "accounts.google.com"
},
"eventTime": "2016-03-23T01:39:51Z",
"eventSource": "sts.amazonaws.com",
"eventName": "AssumeRoleWithWebIdentity",
"awsRegion": "us-east-2",
"sourceIPAddress": "192.0.2.101",
"userAgent": "aws-cli/1.3.23 Python/2.7.6 Linux/2.6.18-164.el5",
"requestParameters": {
"durationSeconds": 3600,
"roleArn": "arn:aws:iam::444455556666:role/FederatedWebIdentityRole",
"roleSessionName": "MyAssignedRoleSessionName"
},
"responseElements": {
"provider": "accounts.google.com",
"subjectFromWebIdentityToken": "[id of user]",
"audience": "[id of application].apps.googleusercontent.com",
"credentials": {
"accessKeyId": "ASIACQRSTUVWRAOEXAMPLE",
"expiration": "Mar 23, 2016 2:39:51 AM",
"sessionToken": "[encoded session token blob]"
},
"assumedRoleUser": {
"assumedRoleId": "AROACQRSTUVWRAOEXAMPLE:MyAssignedRoleSessionName",
"arn": "arn:aws:sts::444455556666:assumed-role/FederatedWebIdentityRole/MyAssignedRoleSessionName"
}
},
"resources": [
{
"ARN": "arn:aws:iam::444455556666:role/FederatedWebIdentityRole",
"accountId": "444455556666",
"type": "AWS::IAM::Role"
}
],
"requestID": "6EXAMPLE-e595-11e5-b2c7-c974fEXAMPLE",
"eventID": "bEXAMPLE-0b30-4246-b28c-e3da3EXAMPLE",
"eventType": "AwsApiCall",
"recipientAccountId": "444455556666"
}
As we can see, the requestParameters contain an element durationSeconds which is the value you are looking for.
Why is the event missing?
First of all, it is necessary to know if you are using the AWS CloudTrail Console or if you are parsing the CloudTrail files which were delivered to the S3 bucket. If you use the CloudTrail console, you are able the view the last 90 days of recorded API activity and events in an AWS Region only!! [3]
So make sure that you use AWS Athena or another solution if you must go further back in time.
You must look into the trail of the correct region! You do this by inspecting the respective S3 prefix for a multi-region trail or by clicking onto the desired region in the top right corner if you use the AWS CloudTrail Console. This is important because regional services are logging to their respective trail!! AWS mentions this as follows:
If you activate AWS STS endpoints in Regions other than the default global endpoint, then you must also turn on CloudTrail logging in those Regions. This is necessary to record any AWS STS API calls that are made in those Regions. For more information, see Turning On CloudTrail in Additional Regions in the AWS CloudTrail User Guide. [4]
Make sure to look into the correct account! You must inspect the trail of the account whose role was assumed. I mention this explicitly because there are multi-account environments which might use centralized identity accounts etc.
References
[1] https://aws.amazon.com/de/blogs/security/how-to-easily-identify-your-federated-users-by-using-aws-cloudtrail/
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html
[3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html
[4] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html
Is there a way to listen to an s3 bucket but only inside a specific "folder" so for instance if i had a folder named input i would say listen on "s3://bucket-name/folder1/*"?
Right now it seems that you can only listen to the entire bucket.My issue is that I want to use the same bucket to trigger CloudWatch based on specific key path, and of course all cloudwatch rules will be triggered.
This is my flow:
CloudTrail (monitor s3://bucket/path) -> CloudWatch (will have an Event Rule for any PUT in that s3://bucket/path) trigger-> StepFunction-> LambdaFunctions
I also tried to restrict this from CloudWatch role to give permissions only to that specific S3 bucket path without luck.
This is my event rule:
{ "source": [ "aws.s3" ], "detail-type": [ "AWS API Call via CloudTrail" ], "detail": { "eventSource": [ "s3.amazonaws.com" ], "eventName": [ "PutObject" ], "requestParameters": { "bucketName": [ " bucketname" ] } } }
Is there any workaround?
As of the date of this writing, I do not currently know of a way to accomplish ON THE RULE. There could be a workaround on the rule but I have not found it...
...HOWEVER:
This can be accomplished by using CloudTrail.
Remove the key from the event rule object you have, and keep the bucket name
Go to CloudTrail. If all data events are turned on, disable them and created your own trail.
In CloudTrail, create a new Trail. Specify object or bucket level operations
Enable S3 Data events - ASSUMING you want to listen for putObject or similar
specify your bucket, AND when it says Individual bucket selection, type in the bucket name AND the path you want to monitor. bucketname/folder1/folder2
specify whether you want read and write actions to be logged to the trail.
Now you have a log trail for that path only. The cloudwatch rule, or eventBridge rule, can now specify the bucket, and whatever operations you want to monitor.
try adding
"requestParameters": { "bucketName": [ " bucketname" ],"key":["folder1"] }
could be work.
I am not able to setup Cross Region Replication when the objects are server side encrypted. I am using awscli to set it up. This is what I have done.
Cross region replication role IAM policy looks like this:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"s3:GetReplicationConfiguration",
"s3:ListBucket"
],
"Resource":[
"arn:aws:s3:::source-bucket"
]
},
{
"Effect":"Allow",
"Action":[
"s3:GetObjectVersion",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging"
],
"Resource":[
"arn:aws:s3:::source-bucket/*"
]
},
{
"Effect":"Allow",
"Action":[
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:ReplicateTags"
],
"Resource":"arn:aws:s3:::destination-bucket/*"
}
]
}
This is how my replication configuration file looks like:
{
"Role": "arn:aws:iam::1234567890:role/replication-role",
"Rules": [
{
"ID": "abcd",
"Prefix": "",
"Status": "Enabled",
"SourceSelectionCriteria": {
"SseKmsEncryptedObjects": {
"Status": "Enabled"
}
},
"Destination": {
"Bucket": "arn:aws:s3:::destinationbucket",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:1234567890:key/849b779d-bdc3-4190-b285-6006657a578c"
}
}
}
]
}
This is how my cli command looks like:
aws s3api put-bucket-replication --bucket "sourcebucket" --replication-configuration file://./replicationconfigfile.json
When I go to S3 bucket after running the cli command, I can see the replication rule being created with KMS-Encrypted Object as replicate but when i click on edit to see the details, it does not have any KMS keys selected.
------Update-------
if i delete the rule created by cli and set it up using console, it selects all the kms keys in the wizard. So the question is why is it not selecting kms keys in source region when I am using cli?
what am I missing here?
KMS list field that is showed in the wizard is missing in the CLI, I have the same issue because I am using KMS to encrypt my origin and my destination bucket and I can't select the key to decrypt the objects in my origin bucket as I am using Terraform to created the replication rule.
As you can see here the only parameter that exists is "Replication criteria" and the value just can be true or false, the list field "Choose one or more keys for decrypting source objects" does not exist in the AWS CLI.
I already sent this issue to them.
What did I do?
I replaced my generated KMS key managed by me to use the key managed by AWS, I just enabled server-side encryption and I choose the AES256 encryption type in both bucket, origin and destination and it works fine to me.
Just in case anyone else runs into this issue, I had a long conversation with AWS support where they confirmed that there is no way to set the key for decrypting source objects programmatically (or in CloudFormation). In my case, I had to set up the configuration with the SDK and then manually set the decryption key in the console. Fairly annoying that they haven't fixed this as of 7/8/2020.
Looking around at a Terraform thread where they discuss this same issue, I believe they get around this by setting the IAM policy for CRR directly, but I'm unsure of exactly how you do that. https://github.com/terraform-providers/terraform-provider-aws/issues/6046#issuecomment-427960842