Splunk props config for AWS Cloudtrail json logs - amazon-web-services

Question

I need to ingest AWS cloudtrail logs pulled from S3. The files contain a single json payload that contains individual cloudtrail events. However splunk is not recognising the individual events and not splitting correctly. its just one big lump as a single event
Each file contains this format:
"Records": [
{
"apiVersion": "2012-06-01",
"awsRegion": "us-west-1",
"eventID": "c-c245-2c4-32v6-vfff",
"eventName": "DescribeLoadBalancers",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventTime": "2019-11-30T18:15:33Z",
"eventType": "AwsApiCall",
"eventVersion": "1.05",
"recipientAccountId": "redacted",
"requestID": "2xc454xc-2345-234cv5-2345",
"requestParameters": null,
"responseElements": null,
"sourceIPAddress": "1.1.1.1",
"userAgent": "aws-sdk-ruby3/3.75.0 jruby/2.3.3 java aws-sdk-elasticloadbalancing/1.19.0 cloudhealth",
"userIdentity": {
"accessKeyId": "redacted",
"accountId": "redacted",
"arn": "arn:aws:sts::redacted:assumed-role/team/AssumeRoleSession",
"principalId": "redacted:AssumeRoleSession",
"sessionContext": {
"attributes": {
"creationDate": "2019-11-30T17:45:06Z",
"mfaAuthenticated": "false"
},
"sessionIssuer": {
"accountId": "redacted",
"arn": "arn:aws:iam::redacted:team/company",
"principalId": "redacted",
"type": "Role",
"userName": "redacted"
},
"webIdFederationData": {}
},
"type": "AssumedRole"
}
},{
"apiVersion": "2012-06-01",
"awsRegion": "us-west-1",
"eventID": "c-c245-2c4-32v6-vfff",
"eventName": "DescribeLoadBalancers",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventTime": "2019-11-30T18:16:33Z",
"eventType": "AwsApiCall",
"eventVersion": "1.05",
"recipientAccountId": "redacted",
"requestID": "2xc454xc-2345-234cv5-2345",
"requestParameters": null,
"responseElements": null,
"sourceIPAddress": "1.1.1.1",
"userAgent": "aws-sdk-ruby3/3.75.0 jruby/2.3.3 java aws-sdk-elasticloadbalancing/1.19.0 cloudhealth",
"userIdentity": {
"accessKeyId": "redacted",
"accountId": "redacted",
"arn": "arn:aws:sts::redacted:assumed-role/team/AssumeRoleSession",
"principalId": "redacted:AssumeRoleSession",
"sessionContext": {
"attributes": {
"creationDate": "2019-11-30T17:45:06Z",
"mfaAuthenticated": "false"
},
"sessionIssuer": {
"accountId": "redacted",
"arn": "arn:aws:iam::redacted:role/team",
"principalId": "redacted",
"type": "Role",
"userName": "redacted"
},
"webIdFederationData": {}
},
"type": "AssumedRole"
}
}
]
}
My Props looks like this
[cloudtrail]
KV_MODE = json

Answer 1

some further googling and trial and error resulted in this props config which seems to break the events correctly
[cloudtrail]
KV_MODE = json
SHOULD_LINEMERGE=false
LINE_BREAKER=((?<=}),(?={)|[\r\n]+)
SEDCMD-remove_prefix=s/{"Records":\[//g
SEDCMD-remove_suffix=s/\]}//g