AWS EKS Fargate logging to AWS Cloudwatch: log groups are not creating - amazon-web-services

Question

I used official procedure from AWS and this one to enable logging.
Here is yaml files I've applied:
---
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "true"
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region us-east-1
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group true
log_key log
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
Inside the pod I can see that logging was enabled:
apiVersion: v1
kind: Pod
metadata:
annotations:
CapacityProvisioned: 2vCPU 4GB
Logging: LoggingEnabled
kubectl.kubernetes.io/restartedAt: "2023-01-17T19:31:20+01:00"
kubernetes.io/psp: eks.privileged
creationTimestamp: "2023-01-17T18:31:28Z"
Logs exists inside the container:
kubectl logs dev-768647846c-hbmv7 -n dev-fargate
But in AWS CloudWatch log groups are not created, even for fluent-bit itself
From the pod cli I can create log groups in AWS Cloudwatch, so the permissions are ok
I also tried cloudwatch instead of cloudwatch_logs plugin, but no luck

Answer 1

I've solved my issue.
The tricky thing is: IAM policy must be attached to the default pod execution role which created automatically with the namespace and it has no relation to the service account \ custom pod execution role