How to add AWS ECS task info as labels with OpenTelemetry - amazon-web-services

Question

I'm trying to setup monitoring for my ECS services. The idea is to add the public.ecr.aws/aws-observability/aws-otel-collector:latest as a second container in each ECS task, and configure it such that it scapes the prometheus endpoint of the application, and then writes it the Amazon Managed Service for Prometheus. I want to add labels to all the metrics to see which ECS service and task the metrics are from. Ideally, to re-use existing grafana dashboards, I want the labels to be named job and instance for the service 'family' name, and the task id respectively.
I'm using terraform for configuration. The task definition looks like:
resource "aws_ecs_task_definition" "task" {
family = var.name
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = var.task_cpu
memory = var.task_memory
execution_role_arn = aws_iam_role.task_execution.arn
task_role_arn = aws_iam_role.task_role.arn
runtime_platform {
cpu_architecture = "ARM64"
}
container_definitions = jsonencode([
{
name = "app"
image = "quay.io/prometheus/node-exporter:latest"
cpu = var.task_cpu - 256
memory = var.task_memory - 512
essential = true
mountPoints = []
volumesFrom = []
portMappings = [{
protocol = "tcp"
containerPort = 8080
hostPort = 8080
}]
command = ["--web.listen-address=:8080"]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.task.name
awslogs-region = data.aws_region.current.name
awslogs-stream-prefix = "ecs"
}
}
},
{
name = "otel-collector"
image = "public.ecr.aws/aws-observability/aws-otel-collector:latest"
cpu = 256
memory = 512
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.otel.name
awslogs-region = data.aws_region.current.name
awslogs-stream-prefix = "ecs"
}
}
environment = [
{
name = "AOT_CONFIG_CONTENT",
value = local.adot_config
}
]
}
])
}
And the open telemetry config i'm using looks like:
extensions:
sigv4auth:
service: "aps"
region: ${yamlencode(region)}
receivers:
prometheus:
config:
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: "app"
static_configs:
- targets: [0.0.0.0:8080]
processors:
resourcedetection/ecs:
detectors: [env, ecs]
timeout: 2s
override: false
metricstransform:
transforms:
- include: ".*"
match_type: regexp
action: update
operations:
- action: update_label
label: aws.ecs.task.arn
new_label: instance_foo
- action: add_label
new_label: foobar
new_value: some value
exporters:
prometheusremotewrite:
endpoint: ${yamlencode("${endpoint}api/v1/remote_write")}
auth:
authenticator: sigv4auth
resource_to_telemetry_conversion:
enabled: true
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [prometheus]
processors: [resourcedetection/ecs, metricstransform]
exporters: [prometheusremotewrite]
However the foobar label is added to all metrics, but the instance_foo label is not added with the aws.ecs.task.arn value. In grafana the labels of resourcedetection are visible, but not the instance_foo label.
I did try to debug the open-telemetry application locally, and noticed the resourcedetection labels are not yet available in the metricstransform.
So is it possible to rename labels using the metricstransform that are provided by resourcedetection, or are there other ways to set this up?