Terraform gcloud - google-cloud-platform

Question

I have the following setup in Google Cloud:
application 'generator' which publishes messages to a Google Cloud PubSub topic.
application 'worker' which consumes a unique message.
any invalid PubSub messages should end up in a 'dead letter' topic.
This topic should have a 'dead letter' topic where invalid messages end up.
However, whenever I configure this via Terraform, the google cloud console mentions I do not have the 'subscriber' and 'publisher' role attached to my project pubsub service account:
I have the following terraform configuration which seems to be correct AFAIK:
resource "google_project_service_identity" "pubsub_sa" {
provider = google-beta
project = var.project_id
service = "pubsub.googleapis.com"
}
/* ... topic and dead-letter topic config here ... */
data "google_iam_policy" "project_pubsub_publishers" {
binding {
role = "roles/pubsub.publisher"
members = [
"serviceAccount:${google_service_account.project_generator_serviceaccount.email}",
"serviceAccount:${google_service_account.project_worker_serviceaccount.email}",
"serviceAccount:${google_project_service_identity.pubsub_sa.email}",
]
}
}
resource "google_pubsub_topic_iam_policy" "project_request_publishers" {
project = var.project_id
topic = google_pubsub_topic.generator_request_pubsub.name
policy_data = data.google_iam_policy.project_pubsub_publishers.policy_data
}
data "google_iam_policy" "project_pubsub_subscribers" {
binding {
role = "roles/pubsub.subscriber"
members = [
"serviceAccount:${google_service_account.project_generator_serviceaccount.email}",
"serviceAccount:${google_service_account.project_worker_serviceaccount.email}",
"serviceAccount:${google_project_service_identity.pubsub_sa.email}",
]
}
}
resource "google_pubsub_topic_iam_policy" "project_request_subscribers" {
topic = google_pubsub_topic.generator_request_pubsub.name
project = var.project_id
policy_data = data.google_iam_policy.project_pubsub_subscribers.policy_data
}
Clicking 'Add' in the web gui and then doing a terraform plan shows following changes:
Terraform will perform the following actions:
# module.gcloud.google_pubsub_topic_iam_policy.project_invalid_request_publishers will be updated in-place
~ resource "google_pubsub_topic_iam_policy" "project_invalid_request_publishers" {
id = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub-invalid"
~ policy_data = jsonencode(
~ {
~ bindings = [
~ {
~ members = [
+ "serviceAccount:cicd-generator-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
+ "serviceAccount:cicd-worker-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
"serviceAccount:service-251572179467#gcp-sa-pubsub.iam.gserviceaccount.com",
]
# (1 unchanged element hidden)
},
- {
- members = [
- "serviceAccount:cicd-generator-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
- "serviceAccount:cicd-worker-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
- "serviceAccount:service-251572179467#gcp-sa-pubsub.iam.gserviceaccount.com",
]
- role = "roles/pubsub.subscriber"
},
]
}
)
# (3 unchanged attributes hidden)
}
# module.gcloud.google_pubsub_topic_iam_policy.project_invalid_request_subscribers will be updated in-place
~ resource "google_pubsub_topic_iam_policy" "project_invalid_request_subscribers" {
id = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub-invalid"
~ policy_data = jsonencode(
~ {
~ bindings = [
- {
- members = [
- "serviceAccount:service-251572179467#gcp-sa-pubsub.iam.gserviceaccount.com",
]
- role = "roles/pubsub.publisher"
},
{
members = [
"serviceAccount:cicd-generator-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
"serviceAccount:cicd-worker-sa#MY-GCLOUD-PROJECTID.iam.gserviceaccount.com",
"serviceAccount:service-251572179467#gcp-sa-pubsub.iam.gserviceaccount.com",
]
role = "roles/pubsub.subscriber"
},
]
}
)
# (3 unchanged attributes hidden)
}
# module.gcloud.google_pubsub_topic_iam_policy.project_request_subscribers will be updated in-place
~ resource "google_pubsub_topic_iam_policy" "project_request_subscribers" {
id = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub"
~ policy_data = jsonencode(
~ {
~ bindings = [
~ {
~ role = "roles/pubsub.publisher" -> "roles/pubsub.subscriber"
# (1 unchanged element hidden)
},
]
}
)
# (3 unchanged attributes hidden)
}
But I'm not sure what I'm doing wrong here. Any ideas?

Answer 1

As per the documentation, seems that you need to first actually set the configuration for a 'dead-letter topic' in GCP.
Setting a dead-letter topic
Which (among some other information) states that:
To create a subscription and set a dead-letter topic, use the gcloud pubsub subscriptions create command:
gcloud pubsub subscriptions create subscription-id \
--topic=topic-id \
--dead-letter-topic=dead-letter-topic-id \
[--max-delivery-attempts=max-delivery-attempts] \
[--dead-letter-topic-project=dead-letter-topic-project]
To update a subscription and set a dead-letter topic, use the gcloud pubsub subscriptions update command:
gcloud pubsub subscriptions update subscription-id \
--dead-letter-topic=dead-letter-topic-id \
[--max-delivery-attempts=max-delivery-attempts] \
[--dead-letter-topic-project=dead-letter-topic-project]
Granting forwarding permissions
To forward undeliverable messages to a dead-letter topic, Pub/Sub must have permission to do the following:
Publish messages to the topic.
Acknowledge the messages, which removes them from the subscription.
Pub/Sub creates and maintains a service account for each project: service-project-number#gcp-sa-pubsub.iam.gserviceaccount.com. You can grant forwarding permissions by assigning publisher and subscriber roles to this service account. If you configured the subscription using Cloud Console, the roles are granted automatically.
Assigning Pub/Sub the publisher role
To grant Pub/Sub permission to publish messages to a dead-letter topic, run the following command:
PUBSUB_SERVICE_ACCOUNT="service-${project-number}#gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding dead-letter-topic-id \
--member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
--role="roles/pubsub.publisher"
Assigning Pub/Sub the subscriber role
To grant Pub/Sub permission to acknowledge forwarded undeliverable messages, run the following command:
PUBSUB_SERVICE_ACCOUNT="service-${project-number}#gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub subscriptions add-iam-policy-binding subscription-id \
--member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
--role="roles/pubsub.subscriber"
Hope this is helpful for you.
Regards.

Answer 2

Jaime is right, you need to add those IAM policies to
"service-${project-number}#gcp-sa-pubsub.iam.gserviceaccount.com"
It is a specific sa hidden from the main ones. You can find it in the console in >IAM and select the check box on the top right corner "include google-provided role grants"
There is also needed to add a google_pubsub_topic_iam_policy.
Here is a Terraform working example
data "google_project" "current" {}
data "google_iam_policy" "publisher" {
binding {
role = "roles/pubsub.publisher"
members = [
"serviceAccount:service-${data.google_project.current.number}#gcp-sa-pubsub.iam.gserviceaccount.com",
]
}
}
resource "google_pubsub_topic_iam_policy" "policy" {
project = var.project
topic = google_pubsub_topic.yourTopic.name
policy_data = data.google_iam_policy.publisher.policy_data
}
data "google_iam_policy" "subscriber" {
binding {
role = "roles/pubsub.subscriber"
members = [
"serviceAccount:service-${data.google_project.current.number}#gcp-sa-pubsub.iam.gserviceaccount.com",
]
}
}
resource "google_pubsub_subscription_iam_policy" "policy" {
subscription = google_pubsub_subscription.yourSubscription.name
policy_data = data.google_iam_policy.subscriber.policy_data
}