Certmanager Cross Project DNS01 Challenge Fail - google-cloud-platform

Question

Project Prod and Project Staging have been setup and each running a GKE cluster. Cert-manager is installed to automate the process of certificate issuance as explained in official docs.
Project Prod has DNS that maps to both prod and staging cluster istio gateway IP addresses.
On DNS01 challenge for cluster in Project Prod, manages to authenticate, and certificate is issued successfully.
But the cluster running in Project Staging, fails to get certificate due to not enough permission to authenticate and verify via Cloud DNS setup in Project Prod.
In Project Prod, there is a service account with dns/admin role that is setup via GKE secret and accessed in clusterissuer like so
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod-clusterissuer
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: name#name.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod-clusterissuer
solvers:
# ACME DNS-01 provider configurations
- dns01:
# Google Cloud DNS
cloudDNS:
# Secret from the google service account key
serviceAccountSecretRef:
name: cloud-dns-key
key: key.json
# The project in which to update the DNS zone
project: iprocure-server-prod
Certificate is issued successfully in Project Prod GKE cluster.
Project Staging GKE cluster, clusterissuer has its service account with dns/admin role just like in Project Prod, but fails to perform dns01 challenge in Project Prod DNS.
Following error is seen when kubectl describe challenge
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PresentError 2m56s (x19 over 7h14m) cert-manager Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden
What should be done to Project Staging service account to enable dns01 challenge to be performed in Project Prod Clous DNS

Answer 1

I faced this problem too. I had to replicate the service account in my Prod Project with DNS ADMIN permission to Staging Project so that the GKE cluster in Staging can have enough permission to authenticate and verify via Cloud DNS setup in the Prod project
You have to create a SA in Prod Project with DNS ADMIN permission and master the email of that SA then go to project B and make that SA email a member by adding it as a member with also DNS ADMIN permission.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: {{ .Values.app.certificate.issuer.name }}
namespace: {{ .Values.app.namespace }}
labels:
app.kubernetes.io/managed-by: "Helm"
spec:
acme:
email: {{ .Values.app.certificate.acme.email }}
privateKeySecretRef:
name: {{ .Values.app.certificate.issuer.name }}
server: {{ .Values.app.certificate.acme.server }}
solvers:
- dns01:
cloudDNS:
project: {{ .Values.app.project_id }} ## Make sure this is the project of where DNS is e.g Prod Project
serviceAccountSecretRef:
name: {{ .Values.secrets.name }}
# Secret from the google service account key
key: key.json