GCP Dataproc - unable to access GCP Cloud storage - google-cloud-platform

Question

I have a Dataproc(Spark Structured Streaming) job which takes data from Kafka, and does some processing. The checkpoint is a GCP Cloud storage, and it is somehow unable to list the objects in GCP Storage
Here is the error :
Traceback (most recent call last):
File "/tmp/3c930be06f4242378251aa6760390f8d/main-v1-test.py", line 352, in <module>
sys.exit(main())
File "/tmp/3c930be06f4242378251aa6760390f8d/main-v1-test.py", line 348, in main
ss.readData()
File "/tmp/3c930be06f4242378251aa6760390f8d/main-v1-test.py", line 301, in readData
query = df_stream.selectExpr("CAST(value AS STRING)", "timestamp", "topic").writeStream \
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/streaming.py", line 1491, in start
File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o96.start.
: java.io.IOException: Error accessing gs://ss-checkpoint-10m-test-v1/metadata
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:2221)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:2108)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfoInternal(GoogleCloudStorageFileSystem.java:1091)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfo(GoogleCloudStorageFileSystem.java:1065)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:955)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1691)
at org.apache.spark.sql.execution.streaming.StreamMetadata$.read(StreamMetadata.scala:53)
at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:174)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.<init>(MicroBatchExecution.scala:50)
at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:317)
at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:359)
at org.apache.spark.sql.streaming.DataStreamWriter.startQuery(DataStreamWriter.scala:466)
at org.apache.spark.sql.streaming.DataStreamWriter.startInternal(DataStreamWriter.scala:414)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/ss-checkpoint-10m-test-v1/o/metadata?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "939354532596-compute#developer.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).",
"reason" : "forbidden"
} ],
"message" : "939354532596-compute#developer.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist)."
}
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:2215)
... 24 more
Additional details :
I'm using Service account kafka-admin#versa-sml-googl.iam.gserviceaccount.com to start the job, however the Dataproc VMs seem to be using SA -> 939354532596-compute#developer.gserviceaccount.com to access the buckets :
(base) Karans-MacBook-Pro:Versa-StructuredStreaming karanalang$ gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
dataproc-access#versa-kafka-poc.iam.gserviceaccount.com
* kafka-admin#versa-sml-googl.iam.gserviceaccount.com
karan.alang#gmail.com
karan#versa-networks.com
loki-748#versa-sml-googl.iam.gserviceaccount.com
loki-storage#versa-sml-googl.iam.gserviceaccount.com
Here is the list of roles in the above 2 Service Accounts:
kafka-admin#versa-sml-googl.iam.gserviceaccount.com
(base) Karans-MacBook-Pro:Versa-StructuredStreaming karanalang$ gcloud projects get-iam-policy versa-sml-googl \
> --flatten="bindings[].members" \
> --format='table(bindings.role)' \
> --filter="bindings.members:kafka-admin#versa-sml-googl.iam.gserviceaccount.com"
ROLE
roles/compute.admin
roles/compute.storageAdmin
roles/container.admin
roles/container.clusterAdmin
roles/dataproc.admin
roles/dataproc.editor
roles/dataproc.worker
roles/iam.serviceAccountTokenCreator
roles/iam.serviceAccountUser
roles/metastore.editor
roles/owner
roles/storage.admin
roles/storage.objectAdmin
roles/storage.objectViewer
(base) Karans-MacBook-Pro:Versa-StructuredStreaming karanalang$ gcloud projects get-iam-policy versa-sml-googl \
> --flatten="bindings[].members" \
> --format='table(bindings.role)' \
> --filter="939354532596-compute#developer.gserviceaccount.com"
ROLE
roles/compute.storageAdmin
roles/editor
roles/metastore.serviceAgent
roles/storage.objectAdmin
roles/storage.objectViewer
The ServiceAccounts seem to have the required roles, so am not sure what the issue is ?
Any input on how to debug/fix this issue ?
tia!