Unable to set GOOGLE_APPLICATION_CREDENTIALS in pyspark shell - google-cloud-platform

Question

I am trying to access GCS from pyspark running in docker and for that I have the json file which copied in docker container .
spark._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark._jsc.hadoopConfiguration().set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
# This is required if you are using service account and set true,
spark._jsc.hadoopConfiguration().set("fs.gs.auth.service.account.enable", "true")
then setting up the GOOGLE_APPLICATION_CREDENTIALS like below -
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS']=r"abc.json"
Now when trying to access gcs objects like below it's throwing error -
df= spark.read.csv("gs://integration-o9/california_housing_train.csv")
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "**111111-compute#developer.gserviceaccount.com** does not have storage.objects.get access to the Google Cloud Storage object. Permission 'stora
ge.objects.get' denied on resource (or it may not exist).",
"reason" : "forbidden"
} ],
"message" : "**111111-compute#developer.gserviceaccount.com** does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage
.objects.get' denied on resource (or it may not exist)."
}
This is not the service account mentioned in the Json file , but if I set this via below -
export GOOGLE_APPLICATION_CREDENTIALS=abc.json
it's working fine
Any advice what's to look into , need to make it work via os.environ properties

Answer 1

Posting the answer here as below is working (need to set in the config part with .p12 key)
spark=SparkSession.builder.appName("test").config("google.cloud.auth.service.account.enable", "true").config("google.cloud.auth.service.account.email", "o9int-759#integration-3-344806.iam.gserviceaccount.com").config("google.cloud.auth.service.account.keyfile", "/path/file.p12").getOrCreate()