I'm attempting to create a dataproc cluster using the https://github.com/googleapis/java-dataproc library, following the example here: https://github.com/googleapis/java-dataproc/blob/main/samples/snippets/src/main/java/CreateCluster.java
My (translated to scala) code:
import com.google.cloud.dataproc.v1._
object CreateCluster extends App {
val projectId = "my-project-id"
val region = "europe-west1"
val clusterName = "test-cluster"
val regionEndpoint = s"$region-dataproc.googleapis.com:443"
val clusterControllerSettings = ClusterControllerSettings.newBuilder()
.setEndpoint(regionEndpoint)
.build()
val clusterControllerClient = ClusterControllerClient.create(clusterControllerSettings)
val masterConfig = InstanceGroupConfig.newBuilder.setMachineTypeUri("n1-standard-2").setNumInstances(1).build
val workerConfig = InstanceGroupConfig.newBuilder.setMachineTypeUri("n1-standard-2").setNumInstances(2).build
val clusterConfig = ClusterConfig.newBuilder.setMasterConfig(masterConfig).setWorkerConfig(workerConfig).build
val cluster = Cluster.newBuilder().setClusterName(clusterName).setConfig(clusterConfig).build()
val createClusterAsyncRequest = clusterControllerClient.createClusterAsync(projectId, region, cluster)
val createResponse: Cluster = createClusterAsyncRequest.get()
println(s"Created cluster: ${createResponse.getClusterName}")
clusterControllerClient.close()
}
I'm getting io.grpc.StatusRuntimeException: PERMISSION_DENIED: Required 'compute.regions.get' permission for 'projects/my-project/regions/europe-west1'.
I'm unclear as to exactly what is meant here: https://github.com/googleapis/java-dataproc#authorization. I'm trying to get this to work from my desktop so what I've done is run gcloud auth application-default login --scopes https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/compute.readonly.
I'm certain my 'normal' user has the necessary permissions as I've executed a 'regions.get' for my project/region from this page: https://cloud.google.com/compute/docs/reference/rest/v1/regions/get, and can create dataproc clusters not using the java library without issue.
I'm clearly missing something, probably something obvious, but am stuck so any help will be greatly appreciated!
Edit 1:
gcloud auth application-default login without specifying --scopes results in the same permission error
Edit 2:
I'm still none the wiser as to why I'm getting the compute.regions.get permission missing error.
I've written some more code which appears to show I do have the necessary permission when using getApplicationDefault:
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport
import com.google.api.client.json.gson.GsonFactory
import com.google.api.services.compute.Compute
import com.google.auth.http.HttpCredentialsAdapter
import com.google.auth.oauth2.GoogleCredentials.getApplicationDefault
object GetRegions extends App {
val project = "my-project-id"
val region = "europe-west1"
val httpTransport = GoogleNetHttpTransport.newTrustedTransport
val jsonFactory = GsonFactory.getDefaultInstance
val httpCredentials = new HttpCredentialsAdapter(getApplicationDefault)
val computeService = new Compute.Builder(httpTransport, jsonFactory, httpCredentials).build
val request = computeService.regions.get(project, region)
val response = request.execute
System.out.println(response) // This successfully prints out details
}
Turned out it was the permissions of the dataproc service agent that were missing this permission, not my user (these had been modified from default permissions).
See https://cloud.google.com/dataproc/docs/concepts/iam/dataproc-principals#service_agent_control_plane_identity.
Not explicitly setting the zone when creating a cluster seems to mean that this service account requires the compute.regions.get permission. Explicitly setting a zone meant it didn't.
Related
I know that there are many similar questions, and this one is no exception
But unfortunately I can't decide on the region for my case, how can I decide on the right region?
For example, when making a request to Postman, I encounter a similar error:
In my console i'm using EU (Frankfurt) eu-central-1 and also in terminal write smth like this:
heroku config:set region="eu-central-1"
And as I understand it, mine does not fit.
Also here is my AWS class:
class AmazonFileStorage : FileStorage {
private val client: S3Client
private val bucketName: String = System.getenv("bucketName")
init {
val region = System.getenv("region")
val accessKey = System.getenv("accessKey")
val secretKey = System.getenv("secretKey")
val credentials = AwsBasicCredentials.create(accessKey, secretKey)
val awsRegion = Region.of(region)
client = S3Client.builder()
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.region(awsRegion)
.build() as S3Client
}
override suspend fun save(file: File): String =
withContext(Dispatchers.IO) {
client.putObject(
PutObjectRequest.builder().bucket(bucketName).key(file.name).acl(ObjectCannedACL.PUBLIC_READ).build(),
RequestBody.fromFile(file)
)
val request = GetUrlRequest.builder().bucket(bucketName).key(file.name).build()
client.utilities().getUrl(request).toExternalForm()
}
}
I think you may have the wrong region code; you do know that a Bucket is available in one and only one Region?
In your logging settings, set this scope to debug:
logging:
level:
org.apache.http.wire: debug
Then you should see something like this:
http-outgoing-0 >> "HEAD /somefile HTTP/1.1[\r][\n]"
http-outgoing-0 >> "Host: YOURBUCKETNAME.s3.eu-west-2.amazonaws.com[\r][\n]"
That log is from a bucket in the London region eu-west-2
To use Kotlin to interact with an Amazon S3 bucket (or other AWS services), consider using the AWS SDK for Kotlin. This SDK is meant for Kotlin developers. You are using the AWS SDK for Java.
To put an object into an Amazon S3 bucket using the AWS SDK for Kotlin, use this code. Notice the region that you want to use is specified in the code block where you define the aws.sdk.kotlin.services.s3.S3Client.
import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.PutObjectRequest
import aws.smithy.kotlin.runtime.content.asByteStream
import java.io.File
import kotlin.system.exitProcess
/**
Before running this Kotlin code example, set up your development environment,
including your credentials.
For more information, see the following documentation topic:
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/setup.html
*/
suspend fun main(args: Array<String>) {
val usage = """
Usage:
<bucketName> <objectKey> <objectPath>
Where:
bucketName - The Amazon S3 bucket to upload an object into.
objectKey - The object to upload (for example, book.pdf).
objectPath - The path where the file is located (for example, C:/AWS/book2.pdf).
"""
if (args.size != 3) {
println(usage)
exitProcess(0)
}
val bucketName = args[0]
val objectKey = args[1]
val objectPath = args[2]
putS3Object(bucketName, objectKey, objectPath)
}
suspend fun putS3Object(bucketName: String, objectKey: String, objectPath: String) {
val metadataVal = mutableMapOf<String, String>()
metadataVal["myVal"] = "test"
val request = PutObjectRequest {
bucket = bucketName
key = objectKey
metadata = metadataVal
body = File(objectPath).asByteStream()
}
S3Client { region = "us-east-1" }.use { s3 ->
val response = s3.putObject(request)
println("Tag information is ${response.eTag}")
}
}
You can find this Kotlin example and many more in the AWS Code Library here:
Amazon S3 examples using SDK for Kotlin
ALso you can read the Kotlin DEV guide too. The link is at the start of the Code Example.
I am using the terraform-google-project-factory module to create multiple GCP projects at once. The projects create just fine and I am using the included option to disable the default GCP compute service account and stand-up a new Service Account in each project.
The module has an "sa_role" input where I assign "roles/compute.admin" to the new S.A. However, I would also like to assign some additional IAM roles to that Service Account in the same deployment. The sa_role input seems to only take one string value:
module "project-factory" {
source = "terraform-google-modules/project-factory/google"
version = "12.0.0"
for_each = toset(local.project_names)
random_project_id = true
name = each.key
org_id = local.organization_id
billing_account = local.billing_account
folder_id = google_folder.DQS.id
default_service_account = "disable"
default_network_tier = "PREMIUM"
create_project_sa = true
auto_create_network = false
project_sa_name = local.service_account
sa_role = ["roles/compute.admin"]
activate_apis = ["compute.googleapis.com","storage.googleapis.com","oslogin.googleapis.com",]
}
The output for the Service Account email looks like this:
output "service_account_email" {
value = values(module.project-factory)[*].service_account_email
description = "The email of the default service account"
}
How can I add additional IAM roles to this Service Account in the same main.tf ? This Stack article comes close to what I wish to achieve:
Want to assign multiple Google cloud IAM roles against a service account via terraform
However, I do not know how to reference my Service Account email addresses from the outputs.tf to make them available to the members = part of the data google_iam_policy. My question is, how to get this to work with the data google_iam_policy, or is there another better way to do this?
I am trying to launch a Dataflow job using the python google api client libraries. Everything worked fine previously, until we had to migrate from default subnetwork to another private subnetwork. Previously I was launching a dataflow job with the following code:
request = dataflow.projects().locations().templates().launch(
projectId = PROJECT_ID,
location = REGION,
gcsPath = TEMPLATE_LOCATION,
body = {
'jobName': job_name,
'parameters': job_parameters,
}
)
response = request.execute()
However the job now will fail because the default subnetwork does not exist anymore, and I now need to specify to use data-subnet subnetwork.
From this documentation and also this other question, the solution would be trivial if i were to launch the script from command line by adding the flag --subnetwork regions/$REGION/subnetworks/$PRIVATESUBNET. However my case is different becuase I am trying to do it from code, and in the documentation I can't find any subnet parameter option.
You can specify a custom subnetwork like so to your pipeline
request = dataflow.projects().locations().templates().launch(
projectId = PROJECT_ID,
location = REGION,
gcsPath = TEMPLATE_LOCATION,
body = {
'jobName': job_name,
'parameters': job_parameters,
'environment': {
'subnetwork': SUBNETWORK,
}
}
)
response = request.execute()
Make sure SUBNETWORK is in the form "https://www.googleapis.com/compute/v1/projects/<project-id>/regions/<region>/subnetworks/<subnetwork-name>"
basically what I trying to do is creating a message on Pub/Sub that triggers a GCF which creates a instance from a Regional Managed Instance Group in whatever available zone it has at the time.
The issue I'm trying to solve here is a rather recurrent ZONE_RESOURCE_POOL_EXHAUSTED which the regional MIG deals with.
Is this solution possible? I've tried using createInstances method but Logging just states PRECONDITION_FAILED.
The code snippet I'm using is as follows:
from googleapiclient import discovery
def launch_vm(project, region, igm, body)
service = discovery.build('compute', 'v1')
response = service.regionInstanceGroupManagers()\
.createInstances(
project=project,
region=region,
instanceGroupManager=igm,
body=body)
return response.execute()
request_body = {"instances":[{"name": "testinstance"}]}
launch_vm('project-name', 'us-central1', 'instace-group-name', request_body)
####### EDIT :
I just found out what happened, when I tried on another project with a recently created instance group, I found out that instance redistribution was enabled, which can NOT be the case as with the response from the CLI:
ERROR: (gcloud.compute.instance-groups.managed.create-instance) CreateInstances can be used only when instance redistribution is disabled (set to NONE).
I checked out the instance redistribution check and now it works wonders :) Thanks everyone for the help!
I'm able to createInstance:
import os
from googleapiclient import discovery
PROJECT = os.environ["PROJECT"]
REGION = os.environ["REGION"]
NAME = os.environ["NAME"]
service = discovery.build('compute', 'v1')
def launch_vm(project,region, name, body):
rqst = service.regionInstanceGroupManagers().createInstances(
project=project,
region=region,
instanceGroupManager=name,
body=body)
return rqst.execute()
body = {
"instances": [
{
"name": "testinstance"
}
]
}
launch_vm(PROJECT, REGION, NAME, body)
I am developing an application in Java and it requires the user to have a policy document. The user enters the access key and secret key. I got AmazonIdentityManagementClient object using the credentials. My application requires "lambda:InvokeFunction". Can any one pls guide me how to check the user policy has lambdainvoke.
Try below code to get the attached policy as a string.
AmazonIdentityManagementAsync iam = AmazonIdentityManagementAsyncClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(
new BasicAWSCredentials("",
"")))
.withRegion(Regions.fromName(""))
.withClientConfiguration(getClientConfiguration()).build();
ListAttachedUserPoliciesRequest pre = new ListAttachedUserPoliciesRequest();
pre.setUserName(iam.getUser().getUser().getUserName());
ListAttachedUserPoliciesResult re = iam.listAttachedUserPolicies(pre);
re.getAttachedPolicies().forEach(p -> {
GetPolicyRequest preq = new GetPolicyRequest();
preq.setPolicyArn(p.getPolicyArn());
GetPolicyResult r = iam.getPolicy(preq);
GetPolicyVersionRequest req = new GetPolicyVersionRequest();
req.setPolicyArn(p.getPolicyArn());
req.setVersionId(r.getPolicy().getDefaultVersionId());
GetPolicyVersionResult res = iam.getPolicyVersion(req);
System.out.println(URLDecoder.decode(res.getPolicyVersion().getDocument()));
});
You can use AmazonIdentityManagementClient.listAttachedUserPolicies() to list the policies attached to a user. This will get you to a list of policy ARNs which you can pass to AmazonIdentityManagementClient.getPolicy().