linking to a google cloud bucket file in a terminal command? - google-cloud-platform

I'm trying to find my way with Google Cloud.
I have a Debian VM Instance that I am running a server on. It is installed and working via SSH Connection in a browser window. The command to start the server is "./ninjamsrv config-file-path.cfg"
I have the config file in my default google firebase storage bucket as I will need to update it regularly.
I want to start the server referencing the cfg file in the bucket, e.g:
"./ninjamsrv gs://my-bucket/ninjam-config.cfg"
But the file is not found:
error opening configfile 'gs://my-bucket/ninjam-config.cfg'
Error loading config file!
However if I run:
"gsutil acl get gs://my-bucket/"
I see:
[
{
"entity": "project-editors-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "editors"
},
"role": "OWNER"
},
{
"entity": "project-owners-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "owners"
},
"role": "OWNER"
},
{
"entity": "project-viewers-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "viewers"
},
"role": "READER"
}
]
Can anyone advise what I am doing wrong here? Thanks

The first thing to verify is if indeed the error thrown is a permission one. Checking the logs related to the VM’s operations will certainly provide more details in that aspect, and a 403 error code would confirm if this is a permission issue. If the VM is a Compute Engine one, you can refer to this documentation about logging.
If the error is indeed a permission one, then you should verify if the permissions for this object are set as “fine-grained” access. This would mean that each object would have its own set of permissions, regardless of the bucket-level access set. You can read more about this here. You could either change the level of access to “uniform” which would grant access to all objects in the relevant bucket, or make the appropriate permissions change for this particular object.
If the issue is not a permission one, then I would recommend trying to start the server from the same .cfg file hosted on the local directory of the VM. This might point the error at the file itself, and not its hosting on Cloud Storage. In case the server starts successfully from there, you may want to re-upload the file to GCS in case the file got corrupted during the initial upload.

Related

Unable to set GOOGLE_APPLICATION_CREDENTIALS in pyspark shell

I am trying to access GCS from pyspark running in docker and for that I have the json file which copied in docker container .
spark._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark._jsc.hadoopConfiguration().set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
# This is required if you are using service account and set true,
spark._jsc.hadoopConfiguration().set("fs.gs.auth.service.account.enable", "true")
then setting up the GOOGLE_APPLICATION_CREDENTIALS like below -
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS']=r"abc.json"
Now when trying to access gcs objects like below it's throwing error -
df= spark.read.csv("gs://integration-o9/california_housing_train.csv")
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "**111111-compute#developer.gserviceaccount.com** does not have storage.objects.get access to the Google Cloud Storage object. Permission 'stora
ge.objects.get' denied on resource (or it may not exist).",
"reason" : "forbidden"
} ],
"message" : "**111111-compute#developer.gserviceaccount.com** does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage
.objects.get' denied on resource (or it may not exist)."
}
This is not the service account mentioned in the Json file , but if I set this via below -
export GOOGLE_APPLICATION_CREDENTIALS=abc.json
it's working fine
Any advice what's to look into , need to make it work via os.environ properties
Posting the answer here as below is working (need to set in the config part with .p12 key)
spark=SparkSession.builder.appName("test").config("google.cloud.auth.service.account.enable", "true").config("google.cloud.auth.service.account.email", "o9int-759#integration-3-344806.iam.gserviceaccount.com").config("google.cloud.auth.service.account.keyfile", "/path/file.p12").getOrCreate()

How can I troubleshoot Dockerrun parsing errors?

I'm banging my head against a wall trying to figure out the source of the following error I get when trying to deploy this Dockerrun file to EB:
Error: parse Dockerrun.aws.json file failed with error json: invalid use of ,string struct tag, trying to unmarshal unquoted value into int
Here is the file in question:
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "mybucket",
"Key": "myconfig.json"
},
"Image": {
"Name": "1234567890.dkr.ecr.us-east-2.amazonaws.com/myimage:tag",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "3001",
"HostPort": "80"
}
]
}
I've read over the documentation here: https://docs.amazonaws.cn/en_us/elasticbeanstalk/latest/dg/single-container-docker-configuration.html
I can't seem to find any issues with the file. I know that AWS has validators for CloudFormation templates, does something similar exist for Dockerrun files? How would one go about troubleshooting this error?
It turns out the error was unrelated to the actual parsing of the file. I dug through the logs and realized that my ECR authentication token had supposedly expired. This was strange since I was using the same ECR authentication for other Elastic Beanstalk environments without issue. The solution was to generate a new authentication token for ECR, upload a new config file to S3, and point the Dockerrun authentication bucket and key fields to the new file.
If you run into a similar error, look further back in your eb-engine logs for other errors that may be the root of the problem.

Required 'compute.regions.get' permission for 'projects/$project_id/regions/us-central1

I'm pretty new on Google cloud, and I'm trying to use GCloud command line, and I faced the following problem
Error: Forbidden access to resources.
Raw response:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "forbidden",
"message": "Required 'compute.regions.get' permission for 'projects/$project_id/regions/us-central1'"
}
],
"code": 403,
"message": "Required 'compute.regions.get' permission for 'projects/$project_id/regions/us-central1'"
}
}
Can someone help?
Much appreciated
To troubleshoot your issue, please try following:
Where are you running the command: Cloud Shell, Local environment?
If it is local environment, try Cloud Shell instead.
Check that you are using the latest version of gcloud sdk 262.
Did you properly initialize gcloud?
Can you confirm that you have appropriate role to run the command, like editior/owner?
Check if you are using that same location for your products
If above steps don't work, can you share your complete gcloud command to have more context?
Oh, I see where the problem is! When I created the storage, I put the region as "Asia". When I configured it via gcloud init, I put it as "us-central1-a". The "Permission denied" means in this context, I have no permission to access another server region. It is misleading in terms of thinking out the cloud scope. However, the Pawel's answer is more comprehensive, and it is a very good start to lead you to the correct direction.

Deploy Google Cloud Function from Cloud Function

Solved/invalid - see below
I'm trying to deploy a Google Cloud Function from a Google Cloud Function on demand.
However, whatever I try, I get a 403 Forbidden:
HttpError 403 when requesting https://cloudfunctions.googleapis.com/v1/projects/MY_PROJECT/locations/MY_REGION/functions?alt=json returned "The caller does not have permission"
I ended up granting the cloud function service account Project Owner role to make sure it can do anything, yet still I get the same error.
Is this limited intentionally (for example to avoid fork bombs or something) or am I doing something wrong?
Has anyone been able to make this work?
For the record: I ran the same (Python) function locally with Flask using my own account and then it will deploy the new cloud function perfectly, so the code itself seems to be ok.
Update
Code snippet of how I'm trying to deploy the cloud function:
cf_client = discovery.build('cloudfunctions', 'v1')
location = "projects/{MYPROJECT}/locations/europe-west1"
request = {
"name": "projects/{MYPROJECT}/locations/europe-west1/functions/hopper--2376cd24d318cd2d42f000f4f1c31a8f",
"description": "Hopper hopper--2376cd24d318cd2d42f000f4f1c31a8f",
"entryPoint": "pubsub_trigger",
"runtime": "python37",
"availableMemoryMb": 256,
"timeout": "60s",
"sourceArchiveUrl": "gs://staging.{MYPROJECT}.appspot.com/deployment/hopper.zip",
"eventTrigger": {
"eventType": "providers/cloud.pubsub/eventTypes/topic.publish",
"resource": "projects/{MYPROJECT}/topics/hopper-test-input"
},
"environmentVariables": {
"HOPPER_ID": "hopper--2376cd24d318cd2d42f000f4f1c31a8f"
}
}
response = cf_client.projects() \
.locations() \
.functions() \
.create(location=location, body=req) \
.execute()
Update
I feel like such an idiot... it turns out that for some reason I deployed the master function in a different project then the project I gave permissions on. No wonder it didn't work.
The correct answer should be: check that everything is indeed running how/where you expect it to be. Everything was configured correctly and deploying a CF in a CF is not a problem. The project was incorrect, due to a different default project being set on the gcloud utility.

Elasticsearch Snapshot Fails With RepositoryMissingException

Three node ElasticSearch cluster on AWS. Bigdesk and Head both show a healthy cluster. All three nodes are running ES 1.3, and the latest Amazon Linux updates. When I fire off a snapshot request like:
http://localhost:9200/_snapshot/taxanalyst/201409031540-snapshot?wait_for_completion=true
the server churns away for several minutes before responding with the following:
{
"snapshot": {
"snapshot": "201409031521-snapshot",
"indices": [
"docs",
"pdflog"
],
"state": "PARTIAL",
"start_time": "2014-09-03T19:21:36.034Z",
"start_time_in_millis": 1409772096034,
"end_time": "2014-09-03T19:28:48.685Z",
"end_time_in_millis": 1409772528685,
"duration_in_millis": 432651,
"failures": [
{
"node_id": "ikauhFYEQ02Mca8fd1E4jA",
"index": "pdflog",
"reason": "RepositoryMissingException[[faxmanalips] missing]",
"shard_id": 0,
"status": "INTERNAL_SERVER_ERROR"
}
],
"shards": {
"total": 10,
"failed": 1,
"successful": 9
}
}
}
These are three nodes on three different virtual EC2 machines, but they're able to communicate via 9300/9200 without any problems. Indexing and searching works as expected. There doesn't appear to be anything in the elasticsearch log files that speaks to the server error.
Does anyone know what's going on here, or at least where a good place to start would be?
UPDATE: Turns out that each of the nodes in the cluster need to have snapshot directories that match the directory specified when you register the snapshot with the elasticsearch cluster.
I guess the next question is: when you want to tgz up the snapshot directory so you can archive it, or provision a backup cluster, is it sufficient to just tgz the snapshot directory on the Master node? Or do you have to somehow consolidate the snapshot directories of all the nodes. (That can't be right, can it?)
Elasticsearch supports shared file system repository uses the shared file system to store snapshots.
In order to register the shared file system repository it is necessary to mount the same shared filesystem to the same location on all master and data nodes.
All you need to know Put in elasticsearch.yml of all 3 nodes same repository_name.
for eg:-
path.repo:[/my_repository]
I think you are looking for this aws plugin for elasticsearch (I guess you already installed it to configure your cluster) : https://github.com/elasticsearch/elasticsearch-cloud-aws#s3-repository
It will allow you to create a repository mapped to a S3 bucket.
To use (create/restore/whatever) a snapshot, you need to create a repository first. Then when you will do some actions on a snapshot, Elasticsearch will directly manage it on your S3 bucket.