Hi I am trying to upload a file from GCS to Gdrive using
airflow.contrib.operators.gcs_to_gdrive_operator import GcsToGDriveOperator
This is how the dag looks like
copy_to_gdrive = GcsToGDriveOperator(
task_id="copy_to_gdrive",
source_bucket="my_source_bucket_on_gcs",
source_object="airflow-dag-test/report.csv",
destination_object="/airflow-test/report.csv",
gcp_conn_id="bigquery_default",
dag=dag
)
This code executes without any errors and in the logs I can see the file is downloaded to local successfully and uploaded to gdrive successfully as well.
This code is executed by a service account, the issue i am facing is I am not able to find the file or the directory this dag is creating uploading
I have tried several permutation/combinations of path for "destination_object" but nothing seems to work also google docs are not helpful as well.
I can see in the api logs that that the drive.create api is being called but where it is creating the file is unknown. Has anyone experienced this ? any help or tip would be of great help. Thanks!
Your Service account is a Google account, and, as google account, it has access to its own drive. The file are correctly copied to Drive, but to the drive of the service account!
You never specify the account, so, how Airflow can know that it has to use yours?
Look at the operator documentation
delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.
Use this parameter, fill it with your email and activate the domain delegation wide to your service account.
Related
I have an account that is the verified owner for a property listed in the Google Search Console. I'm working to set up API access to it. I have a principal created that has access to each of the Google Cloud projects that I intend to query the data from. I've enabled the Google Search Console API for each of these projects.
After I authenticate the principal via OAuth2 and use the access token (including the 'https://www.googleapis.com/auth/webmasters.readonly' scope) to make a POST request to the query endpoint, I get an error message back that reads in part:
Google Search Console API has not been used in project 256595xxxxxx before or it is disabled
I searched this number in my Google Cloud Console and manually clicked through each of my properties and it doesn't match any of the projects on my account. How do I identify which project this number is referring to and/or how do I change it to point to one of my own Google Cloud properties so I might access its API?
Note: This answer speaks to what it is I'm trying to do (but with Google Search Console API), except that the error doesn't reflect my project ID, so I'm stumped about how to move forward with enabling the API on it.
This didn't take long to solve. After listing all the projects in the Google Cloud CLI per the first half of the instruction here, I was able to identify the project as that which is used as the service principal to do the authentication in the first place (and not actually an account I intend to export any of the data to).
it's my first time to use Google API, now I'm following this tutorial(https://how2electronics.com/diy-ai-camera-with-google-vision-esp32-cam-module/), but I couldn't get the information as tutorial. enter image description here , in my google cloud, it's just this.enter image description here
Now I installed the google could CLI but there still nothing under the set up authentication....
here is my terminal :
These credentials will be used by any library that requests Application Default Credentials (ADC).
Quota project "esp32-text-369110" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.
Thank you so much 😭
Can anyone tell me which part I missed...I really need the information under the set up authentication
I cannot seem to get past the Google Authorization pop that always returns Error 401: deleted_client.
I created a Google sheet script a while back and I need to get it working again. I made a GCP account and tried messing around with that for hosting the app but I realized I didn't need that so I deleted all of my projects and my billing account. But I still can't get past the Google Sheet error when it tries to authorize my account. This worked before I started anything with GCP and was fine.
I've seen other issues saying that there needs to have some fixes in the GCP account and that's fine and all but I do not want this connected to GCP at all. I will never use GCP again and I can't even find a way to shut down the account entirely without removing my Google account.
I've tried clearing all cache and cookies from chrome since the beginning of time like some others have suggested that doesn't fix the issue either.
Thanks in advance!
It's confusing but, if your script uses Google APIs you must associate the script with a Google Cloud Platform (GCP) project.
See Google Cloud Platform projects for an explanation of this relationship.
GCP projects are used to group enabled Google services, to provide identities and configure OAuth.
Please read the document but see (re)creating OAuth credentials.
You'll need to:
Create a GCP project
Enable Google services
Configure OAuth consent
DazWilkin lead me in the right direction!
The last thing that was needed was to go to the existing Google Sheet script, hit the gear in the left panel of the new editor, then change the GCP project number to match the GCP project I created with OAuth creds. Thanks again!
I am running several gcloud services which have assigned urls automatically in following format:
https://SERVICE_NAME-XXXXXXX-ew.a.run.app/
This is not particularly easy to work with and to pass these URLs to clients. Alternative is to use the custom domain, but this needs hardcoding subdomains within DNS records (as far as I understand) and I would like to avoid that and use the default URLs.
What is the best practice to work with these URLs? I can imagine keeping some mapping of service->URL and passing it to clients, but I would like to avoid reinventing the wheels.
Edit: I've released an external tool called runsd that lets you do this. Check it out: https://github.com/ahmetb/runsd
Thanks for this question! The "Service discovery by name" for Cloud Run is very much an active area of work. Though, there are no active timelines we can share yet.
You can see a prototype of me running this on Cloud Run here: https://twitter.com/ahmetb/status/1233147619834118144
APIs like Google Cloud Service Directory linked are geared more towards custom/DIY service discovery you might want to build to your RPC stack such as gRPC. It's more of a managed domain name directory, that you can integrate with your RPC.
If you are interested in participating an alpha for this feature in the future, drop me an email at ahmetb at google.
You can use a beta service Service Directory.
At service deployment
Create your service with a name and the URL as metadata
In your code
Request the service metadata with its name, and get the URL
Use the url
You can't use the endpoint feature of the service because your don't have IP/Port.
However, for now, there is client library and you have to use API directly.
Does anybody have information on how to make sinch framework upload the voice call recordings to AWS s3?
I've created an IAM user on AWS for this, but could not find where to set the AWS credentials so that Sinch uploads the call recording automatically. Is it done on the client side, i.e. IOS code, or done by Sinch team manually? Do we need to change anything on the client side for this behaviour?
Please let me know if you have any information regarding this.
Kind Regards,
Engin
It cannot be set yourself. To do so, send an email to support#sinch.com.