[google-admin-sdk]
I am able to extract reports for each user but this process is taking too much time, as my organization has 5500+ users, and each user is taking about 11seconds to fetch data for.
Yes, I know we can export data directly by going on console and exporting it, but I want this automated through code.
I am using python with Google's reports API.Using Reports Api
Related
I have a Saas billing-model and each user has their own GCP Project. This is similar to this reddit thread, which asks:
I’m thinking about selling a saas service. I’ve decided every customer will get their own gcp project every customer will have a bunch of cloud run services, a cloud sql database and some users in Identity platform. I know the default project limit is around 12 and it can be increased by filling a form.
This works for something like BigQuery, where each user's Dataset or Table will be created within their own GCP project, and thus their billing (and data) will be segmented under their project.
However, I also have some shared endpoints on Google Cloud Functions, for example let's say I have general/shared endpoints to do something like "export data". Now of course the query to grab the data will hit the correct GCP project, but if the export (or some other data processing task) is doing something that is very expensive -- some exports might take over an hour to write the data, if dealing with billions of rows, what would be the suggested way to set that up so the end user is paying for their computation, since I imagine an endpoint such as www.example.com/api/export is just going to be on the main Project account, and we wouldn't have, for example, 1000 different cloud functions that do the same thing just to have each one under their respective project.
What might be a solution to this? In a way I'm looking for something like this I suppose where the requestor pays.
You would probably need to record how long each function call took, and save that data somewhere before exiting the shared function.
The only alternative would be to split the function for each client, and use billing labels to help with allocation.
I'm trying to find a way to retrieve a specific Google Merchant Center report (BestSellers_TopProducts_) and upload it to BigQuery as part of a specific ETL process we're developing for a customer we have at my workplace.
So far, I know you can set up the BigQuery Data Transfer service so it automates the process of downloading this report but I was wondering if I could accomplish the same with Python and some API libraries from Google (like python-google-shopping) but I may be overdoing it and setting up the service is the way to go.
Is there a way to accomplish this rather than resorting to the aforementioned service?
On the other hand, and assuming the BigQuery Data Transfer service is the way to go, I see (in the examples) you need to create and provide the dataset you're going to extract the report data to so I guess the extraction is limited to the GCP project you're working with.
I mean... you can't extract the report data for a third-party even if you had the proper service account credentials, right?
I'm looking to get help on the GCP billing. I know we can get cost info based on the service and project, however, is it possible to get info based on the access email ID? because I'm planning to give access to my colleagues and I want to know how much each one their access cost and against which service.
Something like: Date, Email ID, Service, Cost
With respect to another project, how should we know which access cost us so much?
We are running ~30 sandbox projects internally, each allocated to a specific person that can test and run his/her stuff on GCP.
I strongly suggest you create isolated workspaces (projects) for your colleagues so they don't accidentally delete/update services of other people. You will get a separate billing report for each project as well.
I am also setting up a billing alert for all my colleagues so they get an early notification if they left something running on their testbench.
There are three ways I think you could do that kind of cost segregation, I will number them in order of complexity.
1.- Cloud Export Billing, For this one the best practice is to segregate your resources and users by "Labels", as administrator, you may ask the users to use them and assign them to any resource they create, e.g. If they create a new VM instance, then you will be able to filter by field the exported table and create the reports as you want.(Also your GCP billing dashboard will show these "labels" segregations)
2.- Use Billing API to curl directly the information you need to get from it,you can manage to use in the request the information you need like SKU, User, Date and description.
3.- Usage Reports. This solution is more GSuite scope,and I can't vouch that will work as the documentation say but you can take a look to it, there is an option to get "Usage reports", this usage reports can be made from GSuite to any resource below, GCP included if you already have an organization.
I am working on a health care project we have a device which continiously generates values for the fields ACTIVITY AND FREQUENCY .The values need to be updated continously from python to google fusion table.
The question is quite broad, you probably want to have a look at the documentation of the Google Fusion Tables API if you haven't so far: https://developers.google.com/fusiontables/docs/v1/using
Also it may be worth checking the quota section to make sure that Google Fusion Tables is indeed what you want to use:
https://developers.google.com/fusiontables/docs/v1/using#quota
I'll be glad to try to help if you come up with more specific questions :)
EDIT: since there are quite a few questions around the topic, I'll add some "hints".
A (Google Fusion) table belongs to a Google account. Your script must therefore include a step where it asks for your permission to modify data attached to your Google Account. You can therefore see your script as a web application which needs an authorization to achieve its goal. This web application will use the Google Fusion Tables API and therefore it must be registered in the Google API Console. You will find details about the process of registration and authentication with a Python script here:
https://developers.google.com/fusiontables/docs/articles/oauthfusiontables?hl=fr
I just checked that this works and you can insert rows to a table thereafter, so you may want to have a quick look at my script. Note that you can neither use my application credentials (which are by the way not included) nor my table as you are not authorized to edit it (it's mine!). So you must download your application credentials from the Google API console after having registered and adapt the script so it loads your credentials. Also, the script does not create a table (as of now) so as a first step you can create a table with two columns in the UI and copy paste the table id in the script so it will know in which table to write. Here's the script (sorry it's a bit of a mess right now, I'll do as soon as I can):
https://github.com/etiennecha/master_code/blob/master/code_fusion_tables/code_test_fusion_tables.py
Hope this helps.
Nowadays a lot of web applications are providing API for other applications to use.
I am new to the usage of API so I want to understand the use cases for it.
Lets take Basecamp as an example.
What are the use cases for using their API in my web application?
For inserting current data in my web application into a newly created Basecamp account instead of inserting everything manually which could take days or weeks if the data is huge?
For updating my application data when the user changes something in Basecamp. If so, how do I know for example when a user add/edit/remove a contact in Basecamp. Do I make a request and check every minute from the backend?
For making backup of the Basecamp data so I can move it to other applications if necessary?
Are all the above examples good use cases for the usage of API?
Are there more use cases?
I want to have a clear picture of why it's good to use another web service API and how I can leverage that on my application.
Thanks.
I've found the biggest reason to use and provide web services is to be able to programmatically drive the application with another process. This allows the coupling of different actions in different applications driven by one event/process/trigger.
For example I could create a use a webservice provided by Basecamp, my bug tracking database and the continuous integration server. I could tie all those things together and kick them off from a commit hook script.
I can have a monitor in production automatically open a ticket in our ticket tracker. This could trigger an autoremediation process from the ticket tracker which logs into the box remotely and restarts the service.
The other major reason I've seen to use and provide web service is to reduce double entry. If you do change management in your production environment that usually means you create Change tickets. The changes that occur may also need to be reflected in the Change Management Database which is usually a model of how production is suppose to look. Most of these systems don't automatically drive the update of your configuration item with the data from the change. Using web services you can stitch them together to eliminate the double (manual) entry that would normally occur.
APIs are used any time you want to get data to/from an application without using the default interface.
*I'd bet there's a mobile app would use the basecamp api.
*You could use the api to pull information from basecamp into another application (like project manager software or an individual's todo webpage)
*the geekiest of us may prefer to update basecamp from a script/command line rather than interrupting our work flow to open a web page and click around.