I am working on a health care project we have a device which continiously generates values for the fields ACTIVITY AND FREQUENCY .The values need to be updated continously from python to google fusion table.
The question is quite broad, you probably want to have a look at the documentation of the Google Fusion Tables API if you haven't so far: https://developers.google.com/fusiontables/docs/v1/using
Also it may be worth checking the quota section to make sure that Google Fusion Tables is indeed what you want to use:
https://developers.google.com/fusiontables/docs/v1/using#quota
I'll be glad to try to help if you come up with more specific questions :)
EDIT: since there are quite a few questions around the topic, I'll add some "hints".
A (Google Fusion) table belongs to a Google account. Your script must therefore include a step where it asks for your permission to modify data attached to your Google Account. You can therefore see your script as a web application which needs an authorization to achieve its goal. This web application will use the Google Fusion Tables API and therefore it must be registered in the Google API Console. You will find details about the process of registration and authentication with a Python script here:
https://developers.google.com/fusiontables/docs/articles/oauthfusiontables?hl=fr
I just checked that this works and you can insert rows to a table thereafter, so you may want to have a quick look at my script. Note that you can neither use my application credentials (which are by the way not included) nor my table as you are not authorized to edit it (it's mine!). So you must download your application credentials from the Google API console after having registered and adapt the script so it loads your credentials. Also, the script does not create a table (as of now) so as a first step you can create a table with two columns in the UI and copy paste the table id in the script so it will know in which table to write. Here's the script (sorry it's a bit of a mess right now, I'll do as soon as I can):
https://github.com/etiennecha/master_code/blob/master/code_fusion_tables/code_test_fusion_tables.py
Hope this helps.
Related
I have a Saas billing-model and each user has their own GCP Project. This is similar to this reddit thread, which asks:
I’m thinking about selling a saas service. I’ve decided every customer will get their own gcp project every customer will have a bunch of cloud run services, a cloud sql database and some users in Identity platform. I know the default project limit is around 12 and it can be increased by filling a form.
This works for something like BigQuery, where each user's Dataset or Table will be created within their own GCP project, and thus their billing (and data) will be segmented under their project.
However, I also have some shared endpoints on Google Cloud Functions, for example let's say I have general/shared endpoints to do something like "export data". Now of course the query to grab the data will hit the correct GCP project, but if the export (or some other data processing task) is doing something that is very expensive -- some exports might take over an hour to write the data, if dealing with billions of rows, what would be the suggested way to set that up so the end user is paying for their computation, since I imagine an endpoint such as www.example.com/api/export is just going to be on the main Project account, and we wouldn't have, for example, 1000 different cloud functions that do the same thing just to have each one under their respective project.
What might be a solution to this? In a way I'm looking for something like this I suppose where the requestor pays.
You would probably need to record how long each function call took, and save that data somewhere before exiting the shared function.
The only alternative would be to split the function for each client, and use billing labels to help with allocation.
[google-admin-sdk]
I am able to extract reports for each user but this process is taking too much time, as my organization has 5500+ users, and each user is taking about 11seconds to fetch data for.
Yes, I know we can export data directly by going on console and exporting it, but I want this automated through code.
I am using python with Google's reports API.Using Reports Api
actually I am working with PDI 8.2, however I am able to upgrade to 9.0.
The main issue is that a customer wants to pull data from salesforce which works well so far. But he is using the Enterprise Web Services API with version 48.0, latest Pentaho supports v47.0 only.
I strongly assume that reading via v48.0 won't work with PDI so that I have to build a workaround. Could anyone point me to a feasible solution? To be honest, I don't even know whether the Enterprise or the Partner API is relevant for Pentaho. Have got my own SF-Account so that I could try around with the APIs.
Is the "Web Services lookup" the right step for the workaround?
Any answer would be appreciated! Thanks in advance!
Oh man, what a crazy question, all over the place.
I strongly assume that reading via v48.0 won't work
You'd have to try it but it should work. Salesforce has 3 releases a year and that's when they upgrade API versions. We're in Spring'20 now, it's v.48. That doesn't mean anything below is deprecated. You should have no problems calling with any API version >= 20. From what I remember their master service agreement states that API version released will stay up at least 3 years. Well, v.20 is 9 years old and still going strong...
Check for example https://baa.my.salesforce.com/services/data/ (if your client has "My Domain" enabled you can use that too instead of some unknown company), you should see a list similar to this: https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_versions.htm (no login required, that'd be a chicken & egg situation. You need to choose API version you want when making the login call).
So... what does your integration do. I assume it reads or writes to SF tables (objects), pretty basic stuff. In that sense the 47 vs 48 won't matter much. You should still see Accounts, Contacts, custom objects... You won't see tables created specifically in v 48. Unless you must see something mentioned in Spring'20 release notes I wouldn't worry too much.
If your client wrote a specific class (service) to present you with data and it's written in v.48 it might not be visible when you login as v.47. But then they can just downgrade the version and all should be well. Such custom services are rarely usable by generic ETL tools anyway so it'd be a concern only if you do custom coding.
whether the Enterprise or the Partner API is relevant for Pentaho
Sounds like your ETL tool uses SOAP API. Salesforce offers 2 versions of the WSDL file with service definitions.
"Partner" is generic, all SF orgs in the world produce same WSDL file. It doesn't contain any concrete info about tables, columns, custom services written on top of vanilla salesforce. But it does contain info how to call login() or run a "describe" that gives you all tables your user can see, what are their names, what are columns, data types... So you learn stuff at runtime. "Partner" is great when you're building a generic reusable app that can connect to any SF or you want to be dynamic (some backup tool that learns columns every day and can handle changes without problems. Or there's a "connection wizard" where you specify which tables, which columns, what mapping... new field comes in - just rerun the wizard).
"Enterprise" will be specific to this particular customer. It contains everything "Partner" has but will also have description of current state of database tables etc. So you don't have to call "describe", you already have everything on the plate. You can use this to "consume" the WSDL file, generate your Java/PHP/C# classes out of it and interact with them in your program like any other normal object instead of crafting XML messages.
The downside is that if new field or new table is added - you won't know if your program doesn't call "describes". You'd need to generate fresh WSDL and consume it again and recompile your program...
Picking right one really depends what you need to do. ETL tools I've met generally are fine with "partner".
Is the "Web Services lookup" the right step
No idea, I've used Informatica, Azure Data Factory, Jitterbit, Talend... but no idea about this Pentaho thing. Just try it. If you pull data straight from SF tables without invoking any custom code (you can think of SF custom services like pulling data from stored procedures) - API version shouldn't matter that much. If you go < 41.0 I believe you won't see Individual object for example but I doubt you need to be on so much cutting edge.
I 'm not sure if this is the correct platform to ask architecture related question, actually I have a webapplication developed in nodejs & typescript hosted in AWS, and the backend is mongodb and my requirement is to include a search box with wild card & auto suggest search functionality so when I start typing on the text box, it will autosuggest just like we do in google search, so how would I achieve this, querying everytime to mongodb will be kind of slow and if 100's of user start doing that, then my application might start dangling so need your suggestion.
Not tried as this more of architecture help required
Not tried as this more of architecture help required
It's not a very detailed answer but may point you in a direction.
I just built something similar using AWS Lambda, ElasticSearch and API Gateway.
ElasticSearch is great for text searches but needs to be populated with indexed data.
If your dataset is changing, you will have to remember about updating ElasticSearch.
API Gateway routes requests from HTTP to Lambda, of which there are two:
one for analysing data in my data warehouse and producing indices for ElasticSearch, the other for doing the actual search and returning results.
I have just recently started to work with Google Cloud and I am trying to wrap my head around some of its inner workings, mainly the audit logging part.
What I want do is get the log activity from when my keys are used for anything and also when someone actually logged into the Google Console Cloud (it could be the Key Vault or the Key Ring, too).
I have been using power shell to extract these logs using gcloud read logging and this is where I start to doubt whether I have the right place. I will explain:
I have created new keys and I see in the Activity Panel this action, and I can already extract this through gcloud read logging resource.type=cloudkms_cryptokey (there could be a typo on the command line, since I am writing it from the top of my head, sorry for that!).
Albeit I have this information, I am rather curious if this is the correct course of action here. I saw the CreateCryptoKey and SetIamPolicy methods on my logs, alright, but am I going to see all actions related to these keys? By reading the GCloud docs, I feel as though I am only getting some of the actions?
As I have said, I am trying to work my way around the GCloud Documentation, but it is such an overwhelming amount of information that I am not really getting the proper answer I am looking for, this is why I thought about resorting to this community.
So, to summarize, am I getting all the information related to my keys the way I am doing right now? And what about the people that have access to the Google Cloud Console page, is there a way to find who accessed it and which part (Crypto Keys page, Crypto Vault page for example)? That's something I have not understood from the docs as well, sadly. Perhaps someone could show me the proper page where I can make references to what I am looking for? Because the Cloud Audit Logging page doesn't feel totally clear to me on this front (and I assume I could be at fault here, these past weeks have been harsh!)
Thanks for anyone that takes some time to answer my question!
Admin activities such as creating a key or setting IAM policy are logged by default.
Data access activities such as listing Cloud KMS resources (key rings, keys, etc.), or performing cryptographic operations (encryption, decryption, etc.) are not logged by default. You can enable data access logging, via the steps at https://cloud.google.com/kms/docs/logging. I'm not sure if that is the topic you are referring to, or https://cloud.google.com/logging/docs/audit/.