Google Batch Geocoding API - geocoding

I am using Google Geocoding API to conduct both forward and reverse geocoding work in my projects. While this API is only working for a single address or a single pair of geo coordinates per request, not for batch processing. I know I can copy and paste lines of addresses up to 250 to the Batch Geo Website to process, but it would be so limited and ineffectively to me if I did that. Initially, I called REST API by PHP curl in my codes, while for batch data I have no idea how to handle and I did not find the Batch Processing API document defining call URL, parameters, response, or status. Is there any resource I can find concerning Batch Geocoding API? By the way, I found the same issue on Timezone API.

Google Maps APIs support 1 query per request. To batch geocoding multiple address, you need to send 1 request per address. Implementation is entirely up to you.

As miguev mentioned, Google does not allow batch geocoding. However, HERE does have a batch geocoder that you can use go geocode multiple addresses in one call.

I think there is some misunderstanding when you use the word "batch geocoding".
Sending one request at the time is the way to go to implement batch geocoding service. You think the vendor would handle the batch, where in your case, the vendor lets you implement it locally.
Because your volume amount is so small,simply create a loop that runs all of your rows, one at the time, send them to the API, and gets the result back.
There are plenty API for batch geocoding our there and you should be ok using any of them.
In your case it would be like that:
start loop until end of collection
$ curl -X GET 'https://csv2geo.com/api/geocode?id={email}&key={key}&searchtext={search_text}'
get response back
{
"hum_address": "781 Tremont Street,Boston,MA,2118",
"address_components": {
"number": "781",
"street": "Tremont St",
"city": "Boston",
"state": "MA",
"country": "USA",
"zip": "02118"
},
"full_address": "781 Tremont St, Boston, MA 02118, United States",
"lat": 42.33957,
"lon": -71.08034,
"relevance": 1
}
and store it in a array collection
end look

Related

Dialogflow ES dynamic entity or training phrase

I'm using Cisco CVP IVR to integrate with DialogflowES and Google cloud Speech to text. My IVR collected the caller's account number and retrieved the associated caller's name from a database. Now I want the caller to tell me their name and I want to see if it matches that from the database.
Since there are millions of crazy names/spellings, using an Entity Type of #sys.person or #sys.any isn't very useful.
So, I want to pass the name to Google almost like a 'dynamic grammar' (perhaps along with a few other decoy names), and have Google try to match what the caller says to one of these entries.
Is that possible to pass Google Dialogflow ES agent a list of names (I guess in a payload) and have that be used to build (or expand on) an Entity dynamically during the call?
I haven't used API.AI but is it possible to use fulfillment to create something like this dynamically at runtime based on what's passed in from my IVR? This sample is based on Dialogflow Training Phrases through Dialogflow REST API:
"trainingPhrases": [
{
"name":"",
"type": "EXAMPLE",
"parts": [
{
"text": "<<use a name1 passed from the IVR during the call>>"
},
]
}
]
I've tried modifying fulfillment, and I can at least access what the caller said and what the payload passed in was (the actual person's name). But I don't know how to limit what the agent is listening for to be a small list of names passed in.

Can you start AI platform jobs from HTTP requests?

I have a web app (react + node.js) running on App Engine.
I would like to kick off (from this web app) a Machine Learning job that requires a GPU (running in a container on AI platform or running on GKE using a GPU node pool like in this tutorial, but we are open to other solutions).
I was thinking of trying what is described at the end of this answer, basically making an HTTP request to start the job using project.job.create API.
More details on the ML job in case this is useful: it generates an output every second that is stored on Cloud Storage and then read in the web app.
I am looking for examples of how to set this up? Where would the job configuration live and how should I set up the API call to kick off that job? Are the there other ways to achieve the same result?
Thank you in advance!
On Google Cloud, all is API, and you can interact with all the product with HTTP request. SO you can definitively achieve what you want.
I personally haven't an example but you have to build a JSON job description and post it to the API.
Don't forget, when you interact with Google Cloud API, you have to add an access token in the Authorization: Bearer header
Where should be your job config description? It depends...
If it is strongly related to your App Engine app, you can add it in App Engine code itself and have it "hard coded". The downside of that option is anytime you have to update the configuration, you have to redeploy a new App Engine version. But if your new version isn't correct, a rollback to a previous and stable version is easy and consistent.
If you prefer to update differently your config file and your App Engine code, you can store the config out of App Engine code, on Cloud Storage for instance. Like that, the update is simple and easy: update the config on Cloud Storage to change the job configuration. However there is no longer relation between the App Engine version and the config version. And the rollback to a stable version can be more difficult.
You can also have a combination of both, where you have a default job configuration in your App Engine code, and an environment variable potentially set to point to a Cloud Storage file that contain a new version of the configuration.
I don't know if it answers all your questions. Don't hesitate to comment if you want more details on some parts.
As mentionated, you can use the AI Platform api to create a job via a post.
Following is an example using Java Script and request to trig a job.
Some usefull tips:
Jobs console to create a job manually, then use the api to list this job then you will have a perfect json example of how to trig it.
You can use the Try this API tool to get the json output of the manually created job. Use this path to get the job: projects/<project name>/jobs/<job name>.
Get the authorization token using the OAuth 2.0 Playground for tests purposes (Step 2 -> Access token:). Check the docs for a definitive way.
Not all parameters are required on the json, thtas jus one example of the job that I have created and got the json using the steps above.
JS Example:
var request = require('request');
request({
url: 'https://content-ml.googleapis.com/v1/projects/<project-name>/jobs?alt=json',
method: 'POST',
headers: {"authorization": "Bearer ya29.A0AR9999999999999999999999999"},
json: {
"jobId": "<job name>",
"trainingInput": {
"scaleTier": "CUSTOM",
"masterType": "standard",
"workerType": "cloud_tpu",
"workerCount": "1",
"args": [
"--training_data_path=gs://<bucket>/*.jpg",
"--validation_data_path=gs://<bucket>/*.jpg",
"--num_classes=2",
"--max_steps=2",
"--train_batch_size=64",
"--num_eval_images=10",
"--model_type=efficientnet-b0",
"--label_smoothing=0.1",
"--weight_decay=0.0001",
"--warmup_learning_rate=0.0001",
"--initial_learning_rate=0.0001",
"--learning_rate_decay_type=cosine",
"--optimizer_type=momentum",
"--optimizer_arguments=momentum=0.9"
],
"region": "us-central1",
"jobDir": "gs://<bucket>",
"masterConfig": {
"imageUri": "gcr.io/cloud-ml-algos/image_classification:latest"
}
},
"trainingOutput": {
"consumedMLUnits": 1.59,
"isBuiltInAlgorithmJob": true,
"builtInAlgorithmOutput": {
"framework": "TENSORFLOW",
"runtimeVersion": "1.15",
"pythonVersion": "3.7"
}
}
}
}, function(error, response, body){
console.log(body);
});
Result:
...
{
createTime: '2022-02-09T17:36:42Z',
state: 'QUEUED',
trainingOutput: {
isBuiltInAlgorithmJob: true,
builtInAlgorithmOutput: {
framework: 'TENSORFLOW',
runtimeVersion: '1.15',
pythonVersion: '3.7'
}
},
etag: '999999aaaac='
Thank you everyone for the input. This was useful to help me resolve my issue, but I wanted to also share the approach I ended up taking:
I started by making sure I could kick off my job manually.
I used this tutorial with a config.yaml file that looked like this:
workerPoolSpecs:
machineSpec:
machineType: n1-standard-4
acceleratorType: NVIDIA_TESLA_T4
acceleratorCount: 1
replicaCount: 1
containerSpec:
imageUri: <Replace this with your container image URI>
args: ["--some=argument"]
When I had a job that could be kicked off manually, I switched to using
the Vertex AI Node.js API to start the job or cancel it. The API exists in other languages.
I know my original question was about HTTP requests, but having an API in the language was a lot easier for me, in particular because I didn't have to worry about authentification.
I hope that is useful, happy to provide mode details if needed.

Why does AWS have their own UUID version?

I've been working on an application hosted on the AWS cloud that is part of a data pipeline. The application processes events from EventBridge, does some data mapping and then puts the result on a Kinesis stream.
The incoming events payload looks something like this (truncated for readability):
{
"version": "0",
"id": "9a0f9e20-c518-a968-7fa6-1d8038a5bcfc",
"detail-type": "Some sort of event",
....
}
and the event put onto the Kinesis stream looks something like:
{
"eventId": "9a0f9e20-c518-a968-7fa6-1d8038a5bcfc",
"eventTime": "2021-04-08T06:19:47.683Z",
"eventType": "created",
...
}
I looked at the "id" attribute on the incoming event and at first glance it looks like a UUID. I put a few examples into an online validator and it came back as a valid UUID. Since it is a UUID and is supposed to be "universally unique" I thought I might just reuse that ID for the "eventId" attribute of the outgoing payload. I thought that might even make it easier to trace events back to the source.
However, when I started my integration testing I started to notice alarms going off on unrelated services. There were validation errors happening all over the place. Turns out that the downstream services didn't like the format of "eventId".
The downstream services use the "uuid" NPM module to validate UUIDs in our event envelopes and it seems like it doesn't like the UUIDs that come from AWS. To make sure that I had diagnosed the problem correctly I fired up a node REPL and tried to validate one of the UUIDs that came through and sure enough it came back as invalid!
> const uuid = require('uuid');
> u.validate("9a0f9e20-c518-a968-7fa6-1d8038a5bcfc")
false
I then checked the regex that the 'uuid' module was using to do the validation and I noticed that it was checking for the numbers 1-5 in the first character of the third group of the UUID.
Confused, I checked out the Wikipedia page for UUIDs and discovered that the UUID version of the UUIDs coming from AWS is A, instead of the expected version numbers (1-5)
I have a few related questions:
Why does AWS have it's own UUID version?
Is it even a UUID?
Why would AWS go and violate the principle of least
astonishment
like that, surely it's easier to just use a regular UUID?
I'm hoping someone has an interesting story about how AWS had to invent their own UUID version to deal with some epic engineering problem that only happens at their scale, but I suppose I'll settle for a more simple answer.

How to pass query parameters in API gateway with S3 json file backend

I am new to AWS and I have followed this tutorial : https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html, I am now able to read from the TEST console my AWS object stored on s3 which is the following (it is .json file):
[
{
"important": "yes",
"name": "john",
"type": "male"
},
{
"important": "yes",
"name": "sarah",
"type": "female"
},
{
"important": "no",
"name": "maxim",
"type": "male"
}
]
Now, what I am trying to achieve is pass query parameters. I have added type in the Method Request and added a URL Query String Parameter named type with method.request.querystring.type mapping in the Integration Request.
When I want to test, typing type=male is not taken into account, I still get the 3 elements instead of the 2 male elements.
Any reasons you think this is happening ?
For information, the Resources is the following (and I am using AWS Service integration type to create the GET method as explained in the AWS tutorial)
/
/{folder}
/{item}
GET
In case anyone is interested by the answer, I have been able to solve my problem.
The full detailed solution requires a tutorial but here are the main steps. The difficulty lies in the many moving parts so it is important to test each of them independently to make progress (quite basic you will tell me).
Make sure your SQL query to your s3 DB is correct, for this you can go in your s3 bucket, click on your file and select "query with s3 select" from the action.
Make sure that your lambda function works, so check that you build and pass the correct SQL query from the test event
Setup the API query strings in the Method Request panel and setup the Mapping Template in the Integration Request panel (for me it looked like this "TypeL1":"$input.params('typeL1')") using the json content type
Good luck !

How to get for different pages get fan_count?

GET▾→ /v2.7▾/pepsi?fields=fan_count
I am new to graph API, I am interested in getting the facebook fan counts for different pages.
Using Facebook Graph API explorer, the above code gives me for only 1 page,
but if I want to get the fan counts for different pages - What should be the code ?
For example- I tried GET▾→ /v2.7▾/coke,pepsi?fields=fan_count
but got the error --
{
"error": {
"message": "(#803) Some of the aliases you requested do not exist: coke,pepsi",
"type": "OAuthException",
"code": 803,
"fbtrace_id": "FqAQ297ssFe"
}
}
https://developers.facebook.com/docs/graph-api/using-graph-api/#multiidlookup
You can make a single GET request that retrieves multiple nodes by using the ?ids endpoint with the object IDs of those nodes.
So the request you want to make is: ?ids=coke,pepsi&fields=fan_count
You can retrieve data for up to 50 objects in one go this way.
Not sure why you believe this would be possible, but you need to do two API calls for two Pages. You can only make the API calls faster with batch requests, check out the docs for more information: https://developers.facebook.com/docs/graph-api/making-multiple-requests
They still count as separate API calls though, but it will be as fast as the slowest API call in the batch.