I want to scrape the data from an ArcGIS map. The following map has a popup when we click the red features. How do I access that data programmatically?
Link : https://cslt.maps.arcgis.com/apps/MapSeries/index.html?appid=2c9f3e737cbf4f6faf2eb956fa26cdc5
Note: Please respect the access and use constraints of any ArcGIS Online item you access. When in doubt, don't save a copy of someone else's data.
The ArcGIS Online REST interface makes it relatively simple to get the data behind ArcGIS Online items. You need to use an environment that can make HTTP requests and parse JSON text. Most current programming languages either have these capabilities built in or have libraries available with these capabilities.
Here's a general workflow that your code could follow.
Use the app ID and the item data endpoint to see the app's JSON text:
https://www.arcgis.com/sharing/rest/content/items/2c9f3e737cbf4f6faf2eb956fa26cdc5/data
Search that text for webmap and see that the app uses the following web maps:
d2b4a98c39fd4587b99ac0878c420125
7b1af1752c3a430184fbf7a530b5ec65
c6e9d07e4c2749e4bfe23999778a3153
Look at the item data endpoint for any of those web maps:
https://www.arcgis.com/sharing/rest/content/items/d2b4a98c39fd4587b99ac0878c420125/data
The list of operationalLayers specifies the feature layer URLs from which you could harvest data. For example:
https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0
Then just run a query with a where of 0=0 (or whatever you want) and an outFields of *:
https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0/query?where=0%3D0&outFields=%2A&f=json
Use f=html instead if you want to see a human-readable request form and results.
Note that feature services have a limit of how many features you can get per request, so you will probably want to filter by geometry or attribute values. Read the documentation to learn everything you can do with feature service queries.
Related
Please allow me to ask a rather newbie question. So far, I have been using local tools like imagemagick or GOCR to perform the job, but that is rather old-fashioned, and I am urged to "move to google cloud AI".
The setup
I have a (training) data set of various documents (as JPG and PDF) of different kinds, and by certain features (like prevailing color, repetitive layout) I intend to classify them, e.g. as invoice type 1, invoice type 2, not an invoice. In a 2nd step, I would like to OCR certain predefined areas of each document and extract e.g. the address of the company sending the invoice and the date.
The architecture I am envisioning
In a modern platform as a service (pass), I have already set up an UI where I can upload new files. These are then locally stored in a directory with filenames (or in a MongoDB). Meta info like upload timestamp, user, original file name is stored in a DB.
The newly uploaded file should should then be submitted to google cloud which should do the classification step, and deliver back the label to be saved in the database.
The document pages should be auto-cropped, i.e. black or white margins are removed, most probably with google cloud as well. The parameters of the crop should be persisted in the DB.
In case it is e.g. an invoice, OCR should be performed (again by google cloud) for certain regions of the documents, e.g. a bounding box of spanning from the mid of the page to the right margin in the upper 10% of the cropped page. The results of the OCR should be again persisted locally.
The problem
I seem to be missing the correct search term to figure out how to do it with google cloud. Is there an google-API (e.g. REST), I can use to upload and which gives me back the results of steps 2 to 4?
I think that your best option here is to use Document AI (REST API and Libraries).
Using Document AI, you can:
Convert images to text
Classify documents
Analyze and extract entities
Additionally, for your use case, we have a new Document AI feature that is still in preview and has limited access which is the Invoice parser.
Invoice parser is similar to Form parser but for invoices instead of forms. Check out the Invoice parser page and you will see what I mean by preview and limited access.
AFIK, there isn't any GCP tool for image edition.
We have a company program designed to help us get control over data. It has feature to group all the application of one Client. If I want to take a look at them I click on the Client and I see a list of all applications made for him. Take a look at the picture below:
I was wondering if Microsoft Access can do the same? If yes where should I start looking?
I did some internet search and no solution found.
That is built in, and it is called Subdatasheet. You have relationships properly set between Clients and Order, for instance, when you open the Clients table you will see such small "+" allowing to view the Orders of the current client. You may have to set the Subdatasheet Name property of table Clients to "Orders" in this case.
If you want to work with forms, you can build a continuous from for Clients, then one for Orders, then insert the Orders subform in the Footer of the Clients form. Access might tell you you can't do this, just ignore, it works.
In Access that would simply be a continuous form with a filter. Typically opened from a list of clients, setting a filter for the applications of the selected client.
Unless I'm misunderstanding the question.
I have the following requirement:
1) A user browser fetches data points (essentially time series) from the server and plots on the web page.
2) User should be able to alter(also view) the data points using the mouse and doing this should alter the plot dynamically. In this way, user can alter all the data points until he is satisfied.
3) Finally, user needs to send his "version of data points" back to the server.
I am new to web development and would like to know if there anything out there already that I can refer to or any set of standard tools that I can use?
Regards,
Vivek
You may have a look at the googleVis package of the R software. The package provides and interface between R and the Google Chart Tools. This can be a convenient way to generate the html displaying a graphic of the data.
In this document the authors of the package explain how to use googleVis along with the packages Rook and shiny. These packages allow the user to upload a data file and update the graphic. The user would have to edit the file outside the browser but you could add a form in order to edit the file through the browser and update the graphic when the data are submitted.
This is an example of a Business on Google Maps
It has elements attached such as:
Reviews from various sites (qype, viewlondon, etc...)
Details provided by various sites
Photos and other content
I don't know how to go on about retrieving such Business and associate any items generated on my website.
What I have implemented up to date is a system using geocoding (geopy) which once given an address, it gives back Latitude and Longitude, but such system does not help me with this dilemma.
What you want is this API:
http://code.google.com/apis/ajaxsearch/local.html
Also check this:
http://googleajaxsearchapi.blogspot.com/2007/06/local-search-control-for-maps-api.html
By writing a relay server script you could do things like this, which obtains most of that information with a different layout. I don't know if it's legal to do that.
I am looking for a free database (csv, tab delimited, xml, etc) or even a web service or API which will allow me to query for business listings within a certain distance of a latitude and longitude.
A global scope would be perfect, but I would settle for Canada/US information.
Does anyone know of anything like this?
Google AJAX search API offers this kind of service (local search) if you use the Google Maps API, see http://code.google.com/intl/fr/apis/ajaxsearch/documentation/
Apparently Yahoo! does too (see http://developer.yahoo.com/search/local/localSearch.html), if you do not plan a commercial use.
If you want to do this right, here is what you need:
Database of all locations, with long/lat coords
Database of all businesses
Address --> long/lat conversion system, if your business database is not already geocoded
I can't help you out with the business database, but here is a good resource for the "all locations" database:
http://www.geobytes.com/freeservices.htm
(Scroll down to the bottom of the page - you're looking for "GeoWorldMap".)
It contains a set of text files that are designed to be imported into a relational database. It's the most complete (free) resource that I've been able to find for this sort of task.
The Google Maps Places API provides exactly what you're looking for: https://developers.google.com/places/documentation/search