Figure out what to do with geocodes

[alexhebing]

In #11, we have experimented with finding geocodes for each LOCATION. Now in #17, we will be building a CLI that clients can use, and which consumes multiNER. Question: where / when are the geocodes collected?

Does the CLI do it? This yields the problem of authentication: an API key is required for Google, and a username for Geonames.

Should we have a webservice somewhere that handles this for us? This yields the problem that Google only allows for a limited amount of calls to its API (and these will increase significantly if we centralize these calls).

@BeritJanssen : any thoughts?

[BeritJanssen]

As for the Google Maps APIs, I think they now really let go of the "it's free for x requests a day" model and changed it into billing for all requests:

On June 11, 2018, we launched a new pay-as-you-go pricing model for the Google Maps Platform APIs. We have also introduced some new features, and changed the way that certain APIs and features are billed. This document provides a comparison of SKUs before and after June 11, 2018.

However, it seems you get some credit each month:

For each billing account, for qualifying Google Maps Platform SKUs, a $200 USD Google Maps Platform credit is available each month, and automatically applied to the qualifying SKUs.

It seems that Google Geocoding charges 0.5 cents per request, meaning we'd have 40,000 requests a month covered by the credit. Google Places seems to be completely free, as lat/long information is part of geometry, which is Basic data.

It looks as if Geocoder makes it possible to use the Places API, by putting method='places', e.g.:

g = geocoder.google('white house', method='places')

Should I request a new API key for Places and see if I can use Geocoder with it?
NB: It seems that all Google Maps APIs are limited to 50 requests per second.

[BeritJanssen]

As for other Geocoder libraries, in the lecture of Katie McDonough on October 1st of last year, she mentioned the following options (from my notes in Dutch):

• OpenStreetMap (gegevens van vrijwilligers)
• GeoNames (gegevens van overheid) http://www.geonames.org/
• Getty Thesaurus of Geographic Names http://www.getty.edu/research/tools/vocabularies/tgn/

Getty is not available via Geocoder, the other two are. OpenStreetMaps does not require an API key as far as I can see. Geonames needs a username.

NB: Katie McDonough called these tools Gazetteers. Just in case we run into this term in future meetings. I'm not sure whether from a developer perspective, it would be the correct term, as gazetteers include much more information than lat/long, which we are interested in. For distinguishing between Amsterdam, US and Amsterdam, NL, this extra information might still be interesting, but then linked open data may be the way to go.

[alexhebing]

Thank you very much for delving into all this @BeritJanssen ! I didn't take the time to look into it properly until now, but I'm very happy all these details are here 😃

I have implemented collecting geocodes from OpenStreetMap, GeoNames and Google. Your suggestion to use Google Places works, although it is needed to pass the language parameter as well (tested with Italian).

Google Places seems to be completely free

Can you show me in more detail where you got this idea from? It would be really nice, but I get the feeling that it isn't free after all. In the Geocoder source code you can see that it makes a call to the places API, the text search method: _URL = 'https://maps.googleapis.com/maps/api/place/textsearch/json' (line 123). In the Google documentation, if I look up this method (here under the heading 'SKU: Places – Text Search'), it seems that making this type of requests costs '0.032 USD per each' (about 60,000 a month). Or am I looking in the wrong place 🤞 ?

[alexhebing]

In the meeting today, I have discussd this with the clients, and we have decided not too include Google as one of the Geocoders. This relieves us of the burden of finding a way to work with the request limitations (i.e. pay or whatever). @BeritJanssen : if you still want to continue the discussion of how billing at Google works exactly I would be happy too, it is quite valuable to have this information somewhere in my opinion.

For now, we can simply remove the code that calls Google.

[jgonggrijp]

Perhaps a slightly tangential note, but FYI: OpenStreetMap (for raw data), Leaflet (for map visualization) and MapBox (for tiles and some other services with more attractive usage quota and pricing than Google Maps) have been previously chosen by the lab as preferred over Google for new projects. This was agreed upon by the team about 1,5 years ago.