Scopus API and Sustainable Development Goals search - web-services

I'm trying to automate a Scopus search using the API so we can more easily track publications related to the UN Sustainable Development Goals. However, I can't get the syntax right to 
The search strings for each SDG are long and complex. The search strings for each SDG are documented here: https://data.mendeley.com/datasets/87txkw7khs/1#file-71334e20-2e74-4557-9002-0b8d2b28103e
I'm testing SDG 2, with a string that looks like this:
TITLE-ABS-KEY ( ( {land tenure rights}  OR  ( smallholder  AND  ( farm  OR  forestry  OR  pastoral  OR  agriculture  OR  fishery  OR  {food producer}  OR  {food producers} ) )  OR  malnourish*  OR  malnutrition  OR  undernourish*  OR  {undernutrition}  OR  {agricultural production}  OR  {agricultural productivity}  OR  {agricultural practices}  OR  {agricultural management}  OR  {food production}  OR  {food productivity}  OR  {food security}  OR  {food insecurity}  OR  {land right}  OR  {land rights}  OR  {land reform}  OR  {land reforms}  OR  {resilient agricultural practices}  OR  ( agriculture  AND  potassium )  OR  fertili?er  OR  {food nutrition improvement}  OR  {hidden hunger}  OR  {genetically modified food}  OR  ( gmo  AND  food )  OR  {agroforestry practices}  OR  {agroforestry management}  OR  {agricultural innovation}  OR  ( {food security}  AND  {genetic diversity} )  OR  ( {food market}  AND  ( restriction  OR  tariff  OR  access  OR  {north south divide}  OR  {development governance} ) )  OR  {food governance}  OR  {food supply chain}  OR  {food value chain}  OR  {food commodity market}  AND NOT  {disease} ) )
The curly braces don't work with the API interactive tool at all (even for a single item) so I've tried this version of the query:
TITLE-ABS-KEY(("land tenure rights" OR (smallholder AND (farm OR forestry OR pastoral OR agriculture OR fishery OR "food producer" OR "food producers") ) OR malnourish* OR malnutrition OR undernourish* OR "undernutrition" OR "agricultural production" OR "agricultural productivity" OR "agricultural practices" OR "agricultural management" OR "food production" OR "food productivity" OR "food security" OR "food insecurity" OR "land right" OR "land rights" OR "land reform" OR "land reforms" OR "resilient agricultural practices" OR (agriculture AND potassium) OR fertili?er OR "food nutrition improvement" OR "hidden hunger" OR "genetically modified food" OR (gmo AND food) OR "agroforestry practices" OR "agroforestry management" OR "agricultural innovation" OR ("food security" AND "genetic diversity") OR ("food market" AND (restriction OR tariff OR access OR "north south divide" OR "development governance")) OR "food governance" OR "food supply chain" OR "food value chain" OR "food commodity market" AND NOT "disease"))
Which the Scopus API interactive tool translates as
https://api.elsevier.com/content/search/scopus?query=TITLE-ABS-KEY((%22land%20tenure%20rights%22%C2%A0OR%20(smallholder%20AND%20(farm%20OR%20forestry%20OR%20pastoral%20OR%20agriculture%20OR%20fishery%20OR%20%22food%20producer%22%20OR%20%22food%20producers%22)%C2%A0)%20OR%20malnourish*%20OR%20malnutrition%20OR%20undernourish*%20OR%20%22undernutrition%22%20OR%20%22agricultural%20production%22%20OR%20%22agricultural%20productivity%22%20OR%20%22agricultural%20practices%22%20OR%20%22agricultural%20management%22%20OR%20%22food%20production%22%20OR%20%22food%20productivity%22%20OR%20%22food%20security%22%20OR%20%22food%20insecurity%22%20OR%20%22land%20right%22%20OR%20%22land%20rights%22%20OR%20%22land%20reform%22%20OR%20%22land%20reforms%22%20OR%20%22resilient%20agricultural%20practices%22%20OR%20(agriculture%20AND%20potassium)%20OR%20fertili%3Fer%20OR%20%22food%20nutrition%20improvement%22%20OR%20%22hidden%20hunger%22%20OR%20%22genetically%20modified%20food%22%20OR%20(gmo%20AND%20food)%20OR%20%22agroforestry%20practices%22%20OR%20%22agroforestry%20management%22%20OR%20%22agricultural%20innovation%22%20OR%20(%22food%20security%22%20AND%20%22genetic%20diversity%22)%20OR%20(%22food%20market%22%20AND%20(restriction%20OR%20tariff%20OR%20access%20OR%20%22north%20south%20divide%22%20OR%20%22development%20governance%22))%20OR%20%22food%20governance%22%20OR%20%22food%20supply%20chain%22%20OR%20%22food%20value%20chain%22%20OR%20%22food%20commodity%20market%22%20AND%20NOT%20%22disease%22))&apiKey=xxxMYAPIKEY
The result is a service error: "service-error": { "status": { "statusCode": "INVALID_INPUT", "statusText": "Error translating query"  
Am I missing something? Or are queries this long not supported by the API?


It works for me once...
... the opening curly brace { is replaced with %7B, and
the closing curly brace } with %7D.
(See also this explanation on curly braces in URLs.)
The whole URL is thus:
https://api.elsevier.com/content/search/scopus?query=TITLE-ABS-KEY%20(%20(%20%7Bland%20tenure%20rights%7D%20%20OR%20%20(%20smallholder%20%20AND%20%20(%20farm%20%20OR%20%20forestry%20%20OR%20%20pastoral%20%20OR%20%20agriculture%20%20OR%20%20fishery%20%20OR%20%20%7Bfood%20producer%7D%20%20OR%20%20%7Bfood%20producers%7D%20)%20)%20%20OR%20%20malnourish*%20%20OR%20%20malnutrition%20%20OR%20%20undernourish*%20%20OR%20%20%7Bundernutrition%7D%20%20OR%20%20%7Bagricultural%20production%7D%20%20OR%20%20%7Bagricultural%20productivity%7D%20%20OR%20%20%7Bagricultural%20practices%7D%20%20OR%20%20%7Bagricultural%20management%7D%20%20OR%20%20%7Bfood%20production%7D%20%20OR%20%20%7Bfood%20productivity%7D%20%20OR%20%20%7Bfood%20security%7D%20%20OR%20%20%7Bfood%20insecurity%7D%20%20OR%20%20%7Bland%20right%7D%20%20OR%20%20%7Bland%20rights%7D%20%20OR%20%20%7Bland%20reform%7D%20%20OR%20%20%7Bland%20reforms%7D%20%20OR%20%20%7Bresilient%20agricultural%20practices%7D%20%20OR%20%20(%20agriculture%20%20AND%20%20potassium%20)%20%20OR%20%20fertili?er%20%20OR%20%20%7Bfood%20nutrition%20improvement%7D%20%20OR%20%20%7Bhidden%20hunger%7D%20%20OR%20%20%7Bgenetically%20modified%20food%7D%20%20OR%20%20(%20gmo%20%20AND%20%20food%20)%20%20OR%20%20%7Bagroforestry%20practices%7D%20%20OR%20%20%7Bagroforestry%20management%7D%20%20OR%20%20%7Bagricultural%20innovation%7D%20%20OR%20%20(%20%7Bfood%20security%7D%20%20AND%20%20%7Bgenetic%20diversity%7D%20)%20%20OR%20%20(%20%7Bfood%20market%7D%20%20AND%20%20(%20restriction%20%20OR%20%20tariff%20%20OR%20%20access%20%20OR%20%20%7Bnorth%20south%20divide%7D%20%20OR%20%20%7Bdevelopment%20governance%7D%20)%20)%20%20OR%20%20%7Bfood%20governance%7D%20%20OR%20%20%7Bfood%20supply%20chain%7D%20%20OR%20%20%7Bfood%20value%20chain%7D%20%20OR%20%20%7Bfood%20commodity%20market%7D%20%20AND%20NOT%20%20%7Bdisease%7D%20)%20)&apiKey=xxxxxxxxYOURAPIKEY

Related

Authentication issues when using Bing Maps REST API on Power BI

I keep having this error when using Bing Maps API, as anyone encountered this issue?

I resolved the issue. I kept getting the authentication issues because there was a "#" in the address line. If an apt or street number is written as #.., it will keep asking for credentials. Just change the # to No and it works fine.

I have. What have you done exactly? Why is it complaining about credentials?
Here is an example function I used at the time. It fetches the Coordinates for a desired city. You can paste it in Advanced Editor and make a function. Just make sure to insert your api key.
let
fxGetCoords = (city as text) =>
let
Source = Json.Document(
Web.Contents("http://dev.virtualearth.net/REST/v1/Locations?query=" &
city & "&includeNeighborhood=0&key="xxxxxx-insert-your-key-here-
xxxxxxxx")
),
resourceSets = Source[resourceSets],
resourceSets1 = resourceSets{0},
resources = resourceSets1[resources],
resources1 = resources{0},
point = resources1[point],
coordinates = point[coordinates],
#"Converted to Table" = Table.FromList(coordinates, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Transposed Table" = Table.Transpose(#"Converted to Table"),
#"Renamed Columns" = Table.RenameColumns(#"Transposed Table",{{"Column1", "Latitude"}, {"Column2", "Longitude"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Latitude", type number}, {"Longitude", type number}})
in
#"Changed Type"
in fxGetCoords
If this works the problem is not your credentials, but how you are using them.

Reference another Table in the Query Editor (PowerBI or PowerQuery)

I am running a query to update a historical data table within PowerBI. This update query calls the Adobe Analytics API, and pulls data based on a date range I specify.
As I want this to run as fast as possible, I would only like to query the data that I don't already have in my historical data table. I am okay at putting an end date for this query based off Date.Time.LocalNow().
#date(Date.Year(DateTime.LocalNow()), Date.Month(DateTime.LocalNow()), Date.Day(DateTime.LocalNow())-1)
However, where I am stuck is putting a start date. Ideally I would like to take the MaxDate from a table within my data model, and use this value in the query editor as my start date.
How would I do this?
UPDATE 1 - Full code Below
let
Source = AdobeAnalytics.Cubes()
myreportsuiteid = Source{[Id="myreportsuiteid"]}[Data],
#"Added Items" = Cube.Transform(myreportsuiteid,
{
{Cube.AddAndExpandDimensionColumn, "DateGranularity", {"year", "month", "day"}, {"Date Granularity.Level 1: Year", "Date Granularity.Level 2: Month", "Date Granularity.Level 3: Day"}},
{Cube.AddAndExpandDimensionColumn, "lasttouchchannel", {"lasttouchchannel"}, {"Last Touch Marketing Channel"}},
{Cube.AddMeasureColumn, "Unique Visitors", "uniquevisitors"},
{Cube.AddMeasureColumn, "Visits", "visits"},
{Cube.ApplyParameter, "DateRange", {#date(Date.Year(List.Max(Union[Date])), Date.Month(List.Max(Union[Date])), Date.Day(List.Max(Union[Date]))+1), #date(Date.Year(DateTime.LocalNow()), Date.Month(DateTime.LocalNow()), Date.Day(DateTime.LocalNow())-1)}},
{Cube.ApplyParameter, "Segment", {{"s300000554_5ae201be22fa9950dcdbcd92"}}}
})
in
#"Added Items"
UPDATE 2
WORKING CODE
Daily US is a table created thorough the Query Editor
let
Source = AdobeAnalytics.Cubes(),
todaysDate = Date.AddDays(DateTime.Date(DateTime.LocalNow()),-1),
maxDate = Date.AddDays(List.Max(#"Daily US"[Date]),1),
myreportsuiteid = Source{[Id="myreportsuiteid "]}[Data],
ERROR CODE
Union US is a table created outside of the Query Editor via Union US = DISTINCT(UNION('Seeker US','Daily US'))
let
Source = AdobeAnalytics.Cubes(),
todaysDate = Date.AddDays(DateTime.Date(DateTime.LocalNow()),-1),
maxDate = Date.AddDays(List.Max(#"Union US"[Date]),1),
myreportsuiteid = Source{[Id="myreportsuiteid "]}[Data],

If the column you want to reference for the MaxDate is named Union[Date], then you should be able to use List.Max to get the maximum value from that column.
Try this:
let
Source = AdobeAnalytics.Cubes(),
MaxDate = List.Max(Union[Date]),
TodaysDate = DateTime.Date(DateTime.LocalNow()),
myreportsuiteid = Source{[Id="myreportsuiteid"]}[Data],
#"Added Items" = Cube.Transform(myreportsuiteid,
{
{Cube.AddAndExpandDimensionColumn, "DateGranularity", {"year", "month", "day"}, {"Date Granularity.Level 1: Year", "Date Granularity.Level 2: Month", "Date Granularity.Level 3: Day"}},
{Cube.AddAndExpandDimensionColumn, "lasttouchchannel", {"lasttouchchannel"}, {"Last Touch Marketing Channel"}},
{Cube.AddMeasureColumn, "Unique Visitors", "uniquevisitors"},
{Cube.AddMeasureColumn, "Visits", "visits"},
{Cube.ApplyParameter, "DateRange", {MaxDate, TodaysDate}},
{Cube.ApplyParameter, "Segment", {{"s300000554_5ae201be22fa9950dcdbcd92"}}}
})
in
#"Added Items"

Hele converting MSSQL query to Korma entities

I have the below MSSQL query for which I am not able to figure out the Korma entities. Please help out
select t.d as did from (
select dataid as d , count(dataid) as
cd from <table_name>
WHERE prid = <pid> group by dataid
) as t WHERE t.cd >1;
Thanks

SQL Korma documentation site contains subselect sample:
;; Subselects can be used as entities too!
(defentity subselect-example
(table (subselect users
(where {:active true}))
:activeUsers))

Splunk: how to extract fields using regular expressions? like rex in splunk search

I want to extract Primary and StandyBy DB names from the below string which I found in my splunk search.
Jul 20 14:43:31 XXXXXXXX GuptaA GuptaA - Primary database GuptaC - (*) Physical standby database GuptaB - Physical standby database.
Jul 20 14:43:31 XXXXXXXX KumarA KumarA - Primary database KumarC - (*) Physical standby database KumarD - Physical standby database - Physical standby database KumarE - Physical standby database
Primary DB : GuptaA
SecondaryDBs : GuptaC, GuptaB
I want to show a table with below details.
Primary DB StandyByDB
GuptaA GuptaC, GuptaB
KumarA KumarC, KumarD, KumarE
Any suggestions using splunk search?
Thank you!

rex field=_raw "Primary Database (?<primary>\S+) .* standby database (?<standby>\S+)"
| table primary standby

How to access tables in other schema in korma?

In SQL, accessing tables in other schema is simple:
select *
from other_schema.t
where ...
How can I do this in korma? What I actually to do is to access information_schema.tables table. So defining another db by defdb wouldn't be helpful.
I've tried to define the entity, however, failed.
(defentity information_schema.tables)

I've got to know that there is a way to specify the base table when defining an entity. When specifying the base table, it allows to set the schema with ..
(defentity tables
(table :information_schema.tables))
This works fine for accessing information_schema.tables table, without defining another db.

You should be able to do this by defining another db. I can create a db like this:
CREATE database my_db;
USE my_db;
CREATE TABLE stuff (
things VARCHAR(255)
);
INSERT INTO stuff (things) VALUES ("some things");
Now I define two Korma databases and entities, and query them:
(defdb my-db (mysql {:host "localhost"
:port 3306
:db "my_db"
:user "root"
:password nil}))
(defdb information-schema (mysql {:host "localhost"
:port 3306
:db "information_schema"
:user "root"
:password nil}))
(defentity stuff)
(defentity information-schema)
(select stuff
(database my-db))
;; => ({:things "some things"})
(select TABLES
(database information-schema)
(fields :TABLE_SCHEMA :TABLE_NAME)
(where {:TABLE_SCHEMA "my_db"}))
;; => ({:TABLE_NAME "stuff", :TABLE_SCHEMA "my_db"})