I've used the JIRA API a few timees, but only for small extracts so I've never had to use pagniation to return more results than the max result.
Here's the documentation for the API: https://developer.atlassian.com/cloud/jira/platform/rest/v3/#expansion
I have a query to the API:
let
Source = Json.Document(Web.Contents("*connection URL*,
[Headers=[Accept="application/json"]])),
#"Converted to Table" = Table.FromRecords({Source}),
>Expand the scheme, and define the field data types etc
in
#"Changed Type"
I can see form the documentation I need to use something like this:
{
"startAt" : 0,
"maxResults" : 10,
"total": 200,
"isLast": false,
"values": [
{ /* result 0 */ },
{ /* result 1 */ },
{ /* result 2 */ }
]
}
I'm not sure where the pagination would fit into this? Can anyone help me? I need to return 1500 results.
Thanks
You'll need to make multiple REST requests that iterate through all of the results one "page" at a time:
Read the API response from JIRA and capture the values of maxResults and startAt in the response.
Add the value of maxResults to startAt, update your REST API call startAt parameter, and call the same API endpoint again.
Repeat until startAt is greater than or equal to total, or until isLast is true.
A while loop is a natural fit for an operation like this. To minimize API calls, you should use a larger value for maxResults so you receive more data in each loop.
Related
I am working on a POC using Kendra and Salesforce. The connector allows me to connect to my Salesforce Org and index knowledge articles. I have been able to set this up and it is currently working as expected.
There are a few custom fields and data points I want to bring over to help enrich the data even more. One of these is an additional answer / body that will contain key information for the searching.
This field in my data source is rich text containing HTML and is often larger than 2048 characters, a limit that seems to be imposed in a String data field within Kendra.
I came across two hooks that are built in for Pre and Post data enrichment. My thought here is that I can use the pre hook to strip HTML tags and truncate the field before it gets stored in the index.
Hook Reference: https://docs.aws.amazon.com/kendra/latest/dg/API_CustomDocumentEnrichmentConfiguration.html
Current Setup:
I have added a new field to the index called sf_answer_preview. I then mapped this field in the data source to the rich text field in the Salesforce org.
If I run this as is, it will index about 200 of the 1,000 articles and give an error that the remaining articles exceed the 2048 character limit in that field, hence why I am trying to set up the enrichment.
I set up the above enrichment on my data source. I specified a lambda to use in the pre-extraction, as well as no additional filtering, so run this on every article. I am not 100% certain what the S3 bucket is for since I am using a data source, but it appears to be needed so I have added that as well.
For my lambda, I create the following:
exports.handler = async (event) => {
// Debug
console.log(JSON.stringify(event))
// Vars
const s3Bucket = event.s3Bucket;
const s3ObjectKey = event.s3ObjectKey;
const meta = event.metadata;
// Answer
const answer = meta.attributes.find(o => o.name === 'sf_answer_preview');
// Remove HTML Tags
const removeTags = (str) => {
if ((str===null) || (str===''))
return false;
else
str = str.toString();
return str.replace( /(<([^>]+)>)/ig, '');
}
// Truncate
const truncate = (input) => input.length > 2000 ? `${input.substring(0, 2000)}...` : input;
let result = truncate(removeTags(answer.value.stringValue));
// Response
const response = {
"version" : "v0",
"s3ObjectKey": s3ObjectKey,
"metadataUpdates": [
{"name":"sf_answer_preview", "value":{"stringValue":result}}
]
}
// Debug
console.log(response)
// Response
return response
};
Based on the contract for the lambda described here, it appears pretty straight forward. I access the event, find the field in the data called sf_answer_preview (the rich text field from Salesforce) and I strip and truncate the value to 2,000 characters.
For the response, I am telling it to update that field to the new formatted answer so that it complies with the field limits.
When I log the data in the lambda, the pre-extraction event details are as follows:
{
"s3Bucket": "kendrasfdev",
"s3ObjectKey": "pre-extraction/********/22736e62-c65e-4334-af60-8c925ef62034/https://*********.my.salesforce.com/ka1d0000000wkgVAAQ",
"metadata": {
"attributes": [
{
"name": "_document_title",
"value": {
"stringValue": "What majors are under the Exploratory track of Health and Life Sciences?"
}
},
{
"name": "sf_answer_preview",
"value": {
"stringValue": "A complete list of majors affiliated with the Exploratory Health and Life Sciences track is available online. This track allows you to explore a variety of majors related to the health and life science professions. For more information, please visit the Exploratory program description. "
}
},
{
"name": "_data_source_sync_job_execution_id",
"value": {
"stringValue": "0fbfb959-7206-4151-a2b7-fce761a46241"
}
},
]
}
}
The Problem:
When this runs, I am still getting the same field limit error that the content exceeds the character limit. When I run the lambda on the raw data, it strips and truncates it as expected. I am thinking that the response in the lambda for some reason isn't setting the field value to the new content correctly and still trying to use the data directly from Salesforce, thus throwing the error.
Has anyone set up lambdas for Kendra before that might know what I am doing wrong? This seems pretty common to be able to do things like strip PII information before it gets indexed, so I must be slightly off on my setup somewhere.
Any thoughts?
since you are still passing the rich text as a metadata filed of a document, the character limit still applies so the document would fail at validation step of the API call and would not reach the enrichment step. A work around is to somehow append those rich text fields to the body of the document so that your lambda can access it there. But if those fields are auto generated for your documents from your data sources, that might not be easy.
On Google Admin screen, I can get numbers of available licenses and used licenses shown below:
How can I get these numbers via API?
Note: I read this question and tried, but not worked well.
-- EDIT: 2021/07/15 --
My request:
https://developers.google.com/admin-sdk/reports/reference/rest/v1/customerUsageReports/get
date: (few days before now)
parameters: accounts:gsuite_unlimited_total_licenses (comes from Account Parameters)
Response from API:
{
"kind": "admin#reports#usageReports",
"etag": "\"occ7bTD-Q2yefKPIae3LMOtCT9xQVZYBzlAbHU5b86Q/gt9BLwRjoWowpJCRESV3vBMjYMc\""
}
Expectation: I want to get the data same as 2 available, 1132 assigned as the GUI shows.
To be honestly, I'm not satisfying even if I can get info via this API, because it seems not responding real-time data like GUI.
I think there are 2 ways this information can be obtain, but I can confirm for only one of them.
1. Using the Report API that you mentioned.
NOTE : The report is not live data, so you must run the API call with a "date" parameter set at least 2 days before the execution date
Given that, you would have to run this GET method with the proper date in the {date} param
GET https://admin.googleapis.com/admin/reports/v1/usage/dates/{date}
Then you would need to parse through the parameters to find the desired license you are looking for.
reference - https://developers.google.com/admin-sdk/reports/reference/rest/v1/customerUsageReports#UsageReports
Here is how it look like after parsing
[
{
"BoolValue": null,
"DatetimeValueRaw": null,
"DatetimeValue": null,
"IntValue": 12065,
"MsgValue": null,
"Name": "accounts:gsuite_enterprise_total_licenses",
"StringValue": null
},
{
"BoolValue": null,
"DatetimeValueRaw": null,
"DatetimeValue": null,
"IntValue": 12030,
"MsgValue": null,
"Name": "accounts:gsuite_enterprise_used_licenses",
"StringValue": null
}
]
Important : The repot will always date 2 day back, so you can get the total number of licenses gsuite_enterprise_total_licenses in my example, and then use the Enterprise License Manager API to retrieve all currently assigned licenses
reference https://developers.google.com/admin-sdk/licensing/reference/rest
2. Using the Reseller API
Retrieving the information from the reseller point of view you would need to use the subscriptions.get method, providing your customerId and subscriptionId , calling the following GET request:
GET https://reseller.googleapis.com/apps/reseller/v1/customers/{customerId}/subscriptions/{subscriptionId}
The response of that would be a subscriptions resource, that contains various information about the license and the Seats object , which if you expand looks like this :
{
"numberOfSeats": integer,
"maximumNumberOfSeats": integer,
"licensedNumberOfSeats": integer,
"kind": string
}
numberOfSeats should be the total amount of licenses and licensedNumberOfSeats should be the number of users having that license assigned to them.
NOTE : in order to use this API , the given tenant should have a "fully executed and signed reseller contract" - https://developers.google.com/admin-sdk/reseller/v1/how-tos/prerequisites
Reference - https://developers.google.com/admin-sdk/reseller/reference/rest/v1/subscriptions
Answer:
You can only get the number of assigned licenses using the API, the number available isn't exposed and so does not get returned.
More Information:
Given that you have licenses assigned for your domain, and the user that is querying the API has access to this information, you can retrieve the data with the following request:
curl \
'https://admin.googleapis.com/admin/reports/v1/usage/dates/2021-07-10?parameters=accounts%3Agsuite_unlimited_total_licenses&fields=*&key=[YOUR_API_KEY]' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--compressed
While not necessary, I added the parameter field=* in order to make sure all data is returned.
This gave me a response as such:
{
"kind": "admin#reports#usageReports",
"etag": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"usageReports": [
{
"kind": "admin#reports#usageReport",
"date": "2021-07-10",
"etag": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"entity": {
"type": "CUSTOMER",
"customerId": "C0136mgul"
},
"parameters": [
{
"name": "accounts:gsuite_unlimited_total_licenses",
"intValue": "233"
}
]
}
]
}
Here you can see that the intValue for accounts:gsuite_unlimited_total_licenses is 233 - which is reflected in the UI:
Feature Request:
You can however let Google know that this is a feature that is important for access to their APIs, and that you would like to request they implement it.
Google's Issue Tracker is a place for developers to report issues and make feature requests for their development services, I'd urge you to make a feature request there. The best component to file this under would be the Google Admin SDK component, with the Feature Request template.
I was not able to use Reseller API (ran into some authorization issues) and Reports API (contained null values in all relevant attributes). The only way I was able to find how many licenses were remaining was through Enterprise License Manager API.
After getting assignments, I used sdkName to filter records based on the type of license.
Here is the complete code.
function getRemainingGoogleUserLicensesCount() {
const GOOGLE_USER_LICENSES_TOTAL = 100 // You can find this from Google Admin -> Billing -> Subscriptions
const productId = 'Google-Apps';
const customerId = 'C03az79cb'; // You can find this from this response, https://developers.google.com/admin-sdk/directory/v1/guides/manage-users#json-response
let assignments = [];
let usedLicenseCount = 0
let pageToken = null;
do {
const response = AdminLicenseManager.LicenseAssignments.listForProduct(productId, customerId, {
maxResults: 500,
pageToken: pageToken
});
assignments = assignments.concat(response.items);
pageToken = response.nextPageToken;
} while (pageToken);
for (const assignment of assignments) {
if (assignment["skuName"] == "Google Workspace Business Plus") {
usedLicenseCount += 1
}
}
return GOOGLE_USER_LICENSES_TOTAL - usedLicenseCount
}
I have an AppSync pipeline resolver. The first function queries an ElasticSearch database for the DynamoDB keys. The second function queries DynamoDB using the provided keys. This was all working well until I ran into the 1 MB limit of AppSync. Since most of the data is in a few attributes/columns I don't need, I want to limit the results to just the attributes I need.
I tried adding AttributesToGet and ProjectionExpression (from here) but both gave errors like:
{
"data": {
"getItems": null
},
"errors": [
{
"path": [
"getItems"
],
"data": null,
"errorType": "MappingTemplate",
"errorInfo": null,
"locations": [
{
"line": 2,
"column": 3,
"sourceName": null
}
],
"message": "Unsupported element '$[tables][dev-table-name][projectionExpression]'."
}
]
}
My DynamoDB function request mapping template looks like (returns results as long as data is less than 1 MB):
#set($ids = [])
#foreach($pResult in ${ctx.prev.result})
#set($map = {})
$util.qr($map.put("id", $util.dynamodb.toString($pResult.id)))
$util.qr($map.put("ouId", $util.dynamodb.toString($pResult.ouId)))
$util.qr($ids.add($map))
#end
{
"version" : "2018-05-29",
"operation" : "BatchGetItem",
"tables" : {
"dev-table-name": {
"keys": $util.toJson($ids),
"consistentRead": false
}
}
}
I contacted the AWS people who confirmed that ProjectionExpression is not supported currently and that it will be a while before they will get to it.
Instead, I created a lambda to pull the data from DynamoDB.
To limit the results form DynamoDB I used $ctx.info.selectionSetList in AppSync to get the list of requested columns, then used the list to specify the data to pull from DynamoDB. I needed to get multiple results, maintaining order, so I used BatchGetItem, then merged the results with the original list of IDs using LINQ (which put the DynamoDB results back in the correct order since BatchGetItem in C# does not preserve sort order like the AppSync version does).
Because I was using C# with a number of libraries, the cold start time was a little long, so I used Lambda Layers pre-JITed to Linux which allowed us to get the cold start time down from ~1.8 seconds to ~1 second (when using 1024 GB of RAM for the Lambda).
AppSync doesn't support projection but you can explicitly define what fields to return in the response template instead of returning the entire result set.
{
"id": "$ctx.result.get('id')",
"name": "$ctx.result.get('name')",
...
}
Summary: I have a job board, a user searches a zip code and all the jobs matching that zip code are displayed, I am trying to add a feature that lets you see jobs within a certain mile radius of that zip code. There is a web API ( www.zipcodeapi.com ) that does these calculations and returns zip codes within the specified radius, I am just unsure how to use it.
Using www.zipcodeapi.com , you enter a zip code and a distance and it returns all zip codes within this distance. The format for API request is as follows: https://www.zipcodeapi.com/rest/<api_key>/radius.<format>/<zip_code>/<distance>/<units>, so if a user enters zip code '10566' and a distance of 5 miles, the format would be https://www.zipcodeapi.com/rest/<api_key>/radius.json/10566/5/miles and this would return:
{
"zip_codes": [
{
"zip_code": "10521",
"distance": 4.998,
"city": "Croton On Hudson",
"state": "NY"
},
{
"zip_code": "10548",
"distance": 3.137,
"city": "Montrose",
"state": "NY"
}
#etc...
]
}
My question is how do I send a GET request to the API using django?
I have the user searched zip code stored in zip = request.GET.get('zip') and the mile radius stored in mile_radius = request.GET['mile_radius']. How can I incorporate those two values in their respective spots in https://www.zipcodeapi.com/rest/<api_key>/radius.<format>/<zip_code>/<distance>/<units> and send the request? Can this be done with Django or do I have this all confused? Does it need to be done with a frontend language? I have tried to search this on google but only find this for RESTful APIS, and I dont think this is what I am looking for. Thanks in advance for any help, if you couldn't tell i've never worked with a web API before.
You can use the requests package, to do exactly what you want. It's pretty straightforward and has good documentation.
Here's an example of how you could perform it for your case:
zip_code = request.GET.get('zip')
mile_radius = request.GET['mile_radius']
api_key = YOUR_API_KEY
fmt = 'json'
units = 'miles'
response = requests.get(
url=f'https://www.zipcodeapi.com/rest/{api_key}/radius.{fmt}/{zip_code}/{mile_radius}/{units}')
zip_codes = response.json().get('zip_codes')
zip_codes should then be an array with those dicts as in your example.
I have entities that look like that:
{
name: "Max",
nicknames: [
"bestuser"
]
}
how can I query for nicknames to get the name?
I have created the following index,
indexes:
- kind: users
properties:
- name: name
- name: nicknames
I use the node.js client library to query the nickname,
db.createQuery('default','users').filter('nicknames', '=', 'bestuser')
the response is only an empty array.
Is there a way to do that?
You need to actually fetch the query from datastore, not just create the query. I'm not familiar with the nodejs library, but this is the code given on the Google Cloud website:
datastore.runQuery(query).then(results => {
// Task entities found.
const tasks = results[0];
console.log('Tasks:');
tasks.forEach(task => console.log(task));
});
where query would be
const query = db.createQuery('default','users').filter('nicknames', '=', 'bestuser')
Check the documentation at https://cloud.google.com/datastore/docs/concepts/queries#datastore-datastore-run-query-nodejs
The first point to notice is that you don't need to create an index to this kind of search. No inequalities, no orders and no projections, so it is unnecessary.
As Reuben mentioned, you've created the query but you didn't run it.
ds.runQuery(query, (err, entities, info) => {
if (err) {
reject(err);
} else {
response.resultStatus = info.moreResults;
response.cursor = info.moreResults == TNoMoreResults? null: info.endCursor;
resolve(entities);
};
});
In my case, the response structure was made to collect information on the cursor (if there is more data than I've queried because I've limited the query size using limit) but you don't need to anything more than the resolve(entities)
If you are using the default namespace you need to remove it from your query. Your query needs to be like this:
const query = db.createQuery('users').filter('nicknames', '=', 'bestuser')
I read the entire plob as a string to get the bytes of a binary file here. I imagine you simply parse the Json per your requirement