I am trying to use the cursor to implement pagination but when I try to use the endCursor that is returned after my first query (queries 10 records), it gives me an error "invalid encoding". By the way I have a total of 16 records. I am expecting that on my next query, it will give me the last 6 records
Here's my code:
router.get("/scan/history/query", async (req: Request, resp: Response) => {
const userId = resp.locals.user && resp.locals.user.sub
const pageCursor = req.query.cursor
if (userId) {
let mainQuery = dataStoreClient.createQuery(process.env.GOOGLE_DATASTORE_KIND_SCAN_RESULTS)
.filter("userId", QUERY_FILTER_OPERATORS.EQUAL, userId)
.filter("isDeletedDocument", QUERY_FILTER_OPERATORS.EQUAL, false)
.select(["__key__", "scanDate", "scanKeyword", "scanFilter",
"hasRecord", "scanThreatStatus", "scanDuration",
"scanType", "scanStatus", "domainName"])
.order("scanDate", { descending: true })
.limit(10)
if (pageCursor) {
mainQuery = mainQuery.start(pageCursor)
}
const results = await mainQuery.run()
const entities = results[0]
const info = results[1]
const hasNextPage = info.moreResults !== "NO_MORE_RESULTS"
const pageResult = new PageResult(entities, info.endCursor, hasNextPage)
return HttpResult.Ok(resp, pageResult)
}
return HttpResult.UriNotFound(resp)
})
UPDATE:
I tried this with thousands of records and my limit is still 10. It works perfectly for like 2 or 3 queries but when I tried to query for the fourth time, it throws me an error "invalid encoding"
I know this is old, but in case anyone else comes across this issue (as I just did), I was able to resolve it by encoding the cursor value using encodeURIComponent(). It looks like the cursor value occasionally contains a + character, which causes issues when not escaped in the URL
Related
I have 2 Lucid models: Ad and Campaign, which are associated using a Many:Many relationship. They have a pivot table which manages the relationship which has additional information, so my table structure is as follows:
ads
id
...
campaign_ads
campaign_id
ad_id
spend
sent
clicks
leads
ftds
campaigns
id
...
I am trying to fetch the results of a paginate query using the Ad models' query function, but in addition to the Ad models' fields, I would also like to fetch the sum of spend, sent, clicks, leads and ftds from the related Campaign models' pivots.
I have come up with the following code, which returns the correct information in the collection, but returns an incorrect value for the count
const Ad = use('App/Models/Ad');
const query = Ad.query()
.leftJoin('campaign_ads', 'ads.id', 'campaign_ads.ad_id')
.select('ads.*')
.sum('campaign_ads.spend as spend')
.sum('campaign_ads.sent as sent')
.sum('campaign_ads.clicks as clicks')
.sum('campaign_ads.leads as leads')
.sum('campaign_ads.ftds as ftds')
.groupBy('ads.id')
.paginate()
I assume that this is related to how the paginate function rewrites or performs the query, but I have no idea how to fix it.
Here is some example usage based on the answer:
const Ad = use('App/Models/Ad');
const query = Ad.query()
.leftJoin('campaign_ads', 'ads.id', 'campaign_ads.ad_id')
.select('ads.*')
.sum('campaign_ads.spend as spend')
.sum('campaign_ads.sent as sent')
.sum('campaign_ads.clicks as clicks')
.sum('campaign_ads.leads as leads')
.sum('campaign_ads.ftds as ftds')
.groupBy('ads.id')
const paginate = async (query, page = 1, perPage = 20) {
// Types of statements which are going to filter from the count query
const excludeAttrFromCount = ['order', 'columns', 'limit', 'offset', 'group']
// Clone the original query which we are paginating
const countByQuery = query.clone();
// Case Page and Per Page as Numbers
page = Number(page)
perPage = Number(perPage)
// Filter the statments from the array above so we have a query which can run cleanly for counting
countByQuery.query._statements = _.filter(countByQuery.query._statements, (statement) => {
return excludeAttrFromCount.indexOf(statement.grouping) < 0
})
// Since in my case, i'm working with a left join, i'm going to ensure that i'm only counting the unique models
countByQuery.countDistinct([Ad.table, 'id'].join('.'));
const counts = await countByQuery.first()
const total = parseInt(counts.count);
let data;
// If we get a count of 0, there's no point in delaying processing for an additional DB query
if (0 === total) {
data = [];
}
// Use the query's native `fetch` method, which already creates instances of the models and eager loads any relevant data
else {
const {rows} = await query.forPage(page, perPage).fetch();
data = rows;
}
// Create the results object that you would normally get
const result = {
total: total,
perPage: perPage,
page: page,
lastPage: Math.ceil(total / perPage),
data: data
}
// Create the meta data which we will pass to the pagination hook + serializer
const pages = _.omit(result, ['data'])
if (Ad.$hooks) {
await Ad.$hooks.after.exec('paginate', data, pages)
}
// Create and return the serialized versions
const Serializer = Ad.resolveSerializer()
return new Serializer(data, pages);
}
paginate(query, 1, 20)
.then(results => {
// do whatever you want to do with the results here
})
.catch(error => {
// do something with the error here
})
So, as I noted before in my notes, the problem that I was have was caused by how Lucid's query builder handles the paginate function, so I was forced to "roll my own". Here's what I came up with:
paginate (query, page = 1, perPage = 20) {
// Types of statements which are going to filter from the count query
const excludeAttrFromCount = ['order', 'columns', 'limit', 'offset', 'group']
// Clone the original query which we are paginating
const countByQuery = query.clone();
// Case Page and Per Page as Numbers
page = Number(page)
perPage = Number(perPage)
// Filter the statments from the array above so we have a query which can run cleanly for counting
countByQuery.query._statements = _.filter(countByQuery.query._statements, (statement) => {
return excludeAttrFromCount.indexOf(statement.grouping) < 0
})
// Since in my case, i'm working with a left join, i'm going to ensure that i'm only counting the unique models
countByQuery.countDistinct([this.#model.table, 'id'].join('.'));
const counts = await countByQuery.first()
const total = parseInt(counts.count);
let data;
// If we get a count of 0, there's no point in delaying processing for an additional DB query
if (0 === total) {
data = [];
}
// Use the query's native `fetch` method, which already creates instances of the models and eager loads any relevant data
else {
const {rows} = await query.forPage(page, perPage).fetch();
data = rows;
}
// Create the results object that you would normally get
const result = {
total: total,
perPage: perPage,
page: page,
lastPage: Math.ceil(total / perPage),
data: data
}
// Create the meta data which we will pass to the pagination hook + serializer
const pages = _.omit(result, ['data'])
// this.#model references the Model (not the instance). I reference it like this because this function is part of a larger class
if (this.#model.$hooks) {
await this.#model.$hooks.after.exec('paginate', data, pages)
}
// Create and return the serialized versions
const Serializer = this.#model.resolveSerializer()
return new Serializer(data, pages);
}
I only use this version of pagination when I detect group by in my query, and it follow's Lucid's own paginate function pretty closely, and returns identical feedback. While it's not a 100% drop-in solution, it's good enough for my needs
My question is about a strange behavious I notice both on my iPhone device and the codenameone simulator (NetBeans).
I invoke the following code below which calls a google web service to provide a list of food places around a GPS coordinate:
The web service that is called is as follows (KEY OBSCURED):
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXXXXXXXXXXXXXXXXX
Each result contains the next page token and thus, the second call (for the subsequent page) is as follows:
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXXXXXXXXXXXXXXXXX&pagetoken=YYYYYYYYYYYYYYYYYY
public static byte[] getWSResponseData(String urlString, boolean usePost)
{
ConnectionRequest r = new ConnectionRequest();
r.setUrl(urlString);
r.setPost(usePost);
InfiniteProgress prog = new InfiniteProgress();
Dialog dlg = prog.showInifiniteBlocking();
r.setDisposeOnCompletion(dlg);
NetworkManager.getInstance().addToQueueAndWait(r);
try
{
Thread.sleep(2000);
}
catch (InterruptedException ex)
{
}
byte[] responseData = r.getResponseData();
return responseData;
}
public static void getLocationsList(double lat, double lng)
{
boolean done = false;
while (!done)
{
byte[] responseData = getWSResponseData(finalURL,false);
result = Result.fromContent(parser.parseJSON(new InputStreamReader(new ByteArrayInputStream(responseData))));
String venueNames[] = result.getAsStringArray("/results/name");
nextToken = result.getAsString("/next_page_token");
if ( nextToken == null || nextToken.equals(""))
done = true;
else
finalURL = completeURL + "&pagetoken=" + nextToken;
}
.....
}
This code works fine with the sleep timer, but when I remove the Thread.sleep, only the first page gets called.
Any help would be appreciated.
Using the debugger does not help as this is a timing issue and the issue does not occur when using the debugger.
Also when I put some print statements into the code
while (!done)
{
String nextToken = null;
**System.out.println(finalURL);**
...
}
System.out.println("Total Number of entries returned: " + itemCount);
I get the following output:
First Run (WITHOUT SLEEP):
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXX
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXX&pagetoken=CqQCF...
Total Number of entries returned: 20
Using the network monitor I see that the response to the second WS call returns:
{
"html_attributions" : [],
"results" : [],
"status" : "INVALID_REQUEST"
}
Which is strange as when I cut and paste the WS URL into my browser, it works fine...
Second Run (WITH SLEEP):
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXXX
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXXX&pagetoken=CqQCFQEAA...
https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=40.714353,-74.00597299999998&radius=200&types=food&key=XXXXXXXXX&pagetoken=CsQDtQEAA...
Total Number of entries returned: 60
Well it seems to be a google API issue as indicated here:
Paging on Google Places API returns status INVALID_REQUEST
I still could not get it to work by changing the WS URL with a random parameter as they suggested, but I will keep trying and post something here if I get it to work. For now I will just keep a 2 second delay between the calls which seems to work.
Well gave up on using the google WS for this and switched to Yelp, works very well:
https://api.yelp.com/v3/businesses/search?.....
I am using MySql C++ Connector latest build in a c++ app
But every query i make to any table it returns true
E.x. code :
string MyQueryTest(){
string returnValue;
try{
stmt->execute("UPDATE testtbl SET Name = 'Test' WHERE id = '2'");
}
catch(SQLException e){
returnValue = "false";
}
returnValue = "true";
return returnValue;
}
string MyValue = MyQueryTest();
The above code return true though in my testtbl there is no id = 2 data, i have only one data entered which is id = 1 Name = MyTests
Does anyone knows any solutions around this ?
I have tried to place the returnValue = "true" inside the try statement,
also i used exit(1) function inside catch error function
None got me the desired returnValue to false which is the correct return of the above code
It's not an exceptional case when there is no data to be updated. SQLException will be thrown if you have problems with connection to Database or SQL query syntax is incorrect.
Statement class has executeUpdate() method that returns number of affected rows, you can use it to achieve your aim.
When I have an array of Sitecore IDs, for example TargetIDs from a MultilistField, how can I query the ContentSearchManager to return all the SearchResultItem objects?
I have tried the following which gives an "Only constant arguments is supported." error.
using (var s = Sitecore.ContentSearch.ContentSearchManager.GetIndex("sitecore_master_index").CreateSearchContext())
{
rpt.DataSource = s.GetQueryable<SearchResultItem>().Where(x => f.TargetIDs.Contains(x.ItemId));
rpt.DataBind();
}
I suppose I could build up the Linq query manually with multiple OR queries. Is there a way I can use Sitecore.ContentSearch.Utilities.LinqHelper to build the query for me?
Assuming I got this technique to work, is it worth using it for only, say, 10 items? I'm just starting my first Sitecore 7 project and I have it in mind that I want to use the index as much as possible.
Finally, does the Page Editor support editing fields somehow with a SearchResultItem as the source?
Update 1
I wrote this function which utilises the predicate builder as dunston suggests. I don't know yet if this is actually worth using (instead of Items).
public static List<T> GetSearchResultItemsByIDs<T>(ID[] ids, bool mustHaveUrl = true)
where T : Sitecore.ContentSearch.SearchTypes.SearchResultItem, new()
{
Assert.IsNotNull(ids, "ids");
if (!ids.Any())
{
return new List<T>();
}
using (var s = Sitecore.ContentSearch.ContentSearchManager.GetIndex("sitecore_master_index").CreateSearchContext())
{
var predicate = PredicateBuilder.True<T>();
predicate = ids.Aggregate(predicate, (current, id) => current.Or(p => p.ItemId == id));
var results = s.GetQueryable<T>().Where(predicate).ToDictionary(x => x.ItemId);
var query = from id in ids
let item = results.ContainsKey(id) ? results[id] : null
where item != null && (!mustHaveUrl || item.Url != null)
select item;
return query.ToList();
}
}
It forces the results to be in the same order as supplied in the IDs array, which in my case is important. (If anybody knows a better way of doing this, would love to know).
It also, by default, ensures that the Item has a URL.
My main code then becomes:
var f = (Sitecore.Data.Fields.MultilistField) rootItem.Fields["Main navigation links"];
rpt.DataSource = ContentSearchHelper.GetSearchResultItemsByIDs<SearchResultItem>(f.TargetIDs);
rpt.DataBind();
I'm still curious how the Page Editor copes with SearchResultItem or POCOs in general (my second question), am going to continue researching that now.
Thanks for reading,
Steve
You need to use the predicate builder to create multiple OR queries, or AND queries.
The code below should work.
using (var s = Sitecore.ContentSearch.ContentSearchManager.GetIndex("sitecore_master_index").CreateSearchContext())
{
var predicate = PredicateBuilder.True<SearchResultItem>();
foreach (var targetId in f.Targetids)
{
var tempTargetId = targetId;
predicate = predicate.Or(x => x.ItemId == tempTargetId)
}
rpt.DataSource = s.GetQueryable<SearchResultItem>().Where(predicate);
rpt.DataBind();
}
I have a view page that currently has two columns of data shown, soon to be expanded to four. Each column contains the result of a QuerySet for that particular model.
Here's what I have in my views.py method:
if request.REQUEST["type"] == "text":
client = Client.objects.get(client_name = request.REQUEST["search"])
peerList = ClientPeers.objects.prefetch_related().filter(client = client.client)
compList = ClientCompetitors.objects.prefetch_related().filter(client = client.client)
else:
peerList = ClientPeers.objects.prefetch_related().filter(client = request.REQUEST["search"])
compList = ClientCompetitors.objects.prefetch_related().filter(client = request.REQUEST["search"])
for peer in peerList:
peerlst.append({"pid" : peer.parentorg.parentorg, "pname" : peer.parentorg.parentorgname})
for comp in compList:
complst.append({"cid" : comp.parentorg.parentorg, "cname" : comp.parentorg.parentorgname})
lst.append(simplejson.dumps(peerlst))
lst.append(simplejson.dumps(complst))
return HttpResponse(simplejson.dumps(lst), mimetype = "text/json")
This allows me to send a 2D array of data to the browser in the format
[ { //JSON }, { //JSON } ]
In my jQuery.ajax success function, I have
function handler(results) {
var data = JSON.parse(results);
for (var i = 0; i < data[0].length; i++)
$("#available_peers").append("<li>" + data[0][i].pname + "</li>");
for (var i = 0; i < data[1].length; i++)
$("#available_competitors").append("<li>" + data[1][i].cname + "</li>");
Firebug shows that the GET request works and I can see the data in the response tab. However, the console prints out
SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data
var data = JSON.parse(results)
This error disappears if I replace var data = JSON.parse(results) with
var peers = JSON.parse(data[0]);
var comps = JSON.parse(data[1]);
Why does one method work but another doesn't?
The jQuery ajax() call will make an intelligent guess as to the returned data type. In your example, function handler(results), the results variable will already be a decoded JSON object, containing two items in an array. The reason that JSON.parse(data[0]) works, is that you have returned JSON encoded data as a string.
Don't encode the individual list elements to JSON before placing in the output array:
lst.append(peerlst) # <-- Don't encode to JSON string here
lst.append(complst)
return HttpResponse(simplejson.dumps(lst), mimetype = "application/json") # <-- Single JSON encoding