I have a Rest Webservice that is intended to deliver rows from a domain table Partners.
Access by key
http//..../partner/get?id=key
is working fine as it delivers the desired row
but Access by
http//..../partner
that is intended to deliver all partner rows is limited by 10
Where in grails framework or in the browse or elsewhere is the means to increase that limit?
peter
Related
I am using the Adobe Analytics connector for one of my reports. One of my 'Dimensions' is called 'Websites' and the data is a URL (www.myurl.com), we look at over 100 websites and I am looking at the traffic for each one. When I use the Websites dimension it only displays 10 of the sites out of the total number. Is there a way I can add a line of code in the advanced query editor to remove this limit? Or is this a limitation with the Power BI connector?
Using the 'Top parameter'(returns up to 50,000 dimensions per call) fixes my problem and breaks down my URL column by the correct number of dimensions as opposed to 10.
{Cube.ApplyParameter, "Top", {10000, "evar123"}},
I am working on a startup company where we would sell an IoT device of some sort . these devices will be connected to our server hosted in Google cloud and will send data every 1 second where my server will store it in database as a time series. Let's say we have 1000 device connected and all are sending their data every second , Is it suitable to use google bigquery to insert these data in table every second for each device to it's corresponding table to the owner of the device ?
since my data is in form of a time series i am thinking of using partitioned table for each user ( owner of my device ) but with the limits and quotas listed in the official documentation i am worrying of reaching the limit with my high number of inserts every second ( not to say that I will query the data based on user demand on my phone app ) .
if it's not suitable what would be suited for my use case ?
EDIT : my main concern is the huge amount of inserts per second which can exceeds BigQuery limits or might cause slow down since it's mainly for data warehouse . BigTable seems expensive for us and CloudSQL it seems the way to go but we are worried of slow query times once the table get filled since i am inserting 86400 row per user per day .
Thanks.
You should check out CLOUD IOT CORE - fully managed service to easily and securely connect, manage, and ingest data from globally dispersed devices
Device data captured by Cloud IoT Core gets published to Cloud Pub/Sub for downstream analytics. You can do ad hoc analysis using Google BigQuery, easily run advanced analytics and apply machine learning with Cloud Machine Learning Engine, or visualize IoT data results with rich reports and dashboards in Google Data Studio.
Check also IoT Core with PubSub, Dataflow, and BigQuery
We have written a small monitoring web site for our customer that shows the current status of their Navision system.
This was done by creating a codeunit that has a number of functions that gather data and counts for job queues etc and published the data through a page, which has been configured to be used as a web service.
The website then calls this web service and displays the data dynamically.
My question is this.
One of the counts that the codeunit creates is the number of Sessions (from the Virtual table "Session") where the Application Name is either "Microsoft Dynamics NAV Classic client" or "Application Server for Microsoft Dynamics NAV Classic". When the codeunit is run from Navision (and displayed in a list form, or displayed in an RTC page) then the correct count is displayed.
However, when calling the published page as a web service using ASP.Net, the count is always 0.
It seems the web service is somehow, unable or unwilling to give us this count.
The code in the codeunit that gets this count is as follows.
tblSession.RESET;
tblSession.SETFILTER("Application Name",
'Microsoft Dynamics NAV Classic client|Application Server for Microsoft Dynamics NAV Classic');
intValue := tblSession.COUNT;
I have even tried writing a brand new codeunit just to get the count of sessions by itterating through them, checking the Application Name, and if its one of our chosen types, incrementing a count, and returning this as a return parameter in the function. It still returns 0.
Has anyone else experienced this, or does anyone know why the data does not carry through the web service?
It appears that in Nav 2015 there is a table named "Active Session". If you want a count of the current sessions, try counting the records from this table instead.
This is regarding Microsoft sync framework where we are syncing Ipad data with Sql server database.
It is working fine.
But here, I want to limit my records to be synched to only 20 records at a time. Right now, all the records are getting synched.
I there a out-of box feature by sync framework which will enable us to do so.
If not, how can I write a custom code to achieve this.
you should be able to set batching. but it's in terms of size, not number of rows. you can simply get your row size and multiply by the number of rows you have in mind.
lookup SetDownloadBatchSize and SetBatchSpoolDirectory in the documentation.
e.g. config.SetDownloadBatchSize = some value in Kb
In SOA we should not be building or holding state (or designing dependencies) between client and server. This is understood. But what patterns can be followed in the case that a client wants to consume a real-time service that may return an open ended number of 'rows'?
Web applications, similar to SOA but allowing for state (sessions) have solved this with pagination. Pagination requires (in most cases, especially with SQL) that the server holds the data and that the client request the data in chunks.
If we where to consider pagination-like scenarios for web services, what patterns would these follow that would still allow the tenets of SOA to be adhered (or as close as possible).
Some rules for the thinkers:
1) Backed by a SQL database (therefore there is no concept of a row number in a select set)
2) It is important to not skip a row or duplicate a row in a set during pagination
3) Data may be inserted and deleted at any time into the database by other clients
4) There is no need to consider the dataset a live (update-able) dataset
Personally, I think that 1 and 2 above already spell our the solution by constraining the solution space with the requirements.
My proposed solution would have the data (as much as is selected) be stored in a read-only store/cache where it can be assigned a row number within the result set and allow pagination to occur on this data snapshot. I have would have infrastructure to store snapshots (servers, external caches, memcached or ehcache - this must scale quite large). The result of such a query would be a snapshot ID and clients could retrieve the data from the snapshot using a snapshot API (web services) and the snapshot ID. Results would be processed in a read-only, forward only manner for x records at a time where x was something reasonable.
Competing thoughts and ideas, criticisms or accolades would be greatly appreciated.
Paginated results in a Web Service is actually quite easy to achieve.
All you have to do is add two parameters to the web service call: Page Size, Page Number.
Page Size is the number of results to include in a page. Page Number is the number of the page of results you are looking for.
Your web service then goes back to the database (or cache), retreives the results, figures out which results fit on the requested page, and return only those results.
The client then has to make a single request per page of results they want from the service.
What you propose with memcached will also work with a caching table. The first service call would (1) INSERT results INTO the caching table with a snapshot ID (2) return the first page from the caching table and the snapshot ID. Subsequent calls would return pages based on page size and page number by querying the caching table using the snapshot ID.
I should think this could also be optimized by using an in-memory caching table, but that depends on whether your database supports INSERT-INTO from a disk table to an in-memory table. That might get complicated in a clustered environment though.
Such a cache is stateful by its very nature if you are retaining a client-specific copy between requests, whether storage is in a session object, database table or memcached data store. Given the requirements though, you have no choice but to cache results in some form or another, except you risk the chance of returning deleted or no-longer-relevant records as legitimate results.
SOA is not meant for such low level functionality.
SOA is meant to glue together business areas, not frontends to backends. Not because your application talks to the back end using webservices you have a "SOA" application. This is non sense since SOA is meaningless in the context of 1 isolated system.
From that point of view, it is then clear that, in SOA, the caller should not have known about the SQL table you are paginating, that’s an implementation detail that SOA should hide. In the other hand the server should not know about the client's state, because it should be agnostic to the details of the clients, to be really open.
So, just understand that pagination is not SOA. Do as you wish, just understand that the webservice you are using to paginate is an internal artifact of your application, not to be used for external clients in a SOA bus. Also remember that it can not be transaction consistent with out state in the server. Probably the problem is that you have only one service layer for the application's UI and the SOA bus, you need to separate them.
Using this webservice in a SOA bus would be bad. I can not be consistent as the user paginates and as other applications hang to it they become tied to the specific SQL.
... then you might as well have granted direct SQL access to the table for all that matters.
SOA is for business messages between systems, not to glue an application's frontend to the backend.
Same problem, resolved using the Navision approach.
$ws->getList($first_record_id, $limit)
This return a page of $limit element that start from the the passed id
select * from collection where collection.id > $first_record_id ASC limit $limit
ordered by id ASC
Navision use Key (each element has a key) but in MySQL an autoincrement id is better.
In this case pagination is intended for handle large result sets and not for a frontend pagination...
I am not sure if SOA is of concern here. The problem you have seems to be with paginating your API's. I will point you to how twitter handles their pagination dev.twitter.com/rest/public/timelines