To check if there are any new comments on a post I have to:
Fetch all the posts on a page including the comments.
Compare the just pulled data to the data stored locally.
Is there a more efficient way to do this? For example, checking if post objects are changed since a given date?
You can check updated_time of current post - the time of the last comment on this post.
Two things:
check post total comments number (if it has), or
check and compare the last comments (time or hash)
Related
Heres the relation I'm trying to model in DynamoDB:
My service contains posts and topics. A post may belong to multiple topics. A topic may have multiple posts. All posts have an interest value which would be adjusted based on a combination of likes and time since posted, interest measures the popularity of a post at the current moment. If a post gets too old, its interest value will be 0 and stay that way forever (archival).
The REST api end points work like this:
GET /posts/{id} returns a post-object containing title, text, author name and a link to the authors rest endpoint (doesn't matter for this example) and the number of likes (the interest value is not included)
GET /topics/{name} should return an object with both a list with the N newest posts of the topics as well as one for the N currently most interesting posts
POST /posts/ creates a new post where multiple topics can be specified
POST /topics/
creates a new topic
POST /likes/ creates a like for a specified post (does not actually create an object, just adds the user to the given post-object's list of likers, which is invisible to the users)
The problem now becomes, how do I create a relationship between topics and and posts in DynamoDB NoSql?
I thought about adding a list of copies of posts to tag entries in DynamboDB, where every tag has a list of both the newest and the most interesting Posts.
One way I could do this is by creating a cloudwatch job that would run every 10 minutes and loop through every topic object, finding both the most interesting and newest entries and then replacing the old lists of the topic.
Another job would also have to regularly update the "interest" value of every non archived post (keep in mind both likes and time have an effect on the interest value).
One problem with this is that a lot of posts in the Tag list would be out of date for 10 minutes in case the User makes a change or deletes the post. Likes will also not be properly tracked on the Tags post list. This could perhaps be solved with transactions, although dynamoDB is limited to 10 objects per transaction.
Another problem is that it would require the add-posts-to-tags job to load all the non archived posts into memory in order to manually sort them by both time and interest, split them up by tag and then adding the first N of both sets to the tag lists every 10 minutes.
I also had a another idea, by limiting the tags of a post that are allowed to 1, I could add the tag as a partition key, with the post-time as the sort key, and use a GSI to add Interest as a second sort key.
This does have several downsides though:
very popular tags may be limited to a single parition since all the posts share a single partition key
Tag limit is 1
A cloudwatch job to adjust the Interest value of posts may still be required
It would require use of a GSI which may lead to dangerous race conditions
But it would have the advantage that there are no replications of the post objects aside from the GSI. It would also allow basically infinite paging of all posts by date instead of being limited to just the N newest posts.
So what is a good approach here? It seams both of my solutions have horrible dealbreakers. Is this just one of those problems that NoSQL simply can't solve?
You are trying to model relational data using a non relational DB ,
to do this I would use 2 types of DB ,
I would store in dynamo the post information
in your example it would be :
GET /posts/{id}
POST /posts/
POST /likes/creates
For the topic related information I would use Elastic search (Amazon Elasticsearch Service)
GET /topics/{name} : the search index would stored the full topic info as well post id's that , and the relevant fields you want to search for (in your case update date to get the most recent posts)
what this will entail is background process (in dynamoDB this can be done via streams) that takes changes to the dynamoDB for new post's , update to like count etc.. and populates the search index.
Note: this can also be solved using graphDB but for scaling purposes better separate the source of the data (post's ) and the data relations (topic).
Using the Graph API or FQL, is there a way to efficiently find a user's first post or status? As in, the first one they ever made?
The slow way, I assume, would be to paginate through the feed, but for users like me who joined in 2005 or earlier, that would take a very long time with a huge amount of API calls.
From what I have found, we cannot obtain the date the user registered with Facebook for a good starting point, and we cannot sort by date ascending (not outside of the single page of data returned) to get the oldest post on top.
Is there any reasonable way to do this?
you can use facebook query language (FQL) to get first post information.
Please refer below query for more details :-
SELECT message, time FROM status WHERE uid= me() ORDER BY time ASC LIMIT 1
Please check and let me know in case of any issue.
Thanks and Regards
Durgaprasad
I think the Public API is limited to the depth of information it is allowed to query. Facebook probably put in these constraints for performance and cost concerns. Maybe they've changed it. When I tried to go backwards thru a person's stream about 4 months ago, there seemed to be a limit as to how far back I could go. Maybe it's a time limit or a # posts back limit. If you know when your user first posted, then getting to it should be fairly quick using the since/until time stamps in your queries.
In order to go through all notes of a user I first want to know the total number available. The only way I found so far is to get them all by setting the limit to a high enough value - but this is pretty slow since all note objects will be fetched including their content. There must be a more effective way since the normal public https://www.facebook.com/<user>?sk=notes page displays the total at the bottom line very quick.
You can't get only number of notes without fetching the details for all of 'em.
You can however speed the query by limiting fields retrieved by adding fields argument to notes connection URL:
http://graph.facebook.com/me/notes?fields=id&access_token=...
This will only return ids of notes without all the rest of details (which should be faster that retrieval of all notes data).
I am inserting data into two tables, however I can not figure out (after hours of Googling) how to insert data into the second table after retrieving the new ID created after the first update?
I'm using <CFINSERT>.
use <CFQUERY result="result_name"> and the new ID will be available at result_name.generatedkey .. <cfinsert> and <cfupdate>, while easy and fast for simple jobs, they are pretty limited.
I have never used cfinsert myself, but this blog post from Ben Forta says you may not be able to use cfinsert if you need a generated key http://www.forta.com/blog/index.cfm/2006/10/3/Use-CFINSERT-And-CFUPDATE
Yes, I realize that blog post is old, but it doesn't appear much has changed.
Why not use a traditional INSERT statement wrapped in a <cfquery> tag?
I'm seeing a discrepancy between the number of likes reported in the Graph API vs the number of entries in the "data" that has the name and ID of the people who liked a post.
When I view a certain post on Facebook, I see that it has 5 people who have liked it.
When I use the Graph API to fetch the post, the "likes" field has a "data" field with 3 entries in it, and a "count" field whose value is 5.
When I use the Graph API to fetch the likes for the post (eg, {post_id}/likes), I get a "data" field with 5 entries in it (and no "count" field).
Clearly the true answer to how many people have liked the post is 5. But then why is there only 3 entries in the "data" when I fetch the post object?
Here's another example of the same discrepancy:
https://graph.facebook.com/40796308305_10150394134258306 returns data for a post whose "likes/data" only has 1 entry in it, but whose "likes/count" says that there are 3. But https://graph.facebook.com/40796308305_10150394134258306/likes returns "data" with 3 entries. Finding that same entry on Coca-Cola's page finds that there are, in fact, 3 people who have liked it.
The documentation of the post object doesn't mention that the likes list may be incomplete, and the documentation of the fql stream table explicitly says to use the post object to get the full list, so It's either a bug in the API or in the documentation.
I suspect it may be a deliberate but undesirable "feature" to limit the detailed list for performance reasons, as some posts may have hundreds or even thousands of likes.
It ends up actually causing a huge performance problem as I need to find all posts that have been liked by a particular user, and the only way to do that is to do a separate fetch of likes for each post in the list whose like count is higher than the like list length.
2 people have their privacy settings set to not show their name to people who are not their friends.