Amplify AppSync: custom sorting and filtering with pagination - amazon-web-services

I'm trying to write a schema so that I can query models filtered by multiple keys, sorted by a custom key and paginated.
an example of my model:
type Article {
id: ID!
category: String!
area: String!
publishOn: AWSDate!
}
And an example of the query I would like to do is: retrieve all the Articles which are part of both a given category AND area, returned in descending order by publishOn in chunks of 10 items each (to implement pagination server-side, and have a lightweight UI).
The response should include also the nextToken attribute that can be used to load the "next" page of the filtered articles list.
I have multiple problems with what I can do with the automatically generated schema and can't find a way to implement manually a solution that works for all what I want to do. I try and make a list of what goes wrong:
Filtering
Let's say I want to query 10 articles that belong to the category "Holiday":
listArticles(filter: {category: {eq: "Holiday} }, limit: 10)
I won't get the first 10 articles that match that category, but instead, it seems that AppSync selects the first 10 items in the table, and then it filters these 10 items by the filter criteria.
In other words, it seems that the sequence in which filtering and sorting are applied is the opposite of what expected. Expected: firstly filter the table by the filter critaria, then return the first 10 items of the filtered result sets.
Sorting
I couldn't find a way to add sorting with AppSync, so I added searchable:
type Article (
#searchable
) {
id: ID!
category: String!
area: String!
publishOn: AWSDate!
}
Now if I sort by date, that key will be used as nextToken and brake the pagination. This is a known issue: https://github.com/aws-amplify/amplify-cli/issues/4434
Do you have any good tip on how to find a workaround to these bugs? I dag into the documentation and in couple of issue, but didn't come up with a solution that works well...
Thanks in advance,
Matteo

Filtering
You will need a Global Secondary Index in DynamoDB to achieve such a behaviour. You can create them with the #key annotation. I your case I would create a composite key consisting of the category for the partition key and area and publishOn as the sort key(s).
type Article
#model
#key(fields: ["id"])
#key(name: "byCategory", fields: ["category", "publishOn"])
#key(name: "byCategoryArea", fields: ["category", "area", "publishOn"])
{
id: ID!
category: String!
area: String!
publishOn: AWSDate!
}
Sorting
Sorting is done by the sortDirection property which is either DESC or ASC and can only be done on the sort key.
The #searchable directive enables elasticsearch on the table, which is a fulltext search engine and probably a bit pricy for small applications and wouldn't be required here unless you would want to query based on e.g. the article description text.
listArticles(filter: {category: {eq: "Holiday"} }, limit: 10, sortDirection: DESC)

Amplify AppSync: filtering with pagination
let allClubsList = async (sport) => {
try {
let clubsList;
let clubsInfoList = [];
let nextTokenInfo = null;
do{
let clubs = await client.query({
query: gql(clubBySportStatus),
variables: {
sport: sport,
eq: { status: "ACTIVE" },
},
limit: 100,
nextToken: nextTokenInfo,
fetchPolicy: "network-only",
});
clubsList = clubs.data.clubBySportStatus.items;
clubsList.forEach((item) => clubsInfoList.push(item));
nextTokenInfo = clubs.data.clubBySportStatus.nextToken;
} while (Boolean(nextTokenInfo));
if (clubsInfoList && clubsInfoList.length) {
return {
success: true,
data: clubsInfoList,
};
}
} catch (eX) {
console.error(`Error in allClubsList: ${JSON.stringify(eX)}`);
return {
success: false,
message: eX.message,
};
}
};

Related

Best approach for Posts and PostReactions in AWS Amplify and DynamoDB

I am working on chat functionality using AWS Amplify and I have a simple Post model in my graphql schema:
type Post
...
{
id: ID!
channelId: ID #index(
name: "byChannel", sortKeyFields: ["createdAt"],
queryField: "listPostsByChannel"
)
customerId: ID #index(
name: "byCustomer", sortKeyFields: ["postType", "createdAt"]
)
text: String!
postTempId: String
postType: String
reactions: [PostReaction] #hasMany(fields: ["id"])
createdAt: AWSDateTime
updatedAt: AWSDateTime
}
What I want to achieve is to have similar to other popular chat apps - reactions with emojis attached to each post, so I've created another table and the PostReaction model.
type PostReaction
...
{
postId: ID! #primaryKey(sortKeyFields: ["customerId", "emojiUnicode"])
customerId: String!
customerMeta: CustomerMeta
emojiUnicode: String!
createdAt: AWSDateTime
updatedAt: AWSDateTime
}
Of course, each customer could add multiple emojis to a single post, the custom primary key is for handling duplicates later.
There is one disadvantage here.
Emojis will be listed in an array in the reactions field in the post, even if it's the same emoji added by many people.
Instead of a simple array of reactions that frontend would need to merge for each post, the best would be to get a result from the AppSync query for each Post like:
...
reactions: [{
emojiUnicode: "U+1F44D",
customerIds: ["ID1234", "ID5678"],
...
}, {...}]
I thought that I can use a JSON object in the reactions field, but the DynamoDB has the max size limit for a single item which is 400KB. That's not a problem for now, but next when I will add more attributes to the Post model, and when there will be many reactions from many people at the same time, this might be an issue.
Is there an option how to achieve this in the simplest way?
Best thing to not over-complicate your schema would be to enforce a maximum number of emojis just as Slack does for example:
You can add up to 23 emoji reactions to any message, but the maximum per message is 50 unique emoji.
Other than that, you could keep an item for each emoji reacted
pk
sk
data
thread123
metadata
metadata about thread
thread123
post#001
First message in thread
thread123
post#002
Second message in thread
thread123
post#003
Third message in thread
thread123
post#003#emoji#U+1F44D
[user1, user2, user45]
thread123
post#003#emoji#U+1F33R
[user56, user8, user7, user10]
Now when you want all the data to populate a given thread on your UI, you just issue a query with the pk as a parameter:
SELECT * FROM table WHERE PK = 'thread123'

Filtering List Query By Another Table's Field (a.k.a Cross-Table or Nested Filtering) in AWS Amplify GraphQL DynamoDB

Which Category is your question related to?
DynamoDB, AppSync(GraphQL)
Amplify CLI Version
4.50.2
Provide additional details e.g. code snippets
BACKGROUND:
I'm new in AWS serverless app systems and as a frontend dev, I'm quite enjoying it thanks to auto-generated APIs, tables, connections, resolvers etc. I'm using Angular/Ionic in frontend and S3, DynamoDB, AppSync, Cognito, Amplify-cli for the backend.
WHAT I HAVE:
Here is a part of my schema. I can easily use auto-generated APIs to List/Get Feedbacks with additional filters (i.e. score: { ge: 3 }). And thanks to the #connection I can see the User's details in the listed Feedback items.
type User #model #auth(rules: [{ allow: owner }]) {
id: ID!
email: String!
name: String!
region: String!
sector: String!
companyType: String!
}
type Feedback #model #auth(rules: [{ allow: owner }]) {
id: ID!
user: User #connection
score: Int!
content: String
}
WHAT I WANT:
I want to list Feedbacks based on several fields on User type, such as user's region (i.e. user.region: { contains: 'United States' }). Now I searched for a solution quite a lot like, #2311 , and I learned that amplify codegen only creates top-level filtering. In order to use cross-table filtering, I believe I need to modify resolvers, lambda functions, queries and inputs. Which, for a beginner, it looks quite complex.
WHAT I TRIED/CONSIDERED:
I tried listing all Users and Feedbacks separately and filtering them in front-end. But then the client downloads all these unnecessary data. Also because of the pagination limit, user experience takes a hit as they see an empty list and repeatedly need to click Load More button.
Thanks to some suggestions, I also thought about duplicating the User details in Feedback table to be able to search/filter them. Then the problem is that if User updates his/her info, duplicated values will be out-of-date. Also there will be too many duplicated data, as I need this feature for other tables also.
I also heard about using ElasticSearch for this problem but someone mentioned for a simple filtering he got 30$ monthly cost, so I got cold feet.
I tried the resolver solution to add a custom filtering in it. But I found that quite complex for a beginner. Also I will need this cross-table filtering in many other tables as well, so I think would be hard to manage. If that is the best-practice, I'd appreciate it if someone can guide me through it.
QUESTIONS:
What would be the easiest/beginner-friendly solution for me to achieve this cross-table filtering? I am open to alternative solutions.
Is this cross-table filtering a bad approach for a no-SQL setup? Since I need some relationship between two tables. (I thought #connection would be enough). Should I switch to an SQL setup before it is too late?
Is it possible for Amplify to auto-generate a solution for this in the future? I feel like many people are experiencing the same issue.
Thank you in advance.
Amplify, and really DynamoDB in general, requires you to think about your access patterns ahead of time. There is a lot of really good information out there to help guide you through what this thought process can look like. Particularly, I like Nader Dabit's https://dev.to/dabit3/data-modeling-in-depth-with-graphql-aws-amplify-17-data-access-patterns-4meh
At first glance, I think I would add a new #key called byCountry to the User model, which will create a new Global Secondary Index on that property for you in DDB and will give you some new query methods as well. Check out https://docs.amplify.aws/cli/graphql-transformer/key#designing-data-models-using-key for more examples.
Once you have User.getByCountry in place, you should then be able to also bring back each user's Feedbacks.
query USAUsersWithFeedbacks {
listUsersByCountry(country: "USA") {
items {
feedbacks {
items {
content
}
nextToken
}
}
nextToken
}
}
Finally, you can use JavaScript to fetch all while the nextToken is not null. You will be able to re-use this function for each country you are interested in and you should be able to extend this example for other properties by adding additional #keys.
My former answer can still be useful for others in specific scenarios, but I found a better way to achieve nested filtering when I realized you can filter nested items in custom queries.
Schema:
type User #model {
id: ID!
email: String!
name: String!
region: String!
sector: String!
companyType: String!
feedbacks: [Feedback] #connection # <-- User has many feedbacks
}
Custom query:
query ListUserWithFeedback(
$filter: ModelUserFilterInput # <-- Filter Users by Region or any other User field
$limit: Int
$nextToken: String
$filterFeedback: ModelFeedbackFilterInput # <-- Filter inner Feedbacks by Feedback fields
$nextTokenFeedback: String
) {
listUsers(filter: $filter, limit: $limit, nextToken: $nextToken) {
items {
id
email
name
region
sector
companyType
feedbacks(filter: $filterFeedback, nextToken: $nextTokenFeedback) {
items {
content
createdAt
id
score
}
nextToken
}
createdAt
updatedAt
}
nextToken
}
}
$filter can be something like:
{ region: { contains: 'Turkey' } }
$filterFeedback can be like:
{
and: [{ content: { contains: 'hello' }, score: { ge: 4 } }]
}
This way both Users and Feedbacks can be filtered at the same time.
Ok thanks to #alex's answers I implemented the following. The idea is instead of listing Feedbacks and trying to filter them by User fields, we list Users and collect their Feedbacks from the response:
Updated schema.graphql as follows:
type User
#model
#auth(rules: [{ allow: owner }])
#key(name: "byRegion", fields: ["region"], queryField: "userByRegion") # <-- added byRegion key {
id: ID!
email: String!
name: String!
region: String!
sector: String!
companyType: String!
feedbacks: [Feedback] #connection # <-- added feedbacks connection
}
Added userFeedbacksId parameter while calling CreateFeedback. So they will appear while listing Users.
Added custom query UserByRegionWithFeedback under src/graphql/custom-queries.graphl and used amplify codegen to build it:
query UserByRegionWithFeedback(
$region: String
$sortDirection: ModelSortDirection
$filter: ModelUserFilterInput
$limit: Int
$nextToken: String # <-- nextToken for getting more Users
$nextTokenFeedback: String # <-- nextToken for getting more Feedbacks
) {
userByRegion(
region: $region
sortDirection: $sortDirection
filter: $filter
limit: $limit
nextToken: $nextToken
) {
items {
id
email
name
region
sector
companyType
feedbacks(nextToken: $nextTokenFeedback) {
items {
content
createdAt
id
score
}
nextToken
}
createdAt
updatedAt
owner
}
nextToken
}
}
Now I call this API like the following:
nextToken = {
user: null,
feedback: null
};
feedbacks: any;
async listFeedbacks() {
try {
const res = await this.api.UserByRegionWithFeedback(
'Turkey', // <-- region: filter Users by their region, I will add UI input later
null, // <-- sortDirection
null, // <-- filter
null, // <-- limit
this.nextToken.feedback == null ? this.nextToken.user : null, // <-- User nextToken: Only send if Feedback NextToken is null
this.nextToken.feedback // <-- Feedback nextToken
);
// Get User NextToken
this.nextToken.user = res.nextToken;
// Initialize Feedback NextToken as null
this.nextToken.feedback = null;
// Loop Users in the response
res.items.map((user) => {
// Get Feedback NextToken from User if it is not null (Or else last User in the list could overrite it)
if (user.feedbacks.nextToken) {
this.nextToken.feedback = user.feedbacks.nextToken;
}
// Push the feedback items into the list to diplay in UI
this.feedbacks.push(...user.feedbacks.items);
});
} catch (error) {
this.handleError.show(error);
}
}
Lastly I added a Load More button in the UI which calls listFeedbacks() function. So if there is any Feedback NextToken, I send it to the API. (Note that multiple user feedbacks can have a nextToken).
If all feedbacks are ok and if there is a User NextToken, I send that to the API and repeat the process for new Users.
I believe this could be much simpler with an SQL setup, but this will work for now. I hope it helps others in my situation. And if there is any ideas to make this better I'm all ears.

aws mock api able to query but unable to delete entries

shown below is the graphql Schema.
#auth(rules: [{ allow: owner,operations: [create, delete ] ,ownerField: "user"}])
{
id: ID!
videoKey: String!
videoThumbnailKey :String!
videoTitle:String!
videoDescription:String!
channelName:String!
videoLikes: Int
videoDislikes: Int
comments: [Comment] #connection(keyName: "byVideo", fields: ["id"])
user: String
}
type Comment #model
#auth(rules: [{ allow: owner,operations: [create,delete] ,ownerField: "user"}])
#key(name: "byVideo", fields: ["videoID", "comment"]) {
id: ID!
videoID: ID!
comment: String!
video: Video #connection(fields: ["videoID"])
user: String
}
type LikedVideos #model
{
id:ID!
video:[Video]#connection
}
this is the query for listing videos
and these are the results
this is the query for getting a video
and these are the results
all of these works fine, but when I try to delete it I get error.
this is the mutation query
and this is the error response i got
Seems like you have not added sort key in dynamo db table and the other items have the same primary key as of first item. I faced similar issue and worked after adding sort key.
This seems to be another bug in AWS Amplify or local DynamoDB.
I am using the same schema - when I add even just a space to schema.grapql, first delete operation works. The delete operations executed after are not working.
Also, it seems like something is throttling query response (I am using amplify mock command):
E.g. I have 7 items in the database and when I am trying to list all of them, I got responses with 1, 2, or even zero items, multiple times - when I would expect, that they will be loaded in 1 request, at the same time.
I am using AWS Amplify
"aws-amplify": "^3.1.1",
"aws-amplify-react-native": "^4.2.5",
Even if you modify schema.graphql, it's still unreliable.

AWS AppSync only returns 10 items on query on connection

I'm new to AppSync and trying to see how this works and what's the proper way to set this up.
I created schema.graphql looks like below.
type User #model {
id: String!
following: [String]
follower: [String]
journals: [Journal] #connection(name: "UserJournals", sortField: "createdAt")
notifications: [Notification] #connection(name: "UserNotifications", sortField: "createdAt")
}
type Journal #model {
id: ID!
author: User! #connection(name: "UserJournals")
privacy: String!
content: AWSJSON!
loved: [String]
createdAt: String
updatedAt: String
}
and this created queries.js automatically by AppSync.
export const getUser = `query GetUser($id: ID!) {
getUser(id: $id) {
id
following
follower
journals {
items {
id
privacy
content
loved
createdAt
updatedAt
}
nextToken
}
notifications {
items {
id
content
category
link
createdAt
}
nextToken
}
}
}
`;
I noticed that querying getUser only returns 10 journals items and not sure how to set that to more than 10 or proper way to query and add more journals into that 10 items that were queried by getUser.
Since you do not pass the limit argument explicitly in your query, the Request Mapping Template of the journals resolver defaults it to 10 items. If you would like to change this default value, go to your schema page on the AppSync console, navigate to the journals field, found under the Resolvers section of the schema page. This will then show the resolver definition for this field, and you can then update the default value of 10 to anything you like. Alternatively, you can pass this as your query argument.
FYI - This default value is defined in the amplify-cli repo on GitHub and can be found here.

How to make a geo distance query only return results that I have not yet "Liked", Elastic Search, Dynamo DB

I am trying to query my database to compile a list of posts near a location, that a given user has/hasnot liked.
allPostsNearLocationUserHasLiked(
userId: ID,
location: LocationInput,
radius: Int
): [Post]
allPostsNearLocationUserHasNotLiked(
userId: ID,
location: LocationInput,
radius: Int
): [Post]
To do this I am currently using AWSAppSync, with dynamoDB streaming data to Elastic Search. This allows me to easily do geospatial searches and obtain all the posts near a given location.
I am wondering what is the most efficient way is to compute this using dynamoDB? Or would it be better suited for me to switch over to a SQL database for my likes/users?
I have a Users, posts, and Likes DynamoDB table. I was thinking of using a pipeline resolver to:
1) Get the list of all the posts near a users location (Elastic Search)
2) Query the likes table, to get all the likes I have made (DynamoDB)
3) Combine the results item by item.
I have serious doubts about the performance of this especially step 3 which is an O(M*N) operation.
Is there any way to do this whole query natively in Elastic Search?
## DynamoDB Table?? Or maybe SQL?
type Like {
likeId: ID!
userId: ID!
likedPostId: ID!
}
type Query {
#Implement with Elastic Search
allPostsNearLocation(location: LocationInput, radius: Int): [Post]
## Elastic search???
allPostsNearLocationUserHasLiked(
userId: ID,
location: LocationInput,
radius: Int
): [Post]
allPostsNearLocationUserHasNotLiked(
userId: ID,
location: LocationInput,
radius: Int
): [Post]
}
type Location {
lat: Float!
lon: Float!
}
input LocationInput {
way: Float!
lon: Float!
}
type Mutation {
putPost(
author: String!,
title: String!,
content: String!,
location: LocationInput!,
url: String!
): Post
putUser(name:String): User
likePost(userId: ID!, postId: ID!): Like
}
#DynamoDB Table
type User{
userId: ID!
name: String
likes: [Like]
}
#DynamoDB table
type Post {
id: ID!
author: String!
title: String!
content: String!
url: String!
location: Location!
}
schema {
query: Query
mutation: Mutation
}
I'm not sure what is your performance requirements, but I think that your initial plan should be ok, if:
1) Get the list of all the posts near a users location (Elastic Search)
This should be fast, if the Elasticsearch index is set up with the right mapping, size, sharding and hardware depends on the data size.
2) Query the likes table, to get all the likes I have made (DynamoDB)
This can be fast, in case you could have an in memory cache of the 'likes', either fully in memory, or lazy/LRU cache.
3) Combine the results item by item.
if the result size is not too large (use pages of 10-100 items?), then getting the response from Elasticsearch, running on that stream and enriching/filtering it based on an in memory dictionary should be OK.
Good luck!
Your initial solution makes sense to me, though the concerns about latency could prove valid. Have you looked into relational data sources? At this point, it only supports Aurora Serverless, so your comment about SQL as an alternative would be the case here, using it's built in Geo-spacial data types instead of those of Elasticsearch.
You could build out a schema entirely in RDS this way, eliminating the need for a pipeline. A complex SQL query using joins could build up your combination of posts based on locations rather than having a three-part job to do it.