I have a DynamoDB table with feed_guid as the global secondary index. I want to query with a set of feed_guid in that table. Since feed_guid is not my primary keys, I can't use getBatchItem. When I tried the following method, I got this error:
Invalid operator used in KeyConditionExpression: OR
$options = array(
'TableName' => 'feed',
'IndexName' => 'GuidIndex',
'KeyConditionExpression' => 'feed_guid = :v_guid1 or feed_guid = :v_guid2',
'ExpressionAttributeValues' => array (
':v_guid1' => array('S' => '8a8106e48bdbe81bf88d611f4b2104b5'),
':v_guid2' => array('S' => '19cab76242a6d85717de64fe4f8acbd4')
),
'Select' => 'ALL_ATTRIBUTES',
);
$response = $dynamodbClient->query($options);
You can't use OR condition. You should use
rangeAttributeName BETWEEN :rangeval1 AND :rangeval2
if possible or
feed_guid IN (:v_guid1, :v_guid2)
See ExpressionAttributeValues and KeyConditionExpression
In order to achieve what you want here, you'll need to take the union of two separate queries.
Currently, DynamoDB's Query API only supports having one condition on Hash AND Range Keys in only the KeyConditionExpression because this limits the items you search and ultimately reduces cost of say a more complex query like what you've described here.
Related
I have a DynamoDB table called 'frank' with a single GSI. The partition key is called PK, the sort key is called SK, the GSI partition key is called GSI1_PK and the GSI sort key is called GSI1_SK. I have a single 'data' map storing the actual data.
Populated with some test data it looks like this:
The GSI partition key and sort key map directly to the attributes with the same names within the table.
I can run a partiql query to grab the results that are shown in the image. Here's the partiql code:
select PK, SK, GSI1_PK, GSI1_SK, data from "frank"."GSI1"
where
("GSI1_PK"='tesla')
and
(
( "GSI1_SK" >= 'A_VISITOR#2021-06-01-00-00-00-000' and "GSI1_SK" <= 'A_VISITOR#2021-06-20-23-59-59-999' )
or
( "GSI1_SK" >= 'B_INTERACTION#2021-06-01-00-00-00-000' and "GSI1_SK" <= 'B_INTERACTION#2021-06-20-23-59-59-999' )
)
Note how the partiql code references "GSI1_SK" multiple times. The partiql query works, and returns the data shown in the image. All great so far.
However, I now want to move this into a Lambda function. How do I structure a AWS.DynamoDB.DocumentClient query to do exactly what this partiql query is doing?
I can get this to work in my Lambda function:
const visitorStart="A_VISITOR#2021-06-01-00-00-00-000";
const visitorEnd="A_VISITOR#2021-06-20-23-59-59-999";
var params = {
TableName: "frank",
IndexName: "GSI1",
KeyConditionExpression: "#GSI1_PK=:tmn AND #GSI1_SK BETWEEN :visitorStart AND :visitorEnd",
ExpressionAttributeNames :{ "#GSI1_PK":"GSI1_PK", "#GSI1_SK":"GSI1_SK" },
ExpressionAttributeValues: {
":tmn": lowerCaseTeamName,
":visitorStart": visitorStart,
":visitorEnd": visitorEnd
}
};
const data = await documentClient.query(params).promise();
console.log(data);
But as soon as I try a more complex compound condition I get this error:
ValidationException: Invalid operator used in KeyConditionExpression: OR
Here is the more complex attempt:
const visitorStart="A_VISITOR#2021-06-01-00-00-00-000";
const visitorEnd="A_VISITOR#2021-06-20-23-59-59-999";
const interactionStart="B_INTERACTION#2021-06-01-00-00-00-000";
const interactionEnd="B_INTERACTION#2021-06-20-23-59-59-999";
var params = {
TableName: "frank",
IndexName: "GSI1",
KeyConditionExpression: "#GSI1_PK=:tmn AND (#GSI1_SK BETWEEN :visitorStart AND :visitorEnd OR #GSI1_SK BETWEEN :interactionStart AND :interactionEnd) ",
ExpressionAttributeNames :{ "#GSI1_PK":"GSI1_PK", "#GSI1_SK":"GSI1_SK" },
ExpressionAttributeValues: {
":tmn": lowerCaseTeamName,
":visitorStart": visitorStart,
":visitorEnd": visitorEnd,
":interactionStart": interactionStart,
":interactionEnd": interactionEnd
}
};
const data = await documentClient.query(params).promise();
console.log(data);
The docs say that KeyConditionExpressions don't support 'OR'. So, how do I replicate my more complex partiql query in Lambda using AWS.DynamoDB.DocumentClient?
If you look at the documentation of PartiQL for DynamoDB they do warn you, that PartiQL has no scruples to use a full table scan to get you your data: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-reference.select.html#ql-reference.select.syntax
To ensure that a SELECT statement does not result in a full table scan, the WHERE clause condition must specify a partition key. Use the equality or IN operator.
In those cases PartiQL would run a scan and use a FilterExpression to filter out the data.
Of course in your example you provided a partition key, so I'd assume that PartiQL would run a query with the partition key and a FilterExpression to apply the rest of the condition.
You could replicate it that way, and depending on the size of your partitions this might work just fine. However, if the partition will grow beyond 1MB and most of the data would be filtered out, you'll need to deal with pagination even though you won't get any data.
Because of that I'd suggest you to simply split it up and run each or condition as a separate query, and merge the data on the client.
Unfortunately, DynamoDB does not support multiple boolean operations in the KeyConditionExpression. The partiql query you are executing is probably performing a full table scan to return the results.
If you want to replicate the partiql query using the DocumentClient, you could use the scan operation. If you want to avoid using scan, you could perform two separate query operations and join the results in your application code.
I was wondering what is the best practice for fetching collections by complex queries in Doctrine 2. I have been using the Cirteria matching functionality, but I want to know if it is usable for large scale databases.
$expr = Criteria::expr();
$criteria = Criteria::create();
$criteria->where(
$expr->andX(
$expr->gte('start', $start),
$expr->lte('end', $end)
)
);
$result = (new ArrayCollection($em->getRepository(Entity::class)->findAll())->matching($criteria);
What is the difference in performace with same filter written in DQL.
$qb = $em->createQueryBuilder();
$qb->select('e')
->from('Entity', 'e')
->where('e.start >= :start')
->andWhere('e.end <= :end')
->setParameters(array('start' => $start, 'end' => $end));
return $qb->getQuery()->getArrayResult();
And a native SQL one.
$rsm = new ResultSetMapping;
$rsm->addEntityResult('Entity', 'e');
$query = $this->_em->createNativeQuery('SELECT * FROM Entity WHERE start >= ? AND end <= ?', $rsm);
$query->setParameter(1, $start);
$query->setParameter(2, $end);
$result = $query->getResult();
I really like the criteria one as it is easy to read and easy to maintain. But is it usable? How does the performace suffer when fetching 1 000, 10 000 or even 100 000 records.
The second question is does the Doctrine Criteria fetches all result and then filters them or does it create the specific query first?
The docs says this, but does it apply to my case?
"Collections have a filtering API that allows to slice parts of data from a collection. If the collection has not been loaded from the database yet, the filtering API can work on the SQL level to make optimized access to large collections."
Doctrine 2 Criteria documentation
The problem is that you are using Criteria to match records after calling findAll() in your Repo. You should use criteria direct over your Repo, not to the array collection.
$criteria = new Criteria();
$criteria->where(Criteria::expr()->gte('legajo', $this->legajoDesde));
$criteria->andWhere(Criteria::expr()->lte('legajo', $this->legajoHasta));
$criteria->andWhere(Criteria::expr()->eq('idOrganizacion', $this->idOrganizacion));
$criteria->orderBy(['legajo' => Criteria::ASC]);
$empleados = $empleadoRepository->matching($criteria);
i have tried select fields with doctrine query buidler.
$queryBuilder = $entityManager->createQueryBuilder();
$queryBuilder->select('au.id, au.firstName')
->from('Api\V1\Entity\AgencyUser', 'au')
->orderBy('au.firstName', 'asc');
$queryBuilder->andWhere('au.agency = :agencyId')
->setParameter('agencyId', $agency->getId());
print_r($queryBuilder->getQuery()->getResult(Query::HYDRATE_OBJECT));exit;
result :
Array
(
[0] => Array
(
[id] => 1
[firstName] => agency
)
)
why this is an array ? i want to hydrated result. any idea ?
You want to use Doctrine's Partial Object Syntax
$queryBuilder->select('partial au.{id, firstName}')
As to why it's not returning an object, this is from the same documentation linked above:
By default when you run a DQL query in Doctrine and select only a subset of the fields for a given entity, you do not receive objects back. Instead, you receive only arrays as a flat rectangular result set, similar to how you would if you were just using SQL directly and joining some data.
What is the efficient way of getting all items for a given set of hashkeys from a single table? Table has both hashkey and rangekey.
To retrieve a single item I am using
items = tbl.items.query(
:hash_value => "HashKey1",
:select => :all).select.map {|i| i.attributes}
One way to retrieve all item is to loop through keys which is a terrible solution
hashkeys = %w(abcd efgh xyz)
hashkeys.each do |key|
items[dn] = tbl.items.query(
:hash_value => key,
:select => :all).select.map {|i| i.attributes}
end
I am using ruby client for Amazon DynamoDB.
I did not use ruby client but in comparison with php client you could use an array as hashkey value like:
hashkeys = %w(abcd efgh xyz)
items = tbl.items.query(
:hash_value => hashkeys,
:select => :all).select.map {|i| i.attributes}
Is there any way to fetch objets from the DB and list them by Category? I have a table Artist which have a Genre and each Genre have a Category, so when I'm choosing Genre at new Artist creation I would like to see them separately by each one Category. I recall that in previous versions you could do something like:
$this->widgetSchema['genre_id'] = new sfWidgetFormChoice(array(
'choices' => array('-- Select --',
'-- Category1 --'=>Doctrine_Core::getTable('Genre')->getGenres('Category1')),
'-- Category2 --'=>Doctrine_Core::getTable('Genre')->getGenres('Category2')),
'multiple' => false, 'expanded' => false
));
And the result was a drop down with 'CategoryX' in bold, followed by respectively genres.
After a deep search all I've found is:
->add('group', 'entity', array(
'class' => 'Vendor\MyBundle\Entity\Group',
'label'=>'Group:',
'query_builder' => function(EntityRepository $er) {
return $er->createQueryBuilder('g')
->... // whatever you want to do
}
))
I think that is what I need, but have no idea on how to adapt it to what I'm looking for.
Many thanks.