I have a GCP datastore query in my code:
cls.query(
ndb.OR(cls.is_miscategorized == True,
cls.is_category_recommendation_mismatched == True,
cls.is_class_recommendation_mismatched == True),
cls.account_type == account_type,
cls.transaction_date >= start_date,
cls.transaction_date <= end_date,
)
For which I have created this index:
- kind: Table name
properties:
- name: is_miscategorized
- name: is_category_recommendation_mismatched
- name: is_class_recommendation_mismatched
- name: account_type
- name: transaction_date
direction: desc
As per my understanding of indexes, I have included all the expected fields in right order here. But I still get the error that Indexes are missing
no matching index found. recommended index is:
- kind: Table name
properties:
- name: account_type
- name: is_class_recommendation_mismatched
- name: transaction_date
direction: desc
- name: create_time
I have added the composite index suggested in the error message too. But I still get the same error, weirdly with the same suggested index in the error message.
Now I have two questions:
The indexes suggested by GCP seem wrong. They have fields missing. So how does GCP suggest indexes? Are they worth relying upon?
What am I doing wrong with indexes here? Why am I getting the error with my original index?
NDB's implementation of OR does a separate query for each OR clause. So you aren't doing one query to the backend, but 3. E.g.
class.query(cls.is_miscategorized == True,
cls.transaction_date >= start_date,
cls.transaction_date <= end_date)
This query requires a composite index, and the other 2 queries require their own composite index.
Your recommended index has account_type & create_time, but that is not in your example. Did you leave those out?
Related
I am trying to write a query where I can sort by 2 columns see below
GCP Query Window Snapshot
Query only sorts on created_datetime not on both created_datetime and price
Below are the indexes I have created
- kind: indicator_values
properties:
- name: currency
- name: created_datetime
direction: desc
- name: price
- kind: indicator_values
properties:
- name: currency
- name: created_datetime
direction: desc
- name: price
direction: desc
I am not able to figure out what kind of index I have to create. Can I get some help?
You don't have to manually create the indexes. Run your application in a dev (local) environment using datastore emulator and your indexes will automatically be created for you. You can then upload the index when you deploy your App to production.
See documentation
In addition, the emulator can help you generate indexes for your production Datastore instance and delete unneeded indexes.
I have a large schema with ~70 tables and many of them connected to each other(194 #connection directives) like this:
type table1 #model {
id:ID!
name: String!
...
table2: table2 #connection
}
type table2 #model {
id:ID!
....
}
This works fine. Now my data amount is steadily growing and I need to be able to query for results and sort them.
I've read several articles and found one giving me the advice to create a #key directive to generate a GSI with 2 fields so I can say "Filter the results according to my filter property, sort them by the field "name" and return the first 10 entries, the rest accessible via nextToken parameter"
So I tried to add a GSI like this:
type table1 #model
#key(name: "byName", fields:["id","name"], queryField:"idByName"){
id:ID!
name: String!
...
table2: table2 #connection
}
running
amplify push --minify
I receive the error
Attempting to add a local secondary index to the table1Table table in the table1 stack. Local secondary indexes must be created when the table is created.
An error occured during the push operation: Attempting to add a local secondary index to the table1Table table in the table1 stack.
Local secondary indexes must be created when the table is created.
Why does it create a LSI instead of a GSI? Are there any ways to add #key directives to the tables after they have been created and filled? There are so many datasets from different tables linked with each other so just setting up a new schema would take ages.
Billingmode is PAY_PER_REQUEST if this has some impact.
Any ideas how to proceed?
Thanks in advance!
Regards Christian
If you are using new environment, delete folder #current-cloud-backend first.
Then amplify init created the folder again but alas, with only one file in it amplify-meta.json.
Kind name: invoice_header
Name/ID :id=4829628648652800
amount :2900
booking_ids :1,2
created_at :2018-09-04 10:20:30
discount_amount :23
due_amount :9999
indicator :PO
invoice_date :2018-09-04
invoice_id :451
issued_to :P
location_id :12
net_payable_amount :999
order_or_po_id :533
paid_amount :555
partner_id :400
payment_mode_promotion_amount :0
status :NP
tax_amount :34
updated_at :
I tried to fetch the data using below GQL Query from the above kind but I got the following error.
GQL query error: Your Datastore does not have the composite index
(developer- supplied) required for this query.
select invoice_date from invoice_header where location_id ='12' and invoice_date >= '2018-09-01' and invoice_date <= '2018-09-05'
Have you added an index file to your application?
As it is stated in the documentation [1], every Cloud Datastore query made by an application needs a corresponding index. Indexes for simple queries, such as queries over a single property, are created automatically. Indexes for complex queries must be defined in a configuration file named index.yaml. This file is uploaded with the application to create indexes in Cloud Datastore.
Composite indexes support complex queries and are defined in an index configuration file (index.yaml).
In your case you need to add a composite index that includes location_id and invoice_date properties. This is an example of the index.yaml:
indexes:
- kind: invoice_header
properties:
- name: location_id
direction: asc
- name: invoice_date
direction: asc
When you are done modifying your index configuration file, run the "gcloud datastore create-indexes" command to place the indexes into service.
In [2] you will find a broader explanation about the indexes.
Cheers.
I have three entities: HandsetSubscription, Handset and Subscription.
The yaml of HandsetSubscription is:
App\SoBundle\Entity\HandsetSubscription:
type: entity
table: handset_subscription
manyToOne:
handset:
targetEntity: Handset
subscription:
targetEntity: Subscription
id:
id:
type: integer
generator: { strategy: AUTO }
options: { unsigned: true }
fields:
amount:
type: integer
nullable: false
options: { default: 0, unsigned: true }
discount:
type: integer
nullable: false
options: { default: 0, unsigned: true }
The query:
SELECT hs,s,h
FROM \App\SoBundle\Entity\HandsetSubscription hs
JOIN \App\SoBundle\Entity\Subscription s with s.id = hs.subscription
AND s.mins = 150
AND s.mb = 250
AND s.sms = 150
JOIN \App\SoBundle\Entity\Handset h with h.id = hs.handset
These are the class names of the entries retrieved:
App\SoBundle\Entity\HandsetSubscription
Proxies\__CG__\App\SoBundle\Entity\Subscription
Proxies\__CG__\App\SoBundle\Entity\Handset
App\SoBundle\Entity\HandsetSubscription
Proxies\__CG__\App\SoBundle\Entity\Handset
App\SoBundle\Entity\HandsetSubscription
Proxies\__CG__\App\SoBundle\Entity\Handset
…
I would expect to get only HandsetSubscription entities back. Why am I getting proxies of Subscription and Handset too?
By adding fetch eager to the handset and subscription mappings and removing handset and subscription from the SELECT statement in the query I would get only HandsetSubscription but I would like to do this through fetch joins, as stated in the manual (http://doctrine-orm.readthedocs.org/en/latest/reference/dql-doctrine-query-language.html#joins).
UPDATE
Quote from the link posted above:
Fetch join of the address:
<?php
$query = $em->createQuery("SELECT u, a FROM User u JOIN u.address a WHERE a.city = 'Berlin'");
$users = $query->getResult();
When Doctrine hydrates a query with fetch-join it returns the class in the FROM clause on the root level of the result array. In the previous example an array of User instances is returned and the address of each user is fetched and hydrated into the User#address variable. If you access the address Doctrine does not need to lazy load the association with another query.
Big thanks goes to veonik from the #doctrine irc channel for solving this.
Instead of joining with the fully qualified names of the entities you should join with the association. So the query becomes:
SELECT hs,s,h
FROM \App\SoBundle\Entity\HandsetSubscription hs
JOIN hs.subscription s with s.id = hs.subscription
AND s.mins = 150
AND s.mb = 250
AND s.sms = 150
JOIN hs.handset h with h.id = hs.handset
I have a Doctrine model (Assignment), which has a many-to-one relationship with another model (Region). Assignments are owned by users (with each user having only one assignment per region at a time), and I am trying to use indexBy to have the user's array of assignments be keyed by the ID of the assignment's region. However, I only get standard 0..n numeric keys.
When I try to run a DQL query like SELECT am, reg, user FROM Assignment am INDEX BY [...] JOIN am.region reg JOIN am.user user WHERE user.id = ?1, none of these values for INDEX BY work:
region (Error: Invalid PathExpression. Must be a StateFieldPathExpression.)
region_id (Error: Class ...\Assignment has no field or association named region_id)
region.id (Error: Expected end of string, got '.')
Is this possible? If not, then what would be a convenient way to access a User's assignment on a region without indexBy?
I was dealing with the same problem today. Fortunately I've found the solution : )
First of all, you have to declare additional column in ORM mapping:
Abdulklarapl\My\EntityA:
type: entity
table: entityA
manyToOne:
entityB:
targetEntity: EntityB
joinColumn:
name: label_id
referencedColumnName: id
id:
id:
type: integer
id: true
generator:
strategy: AUTO
fields:
value:
type: text
entityB_id:
type: integer
lifecycleCallbacks: { }
notice that I've declared entityB_id as a field + I've configured manyToOne relation by adding a joinColumn clause
so now you can use entityB_id as scalar value
$doctrine->getEntityManager()->createQueryBuilder()
->select('c')
->from('AbdulklaraplMyBundle:EntityA', 'c', 'c.entityB_id')
->getQuery()->getResult();
it will return assoc array
[
c.entityB_id: {
id: "",
value: ""
entityB_id: ""
}
]
you can also use AbstractQuery::HYDRATE_ARRAY as a argument for getResult() - it will return assoc array with array instead the objects
If you need to INDEX BY a foreign key e.g. "region_id" it is possible in your mapping:
/**
* #ORM\OneToMany(targetEntity="Region", mappedBy="user", indexBy="region_id")
*/
protected $regions;
The feature was added here.
Unfortunately it does not seem to be documented that you have to use the name of the column of the foreign key itself.
Working with Indexed Associations
Import the Query class (optional):
use \Doctrine\ORM\Query;
Create the query:
$query = $this->data->em->createQuery('
SELECT a
FROM Assignment a
INDEX BY a.reg //to set array custom key
WHERE a.user = :user');
$query->setParameter('user', 3); //user with id 3
//set the hidration mode in order to work with read-only arrays
$assignments = $query->getResult(Query::HYDRATE_ARRAY);