Annotations in GeoDjango on many-to-many tables - django

short
I am trying to query a model Foo which has a many-to-many relationship to Address where addresses will be within a specified distance from a given point and sort results by ascending distance. Seems like annotation would be able to do this however I can't figure out how to do that in GeoDjango since it does not support geo annotations.
longer
Here is the basic model structure I have:
# app name is bar
from django.contrib.gis.db import models
class Location(models.Model):
latlon = Models.PointFields(spatial_index=True)
# other fields ommitted
objects = models.GeoManager()
class Address(models.Model):
latlon = models.PointField(spatial_index=True)
# other fields omitted
objects = models.GeoManager()
class Foo(models.Model):
addresses = models.ManyToManyField(Address)
# other fields omitted
objects = models.GeoManager()
Using the above models I am able to construct a query which selects all Foo objects which have addresses within a specific distance from a specific point. For example:
from django.contrib.gis.geos import Point
from django.contrib.gis.measure import Distance
new_york = Point(-73.98497, 40.75813) # == Location.latlon
Foo.objects.filter(addresses__latlon__distance_lte=(new_york, Distance(mi=20)))
That generates a query something like:
SELECT
"bar_foo"."id",
...
FROM "bar_foo"
INNER JOIN "bar_foo_address"
ON ("bar_foo"."id" = "bar_foo_address"."foo_id")
INNER JOIN "bar_address"
ON ("bar_foo_address"."address_id" = "bar_address"."id")
WHERE (ST_distance_sphere("bar_address"."latlon",ST_GeomFromEWKB(
'\x0101000020e6100000aaf1d24d628052c096218e75715b4440' :: BYTEA)) <= 32186.88)
That works very well except I run into trouble if I want to sort the all foos by their distance from the given point. I tried something like:
(Foo
.objects
.filter(addresses__latlon__distance_lte=(new_york, Distance(mi=20)))
.distance(Location.latlon)
.order_by('distance'))
# produces
TypeError: ST_Distance output only available on GeometryFields.
When I read some source code I tried to modify the query and yet still getting errors:
(Foo
.objects
.filter(addresses__latlon__distance_lte=(new_york, Distance(mi=20)))
.distance(Location.latlon)
.order_by('distance', field_name='addresses_latlon'))
# produces
ValueError: <django.contrib.gis.db.models.fields.PointField: latlon> not in self.query.related_select_cols
I guess this is related to a fact that Address and Foo have many-to-many relationship. Unfortunately regular annotations are not supported in GeoDjango so I cant do something like:
# hypothetical syntax
(Foo
.objects
.annotate(distance=DistanceAnnotation('addresses__latlon', new_york, unit='mi'))
.filter(distance__lte=20)
.order_by('distance'))
# which would generate
SELECT
"bar_foo"."id",
(ST_distance_sphere("bar_address"."latlon",ST_GeomFromEWKB(
'\x0101000020e6100000aaf1d24d628052c096218e75715b4440' :: BYTEA)) as distance,
...
FROM "bar_foo"
INNER JOIN "bar_foo_address"
ON ("bar_foo"."id" = "bar_foo_address"."foo_id")
INNER JOIN "bar_address"
ON ("bar_foo_address"."address_id" = "bar_address"."id")
WHERE distance <= 32186.88)
ORDER BY distance ASC
So the question is how can I do do regular annotation using existing API? Or maybe some other way I can accomplish the desired result?

Distance annotation was implemented in 1.9
https://docs.djangoproject.com/en/dev/ref/contrib/gis/functions/#distance
Distance
class Distance(expr1, expr2, spheroid=None, **extra) Availability:
MySQL, PostGIS, Oracle, SpatiaLite
Accepts two geographic fields or expressions and returns the distance
between them, as a Distance object. On MySQL, a raw float value is
returned when the coordinates are geodetic.
(...)
In the following example, the distance from the city of Hobart to
every other PointField in the AustraliaCity queryset is calculated:
>>> from django.contrib.gis.db.models.functions import Distance
>>> pnt = AustraliaCity.objects.get(name='Hobart').point
>>> for city in AustraliaCity.objects.annotate(distance=Distance('point', pnt)):
... print(city.name, city.distance)
Wollongong 990071.220408 m
Shellharbour 972804.613941 m
Thirroul 1002334.36351 m
...

Related

Django foreign keys in extra() expression

I'm trying to use the Django extra() method to filter all the objects in a certain radius, just like in this answer: http://stackoverflow.com/questions/19703975/django-sort-by-distance/26219292 but I'm having some problems with the 'gcd' expression as I have to reach the latitude and longitude through two foreign key relationships, instead of using direct model fields.
In particular, I have one Experience class:
class Experience(models.Model):
starting_place_geolocation = models.ForeignKey(GooglePlaceMixin, on_delete=models.CASCADE,
related_name='experience_starting')
visiting_place_geolocation = models.ForeignKey(GooglePlaceMixin, on_delete=models.CASCADE,
related_name='experience_visiting')
with two foreign keys to the same GooglePlaceMixin class:
class GooglePlaceMixin(models.Model):
latitude = models.DecimalField(max_digits=20, decimal_places=15)
longitude = models.DecimalField(max_digits=20, decimal_places=15)
...
Here is my code to filter the Experience objects by starting place location:
def search_by_proximity(self, experiences, latitude, longitude, proximity):
gcd = """
6371 * acos(
cos(radians(%s)) * cos(radians(starting_place_geolocation__latitude))
* cos(radians(starting_place_geolocation__longitude) - radians(%s)) +
sin(radians(%s)) * sin(radians(starting_place_geolocation__latitude))
)
"""
gcd_lt = "{} < %s".format(gcd)
return experiences \
.extra(select={'distance': gcd},
select_params=[latitude, longitude, latitude],
where=[gcd_lt],
params=[latitude, longitude, latitude, proximity],
order_by=['distance'])
but when I try to call the foreign key object "strarting_place_geolocation__latitude" it returns this error:
column "starting_place_geolocation__latitude" does not exist
What should I do to reach the foreign key value? Thank you in advance
When you are using extra (which should be avoided, as stated in documentation), you are actually writing raw SQL. As you probably know, to get value from ForeignKey you have to perform JOIN. When using Django ORM, it translates that fancy double underscores to correct JOIN clause. But the SQL can't. And you also cannot add JOIN manually. The correct way here is to stick with ORM and define some custom database functions for sin, cos, radians and so on. That's pretty easy.
class Sin(Func):
function = 'SIN'
Then use it like this:
qs = experiences.annotate(distance=Cos(Radians(F('starting_place_geolocation__latitude') )) * ( some other expressions))
Note the fancy double underscores comes back again and works as expected
You have got the idea.
Here is a full collection of mine if you like copy pasting from SO)
https://gist.github.com/tatarinov1997/3af95331ef94c6d93227ce49af2211eb
P. S. You can also face the set output_field error. Then you have to wrap your whole distance expression into ExpressionWrapper and provide it an output_field=models.DecimalField() argument.

How to add distance from point as an annotation in GeoDjango

I have a Geographic Model with a single PointField, I'm looking to add an annotation for the distance of each model from a given point, which I can later filter on and do additional jiggery pokery.
There's the obvious queryset.distance(to_point) function, but this doesn't actually annotate the queryset, it just adds a distance attribute to each model in the queryset, meaning I can't then apply .filter(distance__lte=some_distance) to it later on.
I'm also aware of filtering by the field and distance itself like so:
queryset.filter(point__distance_lte=(to_point, D(mi=radius)))
but since I will want to do multiple filters (to get counts of models within different distance ranges), I don't really want to make the DB calculate the distance from the given point every time, since that could be expensive.
Any ideas? Specifically, is there a way to add this as a regular annotation rather than an inserted attribute of each model?
I couldn't find any baked in way of doing this, so in the end I just created my own Aggregation class:
This only works with post_gis, but making one for another geo db shouldn't be too tricky.
from django.db.models import Aggregate, FloatField
from django.db.models.sql.aggregates import Aggregate as SQLAggregate
class Dist(Aggregate):
def add_to_query(self, query, alias, col, source, is_summary):
source = FloatField()
aggregate = SQLDist(
col, source=source, is_summary=is_summary, **self.extra)
query.aggregates[alias] = aggregate
class SQLDist(SQLAggregate):
sql_function = 'ST_Distance_Sphere'
sql_template = "%(function)s(ST_GeomFromText('%(point)s'), %(field)s)"
This can be used as follows:
queryset.annotate(distance=Dist('longlat', point="POINT(1.022 -42.029)"))
Anyone knows a better way of doing this, please let me know (or tell me why mine is stupid)
One of the modern approaches is the set "output_field" arg to avoid «Improper geometry input type: ». Withour output_field django trying to convert ST_Distance_Sphere float result to GEOField and can not.
queryset = self.objects.annotate(
distance=Func(
Func(
F('addresses__location'),
Func(
Value('POINT(1.022 -42.029)'),
function='ST_GeomFromText'
),
function='ST_Distance_Sphere',
output_field=models.FloatField()
),
function='round'
)
)
Doing it like this this works for me, ie I can apply a filter on an annotation.
Broken up for readability.
from models import Address
from django.contrib.gis.measure import D
from django.contrib.gis.db.models.functions import Distance
intMiles = 200
destPoint = Point(5, 23)
queryset0 = Address.objects.all().order_by('-postcode')
queryset1 = queryset0.annotate(distance=Distance('myPointField' , destPoint ))
queryset2 = queryset1.filter(distance__lte=D(mi=intMiles))
Hope it helps somebody :)
You can use GeoQuerySet.distance
cities = City.objects.distance(reference_pnt)
for city in cities:
print city.distance()
Link: GeoDjango distance documentaion
Edit: Adding distance attribute along with distance filter queries
usr_pnt = fromstr('POINT(-92.69 19.20)', srid=4326)
City.objects.filter(point__distance_lte=(usr_pnt, D(km=700))).distance(usr_pnt).order_by('distance')
Supported distance lookups
distance_lt
distance_lte
distance_gt
distance_gte
dwithin
A way to annotate & sort w/out GeoDjango. This model contains a foreignkey to a Coordinates record which contains lat and lng properties.
def get_nearby_coords(lat, lng, max_distance=10):
"""
Return objects sorted by distance to specified coordinates
which distance is less than max_distance given in kilometers
"""
# Great circle distance formula
R = 6371
qs = Precinct.objects.all().annotate(
distance=Value(R)*Func(
Func(
F("coordinates__lat")*Value(math.sin(math.pi/180)),
function="sin",
output_field=models.FloatField()
) * Value(
math.sin(lat*math.pi/180)
) + Func(
F("coordinates__lat")* Value(math.pi/180),
function="cos",
output_field=models.FloatField()
) * Value(
math.cos(lat*math.pi/180)
) * Func(
Value(lng*math.pi/180) - F("coordinates__lng") * Value(math.pi/180),
function="cos",
output_field=models.FloatField()
),
function="acos"
)
).order_by("distance")
if max_distance is not None:
qs = qs.filter(distance__lt=max_distance)
return qs

Django Query - where start = end

in models i have start and end date.
How to get all element where start and end date are diffrents.
>>> Entry.objects.exclude(start = end)
>>> NameError: name 'end' is not defined
I have no idea please help.
https://docs.djangoproject.com/en/dev/topics/db/queries/#filters-can-reference-fields-on-the-model
In the examples given so far, we have constructed filters that compare the value of a model field with a constant. But what if you want to compare the value of a model field with another field on the same model?
Django provides the F() object to allow such comparisons. Instances of F() act as a reference to a model field within a query. These references can then be used in query filters to compare the values of two different fields on the same model instance.
For your case, the following should work.
from django.db.models import F
Entry.objects.exclude(start=F('end'))

Database error using distance() in GeoDjango

Given the following (simplified) models:
from django.contrib.gis.db import models
class City(models.Model):
center = models.PointField(spatial_index=True, null=True)
objects = models.GeoManager()
class Place(models.Model):
city = models.ForeignKey(City, null=True)
lat = models.FloatField(null=True)
lng = models.FloatField(null=True)
objects = models.GeoManager()
Forgetting for the moment that the lat/lng in Place should be moved to a PointField(), I am trying to look through all of the Places and find the closest city. Currently, I am doing:
from django.contrib.gis.geos import Point
places = Property.objects.filter(lat__isnull=False, lng__isnull=False)
for place in places:
point = Point(place.lng, place.lat, srid=4326) # setting srid just to be safe
closest_city = City.objects.distance(point).order_by('distance')[0]
This results in the following error:
DatabaseError: geometry_distance_spheroid: Operation on two GEOMETRIES with different SRIDs
Assuming that the SRIDs were not defaulting to 4326, I included srid=4326 in the above code and verified that all of the cities have City.center has an SRID of 4326:
In [6]: [c['center'].srid for c in City.objects.all().values('center')]
Out[6]: [4326, 4326, 4326, ...]
Any ideas on what could be causing this?
UPDATE:
There seems to be something in how the sql query is created that causes a problem. After the error is thrown, looking at the sql shows:
In [9]: from django.db import connection
In [10]: print connection.queries[-1]['sql']
SELECT (ST_distance_sphere("model_city"."center",
ST_GeomFromEWKB(E'\\001\\001...\\267C#'::bytea))) AS "distance",
"model_city"."id", "model_city"."name", "listing_city"."center"
FROM "model_city" ORDER BY "model_city"."name" ASC LIMIT 21
It looks like django is turning the point argument of distance() into Extended Well-Known Binary. If I then change ST_GeomFromEWKB to ST_GeomFromText everything works fine. Example:
# SELECT (ST_distance_sphere("listing_city"."center",
ST_GeomFromText('POINT(-118 38)',4326))) AS "distance",
"model_city"."name", "model_city"."center" FROM "model_city"
ORDER BY "listing_city"."name" ASC LIMIT 5;
distance | name | center
------------------+-------------+----------------------------------------------------
3124059.73265751 | Akron | 0101000020E6100000795DBF60376154C01CB62DCA6C8A4440
3742978.5514446 | Albany | 0101000020E6100000130CE71A667052C038876BB587534540
1063596.35270877 | Albuquerque | 0101000020E6100000CC0D863AACA95AC036E7E099D08A4140
I can't find anything in the documentation that speaks to how GeoQuerySet.distance() translates into SQL. I can certainly use raw SQL in the query to get things to work, but would prefer to keep everything nicely in the Django framework.
i think this error : "Operation on two GEOMETRIES with different SRIDs"
"geometry_columns" table on your database is set different srid between your table name to process
***** you should change it yourself

GeoDjango distance query for a ForeignKey Relationship

I have the following models (simplified)
from django.contrib.gis.db import models as geomodels
modelB (geomodels.Model):
objects = geomodels.GeoManager()
modelA (geomodels.Model):
point = geomodels.PointField(unique=True)
mb = models.ForeignKey(modelB,related_name='modela')
objects = geomodels.GeoManager()
I am trying to find all modelB objects and sort them by distance from a given location (where distance is defined as distance between a given location and the point object of associated modelA). When I try to run the query
modelB.objects.distance((loc, field_name='modela__point')
I get an error saying
TypeError: ST_Distance output only available on GeometryFields.
Note that loc is a Point object.However, when I run the query
modelB.objects.filter(modela__point__distance_lte = (loc, 1000))
this query works without error and as expected.
Any idea what the mistake could be? I am using django 1.2.4, PostGis 1.5.2, PostGres 8.4.
Thanks.
For the first one, you will need to change it to:
modelB.objects.all().distance(loc, field_name='modela__point')
if you want to see all modelB objects.
The ".distance" is used to calculate and add a distance field to each resulting row of the QuerySet (or GeoQuerySet).