Postgres Recursive query + group by + join in Django - django

My Requirement is to write a sql query to get the sub-region wise (fault)events count that occurred for the managedobjects. My database is postgres 8.4. Let me explain using the table structure.
My tables in django:
Managedobject:
class Managedobject(models.Model):
name = models.CharField(max_length=200, unique=True)
iscontainer = models.BooleanField(default=False,)
parentkey = models.ForeignKey('self', null=True)
Event Table:
class Event(models.Model):
Name = models.CharField(verbose_name=_('Name'))
foid = models.ForeignKey(Managedobject)
Managedobject Records:
NOC
Chennai
MO_1
MO_2
MO_3
Mumbai
MO_4
MO_5
MO_6
Delhi
Bangalore
IP
Calcutta
Cochin
Events Records:
event1 MO_1
event2 MO_2
event3 MO_3
event4 MO_5
event5 MO_6
Now I need to get the events count for all the sub-regions. For example,
for NOC region:
Chennai - 3
Mumbai - 2
Delhi - 0
Bangalore - 0
So far I am able to get the result in two different queries.
Get the subregions.
select id from managedobject where iscontainer = True and parentkey = 3489
For each of the region (using for loop), get the count as follows:
SELECT count(*)
from event ev
WHERE ev.foid
IN (
WITH RECURSIVE q AS (
SELECT h
FROM managedobject h
WHERE parentkey = 3489
UNION ALL
SELECT hi
FROM q
JOIN managedobject hi
ON hi.parentkey = (q.h).id
)
SELECT (q.h).id FROM q
)
Please help to combine the queries to make it a single query and for getting the top 5 regions. Since the query is difficult in django, I am going for a raw sql query.

I got the query:
WITH RECURSIVE q AS (
SELECT h,
1 AS level,
id AS ckey,
displayname as dname
FROM managedobject h
WHERE parentkey = 3489
and logicalnode=True
UNION ALL
SELECT hi,
q.level + 1 AS level,
ckey,
dname
FROM q
JOIN managedobject hi ON hi.parentkey = (q.h).id
)
SELECT count(ckey) as ccount,
ckey,
dname
FROM q
JOIN event as ev on ev.foid_id = (q.h).id
GROUP BY ckey, dname
ORDER BY ccount DESC
LIMIT 5

Related

Trigger to insert new quote number in this format

I am very new at triggers - actually this will be my first one.
What I am trying to achieve is the following:
When I create a new Sales Opportunity, the Quote no field needs to spit out a number starting from this Q1500/09/2017 so Q is the prefix, 1500 is the first quote number, 09 the month and 2017 the year.
Can someone assist me in getting this right - do i need a SP and a table in order for my trigger to work correctly?
Quote_No is a table field so I'm guessing it should look something like this:
CREATE TABLE [dbo].[Quote_Pref_No](
[Yr] [varchar](4) NULL,
[Mth] [varchar](2) NULL,
[Quote_Pref] [varchar](10) NULL,
[Quote_No] [int] NULL
) ON [PRIMARY]
Then the SP:
Create Proc [dbo].[New_Quote_Num] #ClientID varchar(24),#ContactNum int, #Year varchar(4), #mth varchar(2)
As
Begin
Declare #Pref varchar(5),
#Num int,
#QuoteNo varchar(255)
Select #QuoteNo = B.Description
From dbo.AMGR_User_Fields_Tbl A inner join dbo.AMGR_User_Field_Defs_Tbl B
on A.Type_Id = B.Type_Id and A.Code_Id = B.Code_Id and A.Type_Id = 117
Where A.Client_Id = #ClientId and A.Contact_Number = #ContactNum
Select #Pref = Quote_Pref, #Num = Quote_No + 1
From dbo.Quote_Pref_No
Where Yr = #Year and Mth = #Mth
Set #QuoteNum = 'Q/'+Right(#Year,2)+'/'+Right(#mth,2)+'/'+#Pref+case when Len(#Num) <= 3 then Right('000000'+Convert(Varchar(55),#Num),3) else CONVERT(Varchar(55),#Num) end
Insert into dbo.O_QuoteNo(Client_Id,Contact_Number,Type_Id,Code_Id,O_QuoteNo)
Values(#ClientId,#ContactNum,13,0,#QuoteNum)
Update dbo.Quote_Pref_No
Set Quote_No = #Num
Where Yr = #Year and mth = #mth
End
Grant execute on dbo.New_Quote_Num to CRMGroup
GO
I just dont know how to generate the month and year together.
Trigger code:
Create Trigger [dbo].[QuoteNew]
on [dbo].[AMGR_User_Fields_Tbl] after insert
As
Declare #ClientId varchar(24),
#ContactNum int,
#TypeId int,
#Year varchar(4),
#quoteYear varchar(4),
#mth varchar(2)
Select #ClientId = Client_Id,
#ContactNum = Contact_Number,
#TypeId = TYPE_ID,
#Year = CONVERT(varchar(4),Datepart(yyyy,DateCol)) from inserted
#mth = convert(varchar(2), datepart(mm,Datecol)) from inserted
Select #quoteYear = Yr from dbo.quote_Pref_No where Yr = #Year
select #mth = mth from dbo.quote_Pref_No where mth = #mth
If #TypeID = 117 and #QuoteYear is null or #quoteYear = ''
Begin
Execute dbo.New_Year_Certs #Year
Execute dbo.New_quote_Num #ClientId,#ContactNum,#Year
End
If #TypeID = 117 and #quoteYear = #Year
Begin
Execute dbo.New_Cert_Num #ClientId,#ContactNum,#Year
End
GO
Do I need to create a separate SP for month or can I do both month and year in one SP?

Select nth to nth row while table still have values unselected with python and pyodbc

I have a table with 10,000 rows and I want to select the first 1000 rows and then select again and this time, the next set of rows, which is 1001-2001.
I am using the BETWEEN clause in order to select the range of values. I can also increment the values. Here is my code:
count = cursor.execute("select count(*) from casa4").fetchone()[0]
ctr = 1
ctr1 = 1000
str1 = ''
while ctr1 <= count:
sql = "SELECT AccountNo FROM ( \
SELECT AccountNo, ROW_NUMBER() OVER (ORDER BY Accountno) rownum \
FROM casa4 ) seq \
WHERE seq.rownum BETWEEN " + str(ctr) + " AND " + str(ctr1) + ""
ctr = ctr1 + 1
ctr1 = ctr1 + 1000
cursor.execute(sql)
sleep(2) #interval in printing of the rows.
for row in cursor:
str1 = str1 + '|'.join(map(str,row)) + '\n'
print "Records:" + str1 #var in storing the fetched rows from database.
print sql #prints the sql statement(str) and I can see that the var, ctr and ctr1 have incremented correctly. The way I want it.
What I want to achieve is using a messaging queue, RabbitMQ, I will send this rows to another database and I want to speed up the process. Selecting all and sending it to the queue returns an error.
The output of the code is that it returns 1-1000 rows correctly on the 1st but, on the 2nd loop, instead of 1001-2001 rows, it returns 1-2001 rows, 1-3001 and so on.. It always starts on 1.
I was able to recreate your issue with both pyodbc and pypyodbc. I also tried using
WITH seq (AccountNo, rownum) AS
(
SELECT AccountNo, ROW_NUMBER() OVER (ORDER BY Accountno) rownum
FROM casa4
)
SELECT AccountNo FROM seq
WHERE rownum BETWEEN 11 AND 20
When I run that in SSMS I just get rows 11 through 20, but when I run it from Python I get all the rows (starting from 1).
The following code does work using pyodbc. It uses a temporary table named #numbered, and might be helpful in your situation since your process looks like it would do all of its work using the same database connection:
import pyodbc
cnxn = pyodbc.connect("DSN=myDb_SQLEXPRESS")
crsr = cnxn.cursor()
sql = """\
CREATE TABLE #numbered (rownum INT PRIMARY KEY, AccountNo VARCHAR(10))
"""
crsr.execute(sql)
cnxn.commit()
sql = """\
INSERT INTO #numbered (rownum, AccountNo)
SELECT
ROW_NUMBER() OVER (ORDER BY Accountno) AS rownum,
AccountNo
FROM casa4
"""
crsr.execute(sql)
cnxn.commit()
sql = "SELECT AccountNo FROM #numbered WHERE rownum BETWEEN ? AND ? ORDER BY rownum"
batchsize = 1000
ctr = 1
while True:
crsr.execute(sql, [ctr, ctr + batchsize - 1])
rows = crsr.fetchall()
if len(rows) == 0:
break
print("-----")
for row in rows:
print(row)
ctr += batchsize
cnxn.close()

Raw query must include the primary key

I got a raw SQL statement in my views.py
Message.objects.raw('''
SELECT s1.ID, s1.CHARACTER_ID, MAX(s1.MESSAGE) MESSAGE, MAX(s1.c) occurrences
FROM
(SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s1
LEFT JOIN
(SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s2
ON s1.CHARACTER_ID=s2.CHARACTER_ID
AND s1.c < s2.c
WHERE s2.c IS NULL
GROUP BY CHARACTER_ID
ORDER BY occurrences DESC''', [days, days])
The result of this SQL statement (tested on database directly) is:
ID | CHARACTER_ID | MESSAGE | OCCURENCES
----+--------------+---------+--------------
148 | 10 | test | 133
But all I got is a InvalidQuery Exception with the information Raw query must include the primary key
Then I double checked the docs and read:
There is only one field that you can’t leave out - the primary key
field....An InvalidQuery exception will be raised if you forget to include the primary key.
As you can see I got the requested primary key added in my statement. What's wrong?
class Message(models.Model):
character = models.ForeignKey('Character')
message = models.TextField()
location = models.ForeignKey('Location')
ts = models.DateTimeField()
class Meta:
pass
def __unicode__(self):
return u'%s: %s...' % (self.character, self.message[0:20])
Include 1 as id to your query
Message.objects.raw('''
SELECT 1 as id , s1.ID, s1.CHARACTER_ID, MAX(s1.MESSAGE) MESSAGE, MAX(s1.c) occurrences
FROM
(SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s1
LEFT JOIN
(SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s2
ON s1.CHARACTER_ID=s2.CHARACTER_ID
AND s1.c < s2.c
WHERE s2.c IS NULL
GROUP BY CHARACTER_ID
ORDER BY occurrences DESC''', [days, days])
I reproduced the same problem using Python 2.7.5, Django 1.5.1 and Mysql 5.5.
I've saved the result of the raw call to the results variable, so I can check what columns it contains:
>>> results.columns
['ID', 'CHARACTER_ID', 'MESSAGE', 'occurrences']
ID is in uppercase, so in your query I changed s1.ID to s1.id and it works:
>>> results = Message.objects.raw('''
... SELECT s1.id, s1.CHARACTER_ID, MAX(s1.MESSAGE) MESSAGE, MAX(s1.c) occurrences
... FROM
... (SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
... FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s1
... LEFT JOIN
... (SELECT ID, CHARACTER_ID, MESSAGE, COUNT(*) c
... FROM tbl_message WHERE ts > DATE_SUB(NOW(), INTERVAL %s DAY) GROUP BY CHARACTER_ID,MESSAGE) s2
... ON s1.CHARACTER_ID=s2.CHARACTER_ID
... AND s1.c < s2.c
... WHERE s2.c IS NULL
... GROUP BY CHARACTER_ID
... ORDER BY occurrences DESC''', [days, days])
>>> results.columns
['id', 'CHARACTER_ID', 'MESSAGE', 'occurrences']
>>> results[0]
<Message_Deferred_character_id_location_id_message_ts: Character object: hello...>
Make Sure the primary key is part of the select statement.
Example:
This will not work:
`Model.objects.raw("Select Min(id), rider_id from Table_Name group by rider_id")`
But this will work:
`Model.objects.raw("Select id, Min(id), rider_id from Table_Name group by rider_id")`
For those also stuck with this problem, perhaps like me, wondering why Django needs a pk, when you don’t have a pk for the query (eg you want multiple rows) – Django just needs an id field returned, the pk does not need to be part of a where clause. ie:
select * from table where foo = 'bar';
or
select id, description from table where foo = 'bar';
Both of these work, if there is a field id in the table. But this throws the error described by Thomas Schwärzl, because no id field is returned:
select description from table where foo = 'bar';

Select the record of a given id with the highest timestamp using DAO

How do I do the following using DAO on a recordset
SELECT TOP 1 * FROM foo WHERE id = 10 ORDER BY timestamp DESC
Using SetCurrentIndex you can only use one index it seems otherwise using id and timestamp and selecting the first one would work.
I am by no means sure of what you want.
Dim rs As DAO.Recordset
Dim db As Database
Set db = CurrentDB
sSQL = "SELECT TOP 1 * FROM foo WHERE id = 10 ORDER BY timestamp DESC"
Set rs = db.OpenRecordset(sSQL)
Find does not work with all recordsets. This will work:
Set rs = CurrentDb.OpenRecordset("select * from table1")
rs.FindFirst "akey=1 and atext='b'"
If Not rs.EOF Then Debug.Print rs!AKey
This will not:
Set rs = CurrentDb.OpenRecordset("table1")
rs.FindFirst "akey=1 and atext='b'"

conversion sql query to jpa

I have a query
SELECT d.name, count(e.id) FROM department d LEFT OUTER JOIN employee e on e.department_id = d.id and e.salary > 5000
and how i can convert this to jpa
right now i have:
CriteriaQuery<Object[]> criteria = builder.createQuery(Object[].class);
Root<Department> root = criteria.from(Department.class);
Path<String> name = root.get("name");
Expression<Long> empCount = builder.count(root.get("employees").get("id"));
criteria.multiselect(name,empCount);
TypedQuery<Object[]> query = em.createQuery(criteria);
I simplified both examples by removing ordering and grouping
can anyone tell me how i can modifie my jpa code to get same reslults like from my sql query
thanks in advance
You're not far from the result. The problem is that, AFAIK, you can't add any restriction on the on clause, using JPA. So the query wil have to be rewritten as
SELECT d.name, count(e.id) FROM department d
LEFT OUTER JOIN employee e on e.department_id = d.id
where (e.id is null or e.salary > 5000)
Here is the equivalent of this query not tested):
CriteriaQuery<Object[]> criteria = builder.createQuery(Object[].class);
Root<Department> root = criteria.from(Department.class);
Path<String> name = root.get("name");
Join<Department, Employee> employee = root.join("employees", JoinType.LEFT);
Expression<Long> empCount = builder.count(employee.get("id"));
criteria.multiselect(name,empCount);
criteria.where(builder.or(builder.isNull(employee.get("id")),
builder.gt(employee.get("salary"), 5000)));
TypedQuery<Object[]> query = em.createQuery(criteria);