Here is my PostgreSQL statement.
select round(sum("amount") filter(where "date">=now()-interval '12 months')/12,0) as avg_12month from "amountTab"
How to use this in Django?
I have an object called 'Devc', with attribute 'date'.
I want to get the sum of the specific data within past 12 months, not past 365 days.
You can try this to get the data within the past 12 months.
today= datetime.now()
current_month_first_day = today.replace(day = 1)
previous_month_last_day = current_month_first_day - timedelta(days = 1)
past_12_month_first_day = previous_month_last_day - timedelta(days = 360)
past_12_month_first_day = past_12_month_first_day.replace(day = 1)
past_12_month_avg = Devc.objects.filter(date__range=(past_12_month_first_day,current_month_first_day)).aggregate(Sum('amount'))['amount']
import datetime
start = datetime.fromtimestamp(float(1485008513.00000))
end = datetime.fromtimestamp(float(1485788517.80000))
#Duration
duration = end - start
My result is :
9 days, 0:40:04.800000
But it must be like this (without days, only hours, minutes and seconds) :
216:40:04.800000
Thanks a lot !
Not elegant, but works (for your example, durations less then a day and much more then 1000 days) - but its ugly:
import datetime
start = datetime.datetime.fromtimestamp(float(1485788515.0000))
end = datetime.datetime.fromtimestamp(float(1485788517.80000))
#Duration
duration = end - start
dur = str(duration).split(',')
print dur
# less then a day is not str() as 0days, ... so we fix that here by introducing artificial
# zero day if a split only retunrs 1 element
if len(dur) < 2:
d = ["0", dur[0]]
dur = d
dayHours = int(dur[0].replace('days',''))*24 # remove the days, mult with 24
hours = dur[1].split(':')[0] # get the partial hours of this part
minsSecs = ':'.join(dur[1].split(':')[1:]) # split+join the rest from the hours
# print all combined
print (str( dayHours+ int(hours) ) + ':' + minsSecs)
Output:
216:40:04.800000
Maybe better:
totSec = duration.total_seconds()
hours = totSec // (60*60)
mins = (totSec - (hours*60*60)) // 60
secs = totSec - (hours*60*60) - mins * 60
print "{:2}:{:2}:{:09.6f}".format(int(hours),int(mins),secs)
Really struggling with this one... Forgive the longish post.
I have an experiment that on each trial displays some stimulus, collects a response, and then moves on to the next trial.
I would like to incorporate an optimizer that runs in between trials and therefore must have a specific time-window designated by me to run, or it should be terminated. If it's terminated, I would like to return the last set of parameters it tried so that I can use it later.
Generally speaking, here's the order of events I'd like to happen:
In between trials:
Display stimulus ("+") for some number of seconds.
While this is happening, run the optimizer.
If the time for displaying the "+" has elapsed and the optimizer has
not finished, terminate the optimizer, return the most recent set of parameters it tried, and move on.
Here is some of the relevant code I'm working with so far:
do_bns() is the objective function. In it I include NLL['par'] = par or q.put(par)
from scipy.optimize import minimize
from multiprocessing import Process, Manager, Queue
from psychopy import core #for clock, and other functionality
clock = core.Clock()
def optim(par, NLL, q)::
a = minimize(do_bns, (par), method='L-BFGS-B', args=(NLL, q),
bounds=[(0.2, 1.5), (0.01, 0.8), (0.001, 0.3), (0.1, 0.4), (0.1, 1), (0.001, 0.1)],
options={"disp": False, 'maxiter': 1, 'maxfun': 1, "eps": 0.0002}, tol=0.00002)
if __name__ == '__main__':
print('starting optim')
max_time = 1.57
with Manager() as manager:
par = manager.list([1, 0.1, 0.1, 0.1, 0.1, 0.1])
NLL = manager.dict()
q = Queue()
p = Process(target=optim, args=(par, NLL, q))
p.start()
start = clock.getTime()
while clock.getTime() - start < max_time:
p.join(timeout=0)
if not p.is_alive():
break
if p.is_alive():
res = q.get()
p.terminate()
stop = clock.getTime()
print(NLL['a'])
print('killed after: ' + str(stop - start))
else:
res = q.get()
stop = clock.getTime()
print('terminated successfully after: ' + str(stop - start))
print(NLL)
print(res)
This code, on its own, seems to sort of do what I want. For example, the res = q.get() right above the p.terminate() actually takes something like 200ms so it will not terminate exactly at max_time if max_time < ~1.5s
If I wrap this code in a while-loop that checks to see if it's time to stop presenting the stimulus:
stim_start = clock.getTime()
stim_end = 5
print('showing stim')
textStim.setAutoDraw(True)
win.flip()
while clock.getTime() - stim_start < stim_end:
# insert the code above
print('out of loop')
I get weird behavior such as multiple iterations of the whole code from the beginning...
showing stim
starting optim
showing stim
out of loop
showing stim
out of loop
[1.0, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001]
killed after: 2.81303440395
Note the multiple 'showing stim's' and 'out of loop's.
I'm open to any solution that accomplishes my goal :|
Help and thank you!
Ben
General remark
Your solution would give me nightmares! I don't see a reason to use multiprocessing here and i'm not even sure how you grab those updated results before termination. Maybe you got your reason for this approach, but i highly recommend something else (which has a limitation).
Callback-based approach
The general idea i would pursue is the following:
fire up your optimizer with some additional time-limit information and some callback enforcing this
the callback is called in each iteration of this optimizer
if time-limit reached: raise a customized Exception
The limits:
as the callback is only called once in each iteration, there is some limited sequence of points in time where the optimizer might get stopped
the potential difference is highly dependent on iteration-time for your problem! (numerical-differentiation, huge-data, slow function eval; all this matters)
if not exceeding some given time is of highest priority, this approach might be not right or you would need some kind of safeguarded interpolation to reason if one more iteration is possible in time
or: combine your kind of killing off workers with my approach of updating intermediate-results through some callback
Example code (bit hacky):
import time
import numpy as np
import scipy.sparse as sp
import scipy.optimize as opt
np.random.seed(1)
""" Fake task: sparse NNLS """
M, N, D = 2500, 2500, 0.1
A = sp.random(M, N, D)
b = np.random.random(size=M)
""" Optimization-setup """
class TimeOut(Exception):
"""Raise for my specific kind of exception"""
def opt_func(x, A, b):
return 0.5 * np.linalg.norm(A.dot(x) - b)**2
def opt_grad(x, A, b):
Ax = A.dot(x) - b
grad = A.T.dot(Ax)
return grad
def callback(x):
time_now = time.time() # probably not the right tool in general!
callback.result = [np.copy(x)] # better safe than sorry -> copy
if time_now - callback.time_start >= callback.time_max:
raise TimeOut("Time out")
def optimize(x0, A, b, time_max):
result = [np.copy(x0)] # hack: mutable type
time_start = time.time()
try:
""" Add additional info to callback (only takes x as param!) """
callback.time_start = time_start
callback.time_max = time_max
callback.result = result
res = opt.minimize(opt_func, x0, jac=opt_grad,
bounds=[(0, np.inf) for i in range(len(x0))], # NNLS
args=(A, b), callback=callback, options={'disp': False})
except TimeOut:
print('time out')
return result[0], opt_func(result[0], A, b)
return res.x, res.fun
print('experiment 1')
start_time = time.perf_counter()
x, res = optimize(np.zeros(len(b)), A, b, 0.1) # 0.1 seconds max!
end_time = time.perf_counter()
print(res)
print('used secs: ', end_time - start_time)
print('experiment 2')
start_time = time.perf_counter()
x_, res_ = optimize(np.zeros(len(b)), A, b, 5) # 5 seconds max!
end_time = time.perf_counter()
print(res_)
print('used secs: ', end_time - start_time)
Example output:
experiment 1
time out
422.392771467
used secs: 0.10226839151517493
experiment 2
72.8470708728
used secs: 0.3943936788825996
I am working with Django to see how to handle large databases. I use a database with fields name, age, date of birth(dob) and height. The database has about 500000 entries. I have to find the average height of persons of (1) same age and (2) born in same year. The aggregate function in querying table takes about 10s. Is it usual or am I missing something?
For age:
age = [i[0] for i in Data.objects.values_list('age').distinct()]
ht = []
for each in age:
aggr = Data.objects.filter(age=each).aggregate(ag_ht=Avg('height')
ht.append(aggr)
From dob,
age = [i[0].year for i in Data.objects.values_list('dob').distinct()]
for each in age:
aggr = Data.objects.filter(dob__contains=each).aggregate(ag_ht=Avg('height')
ht.append(aggr)
The year has to be extracted from dob. It is SQLite and I cannot use __year (join).
For these queries to be efficient, you have to create indexes on the age and dob columns.
You will get a small additional speedup by using covering indexes, i.e., using two-column indexes that also include the height column.
full version with time compare loop and query set version
import time
from dd.models import Data
from django.db.models import Avg
from django.db.models.functions import ExtractYear
for age
start = time.time()
age = [i[0] for i in Data.objects.values_list('age').distinct()]
ht = []
for each in age:
aggr = Data.objects.filter(age=each).aggregate(ag_ht=Avg('height'))
ht.append(aggr)
end = time.time()
loop_time = end - start
start = time.time()
qs = Data.objects.values('age').annotate(ag_ht=Avg('height')).order_by('age')
ht_qs = qs.values_list('age', 'ag_ht')
end = time.time()
qs_time = end - start
print loop_time / qs_time
for dob year, with easy refactoring your version(add set in the years)
start = time.time()
years = set([i[0].year for i in Data.objects.values_list('dob').distinct()])
ht_year_loop = []
for each in years:
aggr = Data.objects.filter(dob__contains=each).aggregate(ag_ht=Avg('height'))
ht_year_loop.append((each, aggr.get('ag_ht')))
end = time.time()
loop_time = end - start
start = time.time()
qs = Data.objects.annotate(dob_year=ExtractYear('dob')).values('dob_year').annotate(ag_ht=Avg('height'))
ht_qs = qs.values_list('dob_year', 'ag_ht')
end = time.time()
qs_time = end - start
print loop_time / qs_time
I have some problems the first one is that I can't update the plot limits of the y axis and the second is that I want to see 6 lines from each sensor, as you can see in the picture I see only one if I make some changes I see all the sensors variations in one line
here is the code where I create the plot and a picture of this:
http://i.imgur.com/ogFoMDJ.png?1
# Flag variables
self.isLogging = False
# Create data buffers
self.N = 70
self.n = range(self.N)
self.M = 6 # just one lead - i.e. 1 number per sample
self.x = 0 * numpy.ones(self.N, numpy.int)
# Data logging file
self.f = 0
# Create plot area and axes
self.x_max = 500
self.x_min = 330
self.fig = Figure(facecolor='#e4e4e4')
self.canvas = FigureCanvasWxAgg(self, -1, self.fig)
self.canvas.SetPosition((330,50))
self.canvas.SetSize((550,280))
self.ax = self.fig.add_axes([0.08,0.1,0.86,0.8])
self.ax.autoscale(False)
self.ax.set_xlim(0, self.N - 1)
self.ax.set_ylim(self.x_min, self.x_max)
self.ax.plot(self.n,self.x)
# Filter taps
self.taps = [0, 0, 0]
# Create timer to read incoming data and scroll plot
self.timer = wx.Timer(self)
self.Bind(wx.EVT_TIMER, self.GetSample, self.timer)
And here is where I grab the data and I try to update the limits of the plot
if len(sample_string) != 6:
sample_string = sample_string[0:-1]
self.taps[1:3] = self.taps[0:2]
self.taps[0] = int(array[1])
#value = 0.5 * self.taps[0] + 0.5 * self.taps[2]
value = self.taps[0]
self.x[0:self.N-1] = self.x[1:]
self.x[self.N-1] = value
# print sample to data logging file
if self.f != 0:
self.f.write(str(value))
self.f.write("\n")
# update plot limits
maxval = max(self.x[:])
minval = min(self.x[:])
self.x_max += ((maxval + 10) - self.x_max) / 100.0
self.x_min -= (self.x_min - (minval - 10)) / 100.0
# Update plot
self.ax.cla()
self.ax.autoscale(False)
self.ax.set_xlim(0, self.N - 1)
self.ax.set_ylim(self.x_min, self.x_max)
self.ax.plot(self.n, self.x)
self.canvas.draw()
if b7 == True:
self.textctrl0.Clear()
self.textctrl0.AppendText(array[1])
self.textctrl1.Clear()
self.textctrl1.AppendText(array[2])
self.textctrl2.Clear()
self.textctrl2.AppendText(array[3])
self.textctrl3.Clear()
self.textctrl3.AppendText(array[4])
self.textctrl4.Clear()
self.textctrl4.AppendText(array[5])
self.textctrl5.Clear()
self.textctrl5.AppendText(array[6])
b7=False
p.s I removed the faulty code where I tried to add the other sensors,here is only the working code for the one sensor plot..