I'm trying to do the following: loop through a series of records, if one already exists, I update it. If it doesn't exist, I add it.
For some reason, the adding is working fine, but the updating is not occurring. here's the code:
for interchange in data:
ltype='asdf'
plink='qwer'
e = Part.query.filter_by(interchange=interchange, light_type=ltype).first()
if e is None:
notpresent = notpresent + 1
p = Part(interchange=interchange,light_type=ltype,partslink=plink,ptype=ptype)
db.session.add(p)
else:
present = present + 1
e.partslink = plink
e.ptype = ptype
db.session.commit()
I print out the present and notpresent variables and both are > 0.
Am I handling the update part incorrectly when the record is found? That portion is not getting saved.
you are missing a line:
# ...
else:
present = present + 1
e.partslink = plink
e.ptype = ptype
db.session.add(e) # here 'add' is adding changes to the session, not a new item.
you cannot commit changes if you do not register them.
Related
I have a basic electricity system model that I have built in Pyomo and has been working well. I'm now looking to make it a bit more sophisticated, adding in some additional variables, such as start costs for individual generators.
I pasted below some of the code I have added - I can provide more contextual code if required, but it is only these lines that have been added to trigger this error. The error thrown back by Pyomo is shown in quotes in the header to this query. I have deliberately left in some commenting out of lines, where I simplified the code to try to identify where the issue is. To no avail: I still get an error with the commenting out shown below:
model.StartFuelCost = Param(model.GeneratorName, model.Resource, default=0)
model.GeneratorCommitted = Var(model.GeneratorName, model.Hour, domain=Boolean, initialize=0)
model.GeneratorStart = Var(model.GeneratorName, model.Hour, domain=Boolean, initialize=0)
model.StartFuelCostByGenerator = Var(model.GeneratorName, model.Resource, model.Hour, domain=NonNegativeReals, initialize=0.0)
model.StartFuelCostTotal = Var(model.Resource, model. Hour, domain.NonNegativeReals, initialize=0.0)
...
def GeneratorCommitted_rule(model,g,h):
# if model.Generation[g,h] > 0:
return model.GeneratorCommitted[g,h] == 1
# else:
# return model.GeneratorCommitted[g,h] == 0
model.SupplyDemand_Calc2 = Constraint(model.GeneratorName, model.Hour, rule=GeneratorCommitted_rule)
# ISSUE: Either need to remove conditionality or pass info from the last time step
def GeneratorStart_rule(model,g,h):
# if model.Hour > 1:
# return max(0, model.GeneratorCommitted[g,h] - model.GeneratorCommitted[g,h-1]) == model.GeneratorStart[g,h]
# else:
return model.GeneratorCommitted[g,h] == model.GeneratorStart[g,h]
model.SupplyDemand_Calc3 = Constraint(model.GeneratorName, model.Hour, rule=GeneratorStart_rule)
def StartFuelCostByGenerator_rule(model,g,r,h):
return model.StartFuelCost[g,r] * model.ResourcePrice[r] * model.GeneratorStart[g,h] == model.StartFuelCostByGenerator[g,r,h]
model.Costing_Calc5 = Constraint(model.GeneratorName, model.Resource, model.Hour, rule=StartFuelCostByGenerator_rule)
def StartFuelCostTotal_rule(model,r,h):
return sum(model.StartFuelCostByGenerator[g,r,h] for g in model.GeneratorName) == model.StartFuelCostTotal[r,h]
model.Costing_Calc6 = Constraint(model.Resource, model.Hour, rule=StartFuelCostTotal_rule)
This is your problem:
model.StartFuelCostTotal = Var(model.Resource, model. Hour, domain.NonNegativeReals, initialize=0.0)
You have "." (dot) notation there with domain, so it is trying to figure out what this thing is that you call domain... You want to change that to an =.
In the future, it is much easier (and you are more likely) to get help if you post the actual code with the error trace which says which line contains the error.
I've followed the guide in the queryset documentation as per (https://docs.djangoproject.com/en/1.10/ref/models/querysets/#update-or-create) but I think im getting something wrong:
my script checks against an inbox for maintenance emails from our ISP, and then sends us a calendar invite if you are subscribed and adds maintenance to the database.
Sometimes we get updates on already planned maintenance, of which i then need to update the database with the new date and time, so im trying to use "update or create" for the queryset, and need to use the ref no from the email to update or create the record
#Maintenance
if sender.lower() == 'maintenance#isp.com':
print 'Found maintenance in mail: {0}'.format(subject)
content = Message.getBody(mail)
postcodes = re.findall(r"[A-Z]{1,2}[0-9R][0-9A-Z]? [0-9][A-Z]{2}", content)
if postcodes:
print 'Found Postcodes'
else:
error_body = """
Email titled: {0}
With content: {1}
Failed processing, could not find any postcodes in the email
""".format(subject,content)
SendMail(authentication,site_admins,'Unprocessed Email',error_body)
Message.markAsRead(mail)
continue
times = re.findall("\d{2}/\d{2}/\d{4} \d{2}:\d{2}", content)
if times:
print 'Found event Times'
e_start_time = datetime.strftime(datetime.strptime(times[0], "%d/%m/%Y %H:%M"),"%Y-%m-%dT%H:%M:%SZ")
e_end_time = datetime.strftime(datetime.strptime(times[1], "%d/%m/%Y %H:%M"),"%Y-%m-%dT%H:%M:%SZ")
subscribers = []
clauses = (Q(site_data__address__icontains=p) for p in postcodes)
query = reduce(operator.or_, clauses)
sites = Circuits.objects.filter(query).filter(circuit_type='MPLS', provider='KCOM')
subject_text = "Maintenance: "
m_ref = re.search('\[(.*?)\]',subject).group(1)
if not len(sites):
#try use first part of postcode
h_pcode = postcodes[0].split(' ')
sites = Circuits.objects.filter(site_data__postcode__startswith=h_pcode[0]).filter(circuit_type='MPLS', provider='KCOM')
if not len(sites):
#still cant find a site, send error
error_body = """
Email titled: {0}
With content: {1}
I have found a postcode, but could not find any matching sites to assign this maintenance too, therefore no meeting has been sent
""".format(subject,content)
SendMail(authentication,site_admins,'Unprocessed Email',error_body)
Message.markAsRead(mail)
continue
else:
#have site(s) send an invite and create record
for s in sites:
create record in circuit maintenance
maint = CircuitMaintenance(
circuit = s,
ref = m_ref,
start_time = e_start_time,
end_time = e_end_time,
notes = content
)
maint, CircuitMaintenance.objects.update_or_create(ref=m_ref)
#create subscribers for maintenance
m_ref, is the unique field that will match the update, but everytime I run this in tests I get
sites_circuitmaintenance.start_time may not be NULL
but I've set it?
If you want to update certain fields provided that a record with certain values exists, you need to explicitly provide the defaults as well as the field names.
Your code should look like this:
CircuitMaintenance.objects.update_or_create(default=
{'circuit' : s,'start_time' : e_start_time,'end_time' : e_end_time,'notes' : content}, ref=m_ref)
The particular error you are seeing is because update_or_create is creating an object because one with rer=m_ref does not exist. But you are not passing in values for all the not null fields. The above code will fi that.
This involves pretty much the same code I just asked a different question about this morning, so if it looks familiar, that's because it is.
class LbcSubtopicSpider(scrapy.Spider):
...irrelevant/sensitive code...
rawTranscripts = []
rawTranslations = []
def parse(self, response):
rawTitles = []
rawVideos = []
for sel in response.xpath('//ul[1]'): #only scrape the first list
...irrelevant code...
index = 0
for sub in sel.xpath('li/ul/li/a'): #scrape the sublist items
index += 1
if index%2!=0: #odd numbered entries are the transcripts
transcriptLink = sub.xpath('#href').extract()
#url = response.urljoin(transcriptLink[0])
#yield scrapy.Request(url, callback=self.parse_transcript)
else: #even numbered entries are the translations
translationLink = sub.xpath('#href').extract()
url = response.urljoin(translationLink[0])
yield scrapy.Request(url, callback=self.parse_translation)
print rawTitles
print rawVideos
print "translations:"
print self.rawTranslations
def parse_translation(self, response):
for sel in response.xpath('//p[not(#class)]'):
rawTranslation = sel.xpath('text()').extract()
rawTranslation = ''.join(rawTranslation)
#print rawTranslation
self.rawTranslations.append(rawTranslation)
#print self.rawTranslations
My problem is that "print self.rawTranslations" in the parse(...) method prints nothing more than "[]". This could mean one of two things: it could be resetting the list right before printing, or it could be printing before the calls to parse_translation(...) that populate the list from links parse(...) follows are finished. I'm inclined to suspect it's the latter, as I can't see any code that would reset the list, unless "rawTranslations = []" in the class body is run multiple times.
Worth noting is that if I uncomment the same line in parse_translation(...), it will print the desired output, meaning it's extracting the text correctly and the problem seems to be unique to the main parse(...) method.
My attempts to resolve what I believe is a synchronization problem were pretty aimless--I just tried using an RLock object based on as many Google tutorials I could find and I'm 99% sure I misused it anyway, as the result was identical.
The problem here is that you are not understanding how scrapy really works.
Scrapy is a crawling framework, used for creating website spiders, not just for doing requests, that's the requests module for.
Scrapy's requests work asynchronously, when you call yield Request(...) you are adding requests to a stack of requests that will be executed at some point (you don't have control over it). Which means that you can't expect that some part of your code after the yield Request(...) will be executed at that moment. In fact, your method should always end yielding a Request or an Item.
Now from what I can see and most cases of confusion with scrapy, you want to keep populating an item you created on some method, but the information you need is on a different request.
On that case, communication is usually done with the meta parameter of the Request, something like this:
...
yield Request(url, callback=self.second_method, meta={'item': myitem, 'moreinfo': 'moreinfo', 'foo': 'bar'})
def second_method(self, response):
previous_meta_info = response.meta
# I can access the previous item with `response.meta['item']`
...
So this seems like somewhat of a hacky solution, especially since I just learned of Scrapy's request priority functionality, but here's my new code that gives the desired result:
class LbcVideosSpider(scrapy.Spider):
...code omitted...
done = 0 #variable to keep track of subtopic iterations
rawTranscripts = []
rawTranslations = []
def parse(self, response):
#initialize containers for each field
rawTitles = []
rawVideos = []
...code omitted...
index = 0
query = sel.xpath('li/ul/li/a')
for sub in query: #scrape the sublist items
index += 1
if index%2!=0: #odd numbered entries are the transcripts
transcriptLink = sub.xpath('#href').extract()
#url = response.urljoin(transcriptLink[0])
#yield scrapy.Request(url, callback=self.parse_transcript)
else: #even numbered entries are the translations
translationLink = sub.xpath('#href').extract()
url = response.urljoin(translationLink[0])
yield scrapy.Request(url, callback=self.parse_translation, \
meta={'index': index/2, 'maxIndex': len(query)/2})
print rawTitles
print rawVideos
def parse_translation(self, response):
#grab meta variables
i = response.meta['index']
maxIndex = response.meta['maxIndex']
#interested in p nodes without class
query = response.xpath('//p[not(#class)]')
for sel in query:
rawTranslation = sel.xpath('text()').extract()
rawTranslation = ''.join(rawTranslation) #collapse each line
self.rawTranslations.append(rawTranslation)
#increment number of translations done, check if finished
self.done += 1
print self.done
if self.done==maxIndex:
print self.rawTranslations
Basically, I just kept track of how many requests were completed and making some code conditional on a request being final. This prints the fully populated list.
I am trying to set my model form with data as well as initial_dict. They both are working fine indivisiually, but when i put both in the method call, Initial does not work.
here is the code.
state = Territory.objects.filter(abbreviation=request.GET.get('territory', ''))
if state:
initial_dict['state'] = state.get()
ctx['add_customer'] = AddCustomer(data=request.GET, initial=initial_dict)
But the state (Dropdown) does not get selected.
This works fine
ctx['add_customer'] = AddCustomer(initial=initial_dict)
Ideas?
This is actually solved my problem.
initial_dict = request.GET.dict()
state = Territory.objects.filter(abbreviation=initial_dict.get('state', ''))
if state:
initial_dict['state'] = state.get()
ctx['add_customer'] = AddCustomer(initial=initial_dict)
Converted data to Python Dict & use only Initial.
I am still not sure why that problem was occurring.
I am trying to write a facebook app where user can see the status history of his friends.
Everything seems to work fine until I try to save the status information in my DB.
here is code :
class UserStatus(models.Model):
facebookid = models.IntegerField()
time = models.IntegerField()
status_msg = models.CharField(max_length = 2000)
#facebook.require_login()
def canvas(request):
# Get the User object
user, created = FacebookUser.objects.get_or_create(id = request.facebook.uid)
user_lastname = request.facebook.users.getInfo([request.facebook.uid], ['last_name'])[0]['last_name']
query = "SELECT time,message FROM status WHERE uid=%s" % request.facebook.uid
result = request.facebook.fql.query(query)
So result give me all the information of the status.
so my problem is its give error when I try to save it.
userstatus = UserStatus()
for item in result:
userstatus.facebookid = request.facebook.uid
userstatus.time = item.time
userstatus.msg = item.message
userstatus.save()
error:
Errors while loading page from application
Received HTTP error code 500 while loading
So how can I fix this.
thanks.
First you should check if you are getting results from this,
result = request.facebook.fql.query(query)
Make sure that the results are in correct format required by your model ( uid is integer, time is integer and message is string.
Again make sure that result is a valid python object and not a JSON string/Object.
Remember python is not fully compatible with JSON so if result is JSON then do this to convert it to python Object,
import simplejson
result = simpljson.loads(result) # if result was a JSON string
result = simpljson.loads(simplejson.dumps(result)) # if result was a JSON object
Check if now result is a list of dictionaries { "time" : 123456, "messaage": "xyz"}.
for item in result:
userstatus = UserStatus()
userstatus.facebookid = request.facebook.uid
userstatus.time = item["time"]
userstatus.msg = item["message"]
userstatus.save()
And you should not have any errors now.