Database entries added in background task don't show up on heroku - django

I have a background task in my django application, which enters all the rows from a csv into one of the tables in my database. I pass the csv via the admin site, that creates a background task, that I can run with python manage.py process_tasks. This all works locally, but on my heroku app, for some reason it doesn't.
I thought maybe inputting data is impossible from the heroku console, but if I run python manage.py shell on the heroku console, I can input data just fine.
This is the code that inputs the data into the database:
from background_task import background
...
#background(schedule=5)
def save_course_from_df(df):
df = pandas.read_json(df)
db = 0
for index, row in df.iterrows():
print("%s percent done!" % str(db / df.shape[0]))
db += 1
values = dict(row)
values = {key: values[key] for key in values.keys()
if type(values[key]) != float or not math.isnan(values[key])}
try:
Course.objects.update_or_create(
url=row['url'],
defaults=values
)
except IntegrityError:
pass
print('done!')
I run this by opening the heroku console and running 'python manage.py process_tasks'. I get the print messages, and no error is being thrown. Still, my database doesn't change.
I expected that after the task runs I would have a full table. Instead, nothing changed.

It seems the problem was with my migrations. For some reason the database on the heroku app had a not null constraint for a column whose model had the null=True argument.

Related

'something here' matching query does not exist

I created a web app and deployed on Heroku successfully even database migrations. When I open the app I see the error like this:
'something here' matching query does not exist.
App URL: https://lp7.herokuapp.com/lp7/
App isn't working and if I remove this data feild from model, then app works but no single data is coming from database. But, when I go to heroku database it shows:
No. of Tables = 28
No. Rows = 220
Size of data = 9.4Mb
It means, the all migrations exists on heroku but not showing on website.
Any solution..?
You need to update the table for model Topbar in Heroku. You can use admin-site to update it.
Also, for future, you might want to change from:
num = TopBar.objects.get()
to
num = TopBar.objects.last()
So it will return the last object created in queryset. It will return None if no object has been created for TopBar
Looking at the traceback https://lp7.herokuapp.com/lp7/ it is bad here
num = TopBar.objects.get() you should pass something matching
if you dont need it do it like
try:
num = TopBar.objects.get(id=1)
except TopBar.DoesNotExist:
pass
#handle if not found logic here
For migrating data from your local database to heroku database is to run:
python manage.py dumpdata yourapp > yourapp/fixtures/app_data.json
Then you need to commit this file to heroku branch, for example:
git commit heroku main
After commiting run the following command to load data into heroku database:
heroku run python manage.py loaddata app_data

OperationalError1075 'Incorrect table definition'

I changed one of my models to go from
serial_number = models.BigIntegerField(unique=True)
to
serial_number = models.AutoField(primary_key=True)
and ran
$ python manage.py check -v 3
$ python manage.py makemigrations -v 2
$ python manage.py migrate -v 2
However, when I try to run the unit tests (which passed fine before), I get this error:
[...]
File "/home/usr/Envs/intranet/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue django.db.utils.OperationalError: (1075, 'Incorrect table definition; there can be only one auto column and it must be defined as a key')
How to resolve this?
The Error1075 'Incorrect table definition' came from Mysql, not Django. That is because the table u changed got 2 auto_created fields, while it is only one field alled allow for one table in Mysql.
In Django, just add a parm 'auto_created':
serial_number = models.AutoField(primary_key=True, auto_created=False)
then remove u old migrations file last time and make migrations again, continue migrate will be ok.
So far the only "solution" was to trash all the migrations and re-create the database. Thus trashing all the old data. But it is less of a pain than fixing this craziness.

celery parallel tasking error 'no result backend configured'

Running django-celery 3.1.16, Celery 3.1.17, Django 1.4.16. Trying to run some parallel tasks using 3 workers and collect the results using the following:
from celery import group
positions = []
jobs = group(celery_calculate_something.s(data.id) for data in a_very_big_list)
results = jobs.apply_async()
positions.extend(results.get())
The task celery_calculate_something returns an object to place the in the results list:
app.task(ignore_result=False)
def celery_calculate_something(id):
<do stuff>
No matter what I try, I always get the same result when calling get() on results:
No result backend configured. Please see the documentation for more information.
However, the results backend IS configured - I have many other tasks with ignore_result=False merrily adding to the tasks meta table in django_celery. It is something to do with using the results returned from group(). I should note it is not set explicitly in settings - it seems that django-celery has set it automatically for you.
I have the worker collecting events using:
manage.py celery worker -l info -E
and celerycam running with
python manage.py celerycam
Inspecting the results object returned (an instance of GroupResult) I can see that the backend attr is an instance of DisabledBackend. Is this the problem? What have I mis-understood?
You did not configure the results backend, so basically you need tables to store the results, since you have django-celery add it to INSTALLED_APPS in your settings.py file and then perform the migration (python manage.py migrate) After that open your celery.py file and modify your backend to djcelery.backends.database:DatabaseBackend. Here's an example
app = Celery('almanet',
broker='amqp://guest#localhost//',
backend='djcelery.backends.database:DatabaseBackend',
include=['alm_crm.tasks'] #References your tasks. Donc forget to put the whole absolute path.
)
After that you can import results from celery import result Now you can save the result and extract the result by job.id
from celery import group
positions = []
jobs = group(celery_calculate_something.s(data.id) for data in
a_very_big_list)
results = jobs.apply_async()
results.save()
some_task_result = result.GroupResult.restore(results.id)
print some_task_results.ready()

Another South "table does not exist" issue: none of the previously posted solutions working

I am trying to make an app using Django and am using South to handle migrations. After I define the app's models.py, I include south in the "INSTALLED_APPS" in settings.py. Then I sync my database. When I validate the database, I get 0 errors. Then I execute the following commands on the command prompt:
C:\Users\abagaria\Desktop\IntegrateID\website>python manage.py schemamigration w
ebsite.integrate --initial
Creating migrations directory at 'C:\Users\abagaria\Desktop\IntegrateID\website\
website\integrate\migrations'...
Creating __init__.py in 'C:\Users\abagaria\Desktop\IntegrateID\website\website\i
ntegrate\migrations'...
+ Added model integrate.Publisher
+ Added model integrate.Author
+ Added model integrate.Book
+ Added M2M table for authors on integrate.Book
Created 0001_initial.py. You can now apply this migration with: ./manage.py migr
ate integrate
C:\Users\abagaria\Desktop\IntegrateID\website>python manage.py migrate website.i
ntegrate
Running migrations for integrate:
- Migrating forwards to 0001_initial.
> integrate:0001_initial
FATAL ERROR - The following SQL query failed: CREATE TABLE "integrate_publisher"
("id" integer NOT NULL PRIMARY KEY, "name" varchar(30) NOT NULL, "address" varc
har(50) NOT NULL, "city" varchar(60) NOT NULL, "state_province" varchar(30) NOT
NULL, "country" varchar(50) NOT NULL, "website" varchar(200) NOT NULL)
The error was: table "integrate_publisher" already exists
! Error found during real run of migration! Aborting.
! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.
! You *might* be able to recover with: = DROP TABLE "integrate_publisher"; []
= DROP TABLE "integrate_author"; []
= DROP TABLE "integrate_book"; []
= DROP TABLE "integrate_book_authors"; []
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: integrate:0001_initial
DatabaseError: table "integrate_publisher" already exists
I know that a lot of people have faced similar problems while using south, but usually in their case, they make the mistake of executing the "--initial" command more than once-- thereby causing south to make more than one __initial file in the migrations directory. But in my case, South thinks that the table already exists even when I make the first migration!
I have also tried:
deleting the migrations directory
deleting ghost migrations
making a "fake" migration
and then running the actual migration
Can someone please tell me how I fix this problem and can start defining my models again?
If you already have tables in database, do not use --initial, instead you need convert_to_south command. Delete directory "migrations", all tables from database and run the following commands:
python manage.py syncdb
python manage.py convert_to_south appname
python manage.py syncdb --migrate
http://south.readthedocs.org/en/latest/convertinganapp.html

How can I run django shell commands from a bash script

Instead of repeatedly deleting my tables, recreating them and populating with data in my dev env, I decided to create a bash script called reset_db that does this for me. I got it to whack the tables, recreate them. But it's not able to populated the tables with data from the django orm.
I try to do this by calling the django shell from the script and then running ORM commands to populate my tables. But it seems like the django shell commands are not running.
I tried running the django orm commands manually/directly in the shell and they run fine but not from within the bash script.
The errors I get are:
NameError: name 'User' is not defined
NameError: name 'u1' is not defined
NameError: name 'm' is not defined
Here is my script:
#!/bin/bash
set +e
RUN_ON_MYDB="psql -X -U user --set ON_ERROR_STOP=on --set AUTOCOMMIT=off rcamp1"
$RUN_ON_MYDB <<SQL # Whack tables
DROP TABLE rcamp_merchant CASCADE;
DROP TABLE rcamp_customer CASCADE;
DROP TABLE rcamp_point CASCADE;
DROP TABLE rcamp_order CASCADE;
DROP TABLE rcamp_custmetric CASCADE;
DROP TABLE rcamp_ordermetric CASCADE;
commit;
SQL
python manage.py syncdb # Recreate tables
python manage.py shell <<ORM # Start django shell. Problem starts here.
from rcamp.models import Customer, Merchant, Order, Point, CustMetric, OrderMetric
u1 = User.objects.filter(pk=5)
m = Merchant(u1, full_name="Bill Gates")
m
ORM
I'm new to both django and shell scripting. Thanks for your help.
You should look at creating a fixture to populate your db https://docs.djangoproject.com/en/dev/howto/initial-data/
You need to import User explicitly. The django package and a few other things are automatically imported, but not everything you might want.
Also, to avoid not know what to import, there are management commands. This will leverage your Django and Python. You can learn shell scripting later.
clearly seen in your mistakes is not recognized as a model class User django-admin maybe you lack some import or something like this
from django.db import models
User import from django.contrib.auth.models
, by the way In line
User.objects.filter u1 = (pk = 5)
I think I put
u1 = User.objects.filter (pk = 5). First ()
at the end.
Anyway, here I leave some threads that may be of help,
https://docs.djangoproject.com/en/dev/ref/django-admin/
http://www.stackoverflow.com/questions/6197256/django-user-model-fields-at-adminmodel
https://groups.google.com/forum/?fromgroups = #! topic/django-users/WrVp1DDFrX8
Hope this helps.