How would you model this in Django? - django

Question

I want to upload a list of companies to my web app and then run some long background scripts on each company (ie. Webscraping). Each upload will be essentially be a batch task.
I suspect I will probably need to queue these batch processes with something like Redis or Celery, though I am not sure because the batch should start immediately after being being submitted.
I am mainly having trouble figuring out how to create my Models for this logic. So any help there would be greatly appreciated. Right now I've set up a Company model and a Batch model. I'm unsure on how to link these models together appropriately so I can sum up the number of employees from each company in a batch.
Please reference these pictures for visual details.
class Company(models.Model):
domain_name = models.CharField(max_length=100, verbose_name="Domain Name")
company_name = models.CharField(max_length=100, verbose_name="Company Name")
num_employees = models.IntegerField(default=0, verbose_name="# of Employees")
batch = models.ForeignKey('Batch', on_delete=models.CASCADE)
def __str__(self):
return f"Company: {self.company_name}\nEmployees: {self.num_employees}"
class Batch(models.Model):
batch_name = models.CharField(max_length=100, verbose_name="Batch Name")
batch_owner = models.CharField(max_length=100, verbose_name="Batch Owner")
create_date = models.DateTimeField('Date Created', auto_now=True)
# processing = models.BooleanField(verbose_name="Processing Status", default=False)
#property
def total_companies(self):
"""Gets imorted companies sum based on batch"""
pass
#property
def total_employees(self):
"""Gets sum of employees based on batch"""
pass

Answer 1

If there are going to be multiple batch instances for a single company, then a foreign key on the batch model to the company model should work.
class Company(models.Model):
some_fields = ....
class Batch(models.Model):
some_fields = ....
company = models.ForeignKey(Company, on_delete=models.PROTECT)
It looks like you already have a field in the Batch image for a company.

Answer 2

Looks like I figured out the models shortly after, I guess I used stackoverflow as my "rubberduck".
So if anyone runs into a similar issue, the model solution is in my question and in the selected answer.
On top of that, I was also able to map out a mock up of how the form view should work. I used some dummy data in a post_dict variable to imitate form data. And of course the request.method == 'GET' Should be 'POST' But for now I want this to process every time I load the respective URL as I haven't completed the form yet.
from itertools import islice
def create_batch_view(request):
if request.method == 'GET':
post_dict = {'batch_name': 'Some batch name',
'batch_owner': 'Developer',
'batch_items': ['company 1', 'company 2', 'company 3'],
}
batch_model = Batch(batch_name=post_dict['batch_name'],
batch_owner=post_dict['batch_owner'])
batch_model.save()
batch_size = 100
objs = (Company(batch=batch_model, company_name='Test %s' % i) for i in post_dict['batch_items'])
while True:
batch = list(islice(objs, batch_size))
if not batch:
break
Company.objects.bulk_create(batch, batch_size)
return redirect('index')
context = {
'form': BatchForm()
}
return render(request, 'createbatchform.html', context)
Hopefully this helps someone else. Now I just need to figure out the simplest way to run scripts on the batches and report back to the view when its done processing. Should I use a queue manager for this? For example. When a new batch is made, all the companies get added to the database like the example above and also a function is added to a queue manager. That function could look something like this:
def batch_process():
# Get batch
# Get Companies related to batch.
# Loop through companies.
# Scrape Data for company and save it back to database.
# Finish and set Batch.processing = False to signal the process is finished.