I have a set of Django commands that do some complex operations, and I'd like to begin tracking the duration of how long they take, in order to monitor their performance over time.
I've struggled to figure out how to instrument newrelic to capture the duration. I've tried everything from 'assuming' the standard django newrelic agent will magically capture it, to try to instrument them myself as a custom event, all with varying degrees of success.
What is the best way to do this, and does newrelic provide something to facilitate this? The docs were very lacking for this kind of scenario.
Additionally, I would consider these as background_tasks - what is the background_task field in newrelic used for?
Thanks
At New Relic we do have a prototype for some changes we are considering making to the Python agent which will add support for monitoring nominated Django admin commands. We are still working through some details of those changes but it is possible it will make it in to one of the upcoming releases over the next couple of months.
In the interim, what you can do is add something like the following to your agent configuration file:
[newrelic:django-manage]
app_name = Python Application (Django Management Commands)
startup_timeout = 10.0
[background-task:django-manage-syncdb]
enabled = true
function = django.core.management.commands.syncdb:Command.handle_noargs
name = syncdb
group = Django
You would then run the Django manage.py script as:
NEW_RELIC_CONFIG_FILE=newrelic.ini
NEW_RELIC_ENVIRONMENT=django-manage
export NEW_RELIC_CONFIG_FILE
export NEW_RELIC_ENVIRONMENT
newrelic-admin run-python manage.py syncdb
When the data is then reported into our UI, it will appear under 'Type -> Other transactions -> Django' in the 'Transactions' tab for the specified application.
The 'newrelic:django-manage' section is an environment section that allows you to override the 'app_name' setting defining which application in our UI the data should be reported to. The 'NEW_RELIC_ENVIRONMENT' variable says that the environment section should be used as overrides for the default 'newrelic' section'.
Even if you do not override 'app_name' and have it inherit the value from the default 'newrelic' section, you still must set the 'startup_timeout' setting. This is necessary as under normal circumstances the agent will only lazily register itself in the background when the first web request is being handled by a web application. Having registration being done lazily will though cause a problem when tracking Django management commands as a background task. This is because there is only the one function call to be monitored, and not waiting for agent registration to occur will result in no data being captured. Setting a non zero value for the startup timeout therefore says to wait for registration to occur.
If for some reason agent registration takes too long, the timeout will expire after the specified time and the management command allowed to proceed anyway. Either way, the need to wait means that there will be a delay in actually starting the management command. This should be kept in mind if the management command needs to be run as quickly as possible.
If necessary, a 'shutdown_timeout' can also be specified if the uploading of data captured by the agent is consistently taking longer than the default 2.5 seconds and therefore not always being reported.
As to the 'background-task' section, it defines the specific function which should be monitored and reported as a background task. In this case we are targeting the particular function implementing the Django 'syncdb' command.
Related
I've hosted my app in Google Cloud Run, a simple Vue's frontend connected to a django API. The problem arises when I try to set maintenance mode to the API, protecting it from unexpected calls. For this purpose I've used django-maintenance-mode's package, but, as I said, due to the implicit stateless character of GC Run, the enviroment variable that stores the maintenance mode's value drops when there isn't any active instance, getting back to OFF.
I'd like to know any other possible solution or fix overriding any of this package's methods to make it work in my project.
Thanks in advance!!
You can use the Graceful shutdowns which will allow you to capture the environment variable that stores the maintenance mode value. Once the value is captured you can store it in a database (Cloud SQL) or in a file on Cloud Storage. At each startup, you get the last value.
I am testing my app for concurrency
I am using django-concurrency
For every model added
from concurrency.fields import IntegerVersionField
version = IntegerVersionField( )
So i open with 2 browser windows my app. log with 2 different users and the edit with both same record . Save with one then I save with second one and always second one just overwrites the first one . Basically does nothing .
Expected to get an error on my second save.
What could be the issue ?
I have never used Django concurrency and I would run a mile if I run across apps that offer to abstract concurrency, transaction management etc. Based on general concepts of concurrency. This behaviour seems right. There's no error here.
django-concurrency is an optimistic lock [1] implementation for
Django.
Supported Django versions: 1.8.x, 1.9.x, 1.10.x., 1.11.x
It prevents users from doing concurrent editing in Django both from UI
and from a django command.
What this actually means is that two users cannot modify the same instance at the same time. In adhoc two browser testing of the sort that's just tried out, such a situation can never be produced! What usually happens is that one or the other transaction usually runs to completion because when testing on small databases on localhost, there is no latency and everything happens instantly.
The proper way to test this is to open up to separate change shells, open up the same instance and edit as outlined in the django concurrency guide. Or test with a multi threaded client.
I'm looking to access control example. https://github.com/strongloop/loopback-example-access-control
It says we need to put sample-models.js file under server/boot folder. That means, everytime I run the application, the creation process will be made again and again. Of course, I'm getting errors on the second call.
Should I put my own mechanism to disable if ones it run, or is there a functionality in loopback?
Bot scripts are for setting up the application. And run once per application start.
So if you want to initialize database or any initializing which would be persisted by running boot script, you need to check if it is initialized first or not.
For example for initializing roles in db, you need to check if there is desired roles in db or not. And if there is not, so create ones.
There is no other functionality in loopback for this.
I know there are many questions asking about this, especially this one: Django - Set Up A Scheduled Job?.
But what I want to understand is, how does a scheduled task inside Django actually works?
My simplistic way to think about it is that there's an infinite loop somewhere, something like this (runs every 60 seconds),
import time
interval=60 #60 seconds
while True:
some_method()
time.sleep(interval)
Question: where do you put this infinite loop? Is there some part of the Django app that just runs in the background alongside the rest of the app?
Thanks!
Django doesn't do scheduled tasks. If you want scheduled tasks, you need a daemon that runs all the time and can launch your task at the appropriate time.
Django only runs when a http request is made. If no one makes a http request for a month, django doesn't run for a month. If there are 45 http requests this second, django will run 45 times this second (in the absence of caching).
You can write scripts in the django framework (called management commands) that get called from some outside service (like cron). That's as close as you'll get to what you want. If that's the case, then the question/answer you reference is the place to get the how tos.
Probably on a unixy system, cron is the simplest outside service to work with. On recent linux systems, cron has a directory /etc/cron.d into which you can drop your app's cron config file, and it will not interfere with any other cron jobs on the system. No editing of existing files necessary.
I have a program that classifies text and would like to make it interactive with a user on the front-end of my django site. The problem is it takes 20 seconds for the program to load the training set and get going, and that's not feasible every time someone enters input.
Instead, I'd like Django to load the program once when the server starts, and have all user input interact with it via a view.
I looked at launching subprocesses, but if I'm not mistaken, a subprocess would only be launched when a view is called and that is undesirable for this.
Any ideas?
Thanks.
It's possible that Celery would be appropriate here. There is Django integration available with django-celery.
As Jim noticed celery is one of the best options you have for asynchronus task management, but if you want to avoid celery & its dependecies overhead you could just add a status field on the model the process takes place (e.g. text_processed boolean field with default=False) and create an application management command which would handle the process of the created db entries.
Add the command on a cron and you are done.