So right now I have a global counter variable. And it updates whenever a message is sent. Running my program locally I get no weirdness, but on heroku, sometimes the variable just bumps down to zero if I reload the page. Reload it again, its back to a number.
I don't know why that is happening. It only happens on heroku. What am I doing wrong? Is there a better way to do this?
https://github.com/sharkwheels/sendCatsWeb/blob/master/app.py
Thanks!
Your counter variable is running in-memory only.
What this means is that if your Python process stops running (restarts, crashes, whatever) -- the next time this web server boots up, that counter will be reset back to 0 (the default value).
When you run any programs on Heroku, they will restart frequently. This happens randomly, and for many reasons:
Maybe Heroku had to move your application to another web server due to load issues.
Maybe your process crashes, and Heroku restarted it.
Maybe Heroku created another instance of your web app to handle increased load.
This stuff can happen for many reasons.
Regardless -- you should NEVER expect Heroku to run your webapp continuously. Always expect your process to be restarted. This is a best practice in the web development community.
Your application should ideally function something like this:
When visitors visit your page, you send an increment request to a database counter of some sort that is persisted permanently in some form of database.
This way, regardless of how often your web application reboots, your data is never lost.
Related
I'm using Django 1.10 with SQLite as my database back-end.
I have a site running with nginx/uwsgi and the above configuration, which I'm constantly updating with new code.
Each time I want to update the site code, I'm shutting down uwsgi, nginx, git pull ing the new version from my repo, then restarting uwsgi and nginx.
While this works, in the sense that the site is updated with the new version of my code, the side-effect is that in the event that a user is currently logged in the site and working on something (which generally will result in modifying the db), the user's work will be interrupted.
Worse still would be the case that the new version of my code contains migrations, which will change the db structure, with (unpredictable?) consequences on the user's on-going work.
So the question is: is there a way, like a command-line script, to check db.sqlite3 and see if a user is currently logged-in, before deciding to shutdown uwsgi and nginx ?
I would say NO. There is chances like
User logged in and didn't log out for a while.
Logged in, but inactive
We can get logged in users who didnt logout in django_session table.
A better approach would be to minimize the downtime. As a start, there's no reason to restart nginx when you update your code or uwsgi, and you can git pull without affecting the currently running code, so you don't need to stop uwsgi before pulling in the new code.
When it comes to migrations, try to avoid any migrations that will break the currently running code. For example, don't delete fields/models that are still used by the current code, but wait until the next update (when those fields/models are no longer in use). That way you can run your migrations while you're still running the old code, without generating any errors.
Next up, you should reload the uwsgi process rather than stopping and starting it manually. This will finish handling any open requests before reloading the process. It will also keep on listening for new requests, so that these can be handled immediately after the process has reloaded. Users may experience a slowdown, but this will not drop any requests unless the queue fills up before the process can be reloaded. For small-scale websites this should never happen.
So, to avoid downtime you can use this to update your website:
$ git pull # pull in new code while the old process is still running
$ python manage.py migrate # run your migrations, and possibly `collectstatic` etc.
$ kill -HUP `cat <pidfile>` # gracefully reload the uwsgi process
In the near future, I'll be deploying a Django/Gunicorn/Nginx website to paying customers. This is my first public website. There will be times when I'll need to bring the site down temporarily for maintenance. I've learned how to configure Nginx to serve up a "503 Site Temporarily Unavailable" page while I'm doing my maintenance and I'm set to inform my customers in advance via an email.
However, if I have a critical problem and need to change a setting or view or something and I need to restart the site without any advance warning by restarting Gunicorn, what will happen from the customer's point of view? Will the site just slow down for a couple of seconds? Will their sessions remain intact or will all sessions be lost? Clearly there will be database problems if any writes are occuring. What things are going to happen and are there any precautions I can take to limit the damage?
As long as you don't restart gunicorn by physically cutting the power to the server, the OS will give it the chance to shut down normally. That means that all current views will complete and all open transactions will finish.
Assuming you are using a reverse proxy like nginx in front, and you start gunicorn again immediately, users will be unlikely to notice: the request might take a fraction of a second longer, that's all.
Sessions are stored in files or the database so will be unaffected.
I have an app on heroku, running on Puma:
workers 2
threads_count 3
pool 5
It looks like some requests get stuck in the middleware, and it makes the app very slow (VERY!).
I have seen other people threads about this problem but no solution so far.
Please let me know if you have any hint.
!
!
I work for Heroku support and Middleware/Rack/ActiveRecord::QueryCache#call is a commonly reported as a problem by New Relic. Unfortunately, it's usually a red herring as each time the source of the problem lies elsewhere.
QueryCache is where Rails first tries to check out a connection for use, so any problems with a connection will show up here as a request getting 'stuck' waiting. This doesn't mean the database server is out of connections necessarily (if you have Librato charts for Postgres they will show this). It likely means something is causing certain database connections to enter a bad state, and new requests for a connection are waiting. This can occur in older versions of Puma where multiple threads are used and the reaping_frequency is set - if some connections get into a bad state and the others are reaped this will cause problems.
Some high-level suggestions are as follows:
Upgrade Ruby & Puma
If using the rack-timeout gem, upgrade that too
These upgrades often help. If not, there are other options to look into such as switching from threads to worker based processes or using a Postgres connection pool such as PgBouncer. We have more suggestions on configuring concurrent web servers for use with Postgres here: https://devcenter.heroku.com/articles/concurrency-and-database-connections
I will answer my own question:
I simply had to check all queries to my DB. One of them was taking a VERY long time, and even if it was not requested often, it would slow down the whole server for quite some time afterwards(even after the process was done, there was a sort of "traffic jam" on the server).
Solution:
Check all the queries to your database, fix the slowest ones (it might simply mean breaking it down in few steps, it might mean make it run at night when there is no traffic, etc...).
Once this queries are fixed, everything should go back to normal.
I recently started seeing a spike in time spent in ActiveRecord::QueryCache#call. After looking at the source, I decided to try clearing said cache using ActiveRecord::Base.connection.clear_query_cache from a Rails Console attached to the production environment. The error I got back was PG::ConnectionBad: could not fork new process for connection: Cannot allocate memory which lead me to this other SO question at least Heroku Rails could not fork new process for connection: Cannot allocate memory
I inherited a coldfusion MX7 application that has a long running process, which the user kicked by accident.
By my calculations, at the current rate, the job will run for 3 days.
There doesnt seem to be a way through the administrator interface to stop a job that is running.
The table that is being filled can be easily repopulated, so I would think stopping the coldfusion service wont effect anything except the table, which isnt a problem.
Am I right? Is that safe? Is there another way?
a one-time restart of the service should be fine. for the future, you may want to add a required url param or other such safety mechanism to keep this long process from accidentally going off.
Check to see if the task already has an explicit timeout out set
Explicit Timeout
Otherwise the default page time out is used
Server Settings
For newer versions of ColdFusion 8 and above, you can kill a running page with with the Server Monitor in the section labeled "Aborting unresponsive or troublesome requests/threads"
Using server monitor.
It also may be possible to stop the processing by killing the SQL Server Task:
Is there a task manager of sorts for SQL Server 2008 and on?
Ok, strange setup, strange question. We've got a Client and an Admin web application for our SaaS app, running on asp.net-2.0/iis-6. The Admin application can change options displayed on the Client application. When those options are saved in the Admin we call a Webservice on the Client, from the Admin, to flush our cache of the options for that specific account.
Recently we started giving our Client application >1 Worker Processes, thus causing the cache of options to only be cleared on 1 of the currently running Worker Processes.
So, I obviously have other avenues of fixing this problem (however input is appreciated), but my question is: is there any way to target/iterate through each Worker Processes via a web request?
I'm making some assumptions here for this answer....
I'm assuming the client app is using one of the .NET caching classes to store your application's options?
When you say 'flush' do you mean flush them back to a configuration file or db table?
Because the cache objects and data won't be shared between processes you need a mechanism to signal to the code running on the other worker process that it needs to re-read it's options into its cache or force the process to restart (which is not exactly convenient and most likely undesirable).
If you don't have access to the client source to modify to either watch the options config file or DB table (say using a SqlCacheDependency) I think you're kinda stuck with this behaviour.
I have full access to admin and client, by cache, I mean .net's Cache object. By flush I mean removing the item from the Cache object.
I'm aware that both worker processes don't share the cache data. That's sort of my conundrum)
The system is the way it is to remove the need to hit sql every new-session that comes in. So I'm trying to find a solution that can just tell each worker process that the cache needs to be cleared w/o getting sql involved.