Im trying to create a synchronization between [Unfuddle][1] and my local server so that when there is a SVN commit on Unfuddle, my local server's code automatically gets updated. Unfuddle provides a callback and I can get the callback when a commit is made.
Now my webserver runs as the user called "apache", but all the files that are existing are owned by a "development" user. This is the ftp user that my team uses to update code everyday. I want to make it so that my callback will write the files as the "development" user.
These are the ones Ive tried so far to make this work
I cant use sudo -u development as that means I have to relax running "sudo" command for apache.
Changing file permissions to 777 works, but obviously, thats not something that I want to do.
The only option that I can think of now is run a standalone Python HTTP Server and make it run as the "development" user and make that the callback url. That way, I can update the local working copy as "development" user and my other folks can also use it as they were doing it before.
Are there any other ways of doing this?
Thanks!
For personal use, I'd use the Python HTTP Server approach, but if you're happy with Apache -- I imagine a script will then run to getch fetch things from Unfuddle svn into local. In that case, try mod_suexec, which runs scripts as a different user.
Have you considered putting the two users in the same group, and giving permission for the group?
Related
I want to work locally on my django(1.7) project and regularly deploy updates to a production server. How would you do this? I have not found anything about this in the docs. I am confused about that because it seems like many people would want to do that and there should be some kind of standard solution to this. Or am I getting the whole workflow wrong?
I should note that I'm not expecting a step-by-step guide. I am just trying to understand the concept.
Assuming you already have your deployment server setup, and all you need to do is push code to your server, then you can just use git as a form of deployment.
Digital Ocean has a good tutorial at this link https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps
Push sources to a git repository from a dev machine.
pull sources on a production server. Restart uwsgi/whatever.
There is no standard way of doing this, so no, it cannot be included with Django or be thoroughly described in the docs.
If you're using a PaaS how you deploy depends on the PaaS. Ditto for a container like docker, you must follow the rules of that particular container.
If you're old-school and can ssh into a server you can rsync a snapshot of the code to the correct place after everything else is taken care of: database, ports, webserver setup etc. That's what I do, and I control stuff with bash scripts utilizing a makefile.
REMOETHOST=user#yourbox
REMOTEPATH=yourpath
REMOTE=$REMOTEHOST:$REMOTEPATH
make rsync REMOTE_URI=$REMOTE
ssh $REMOTEHOST make -C $REMOTEPATH deploy
My "deploy"-action is a monster but might be as easy as something that touches the wsgi-file used in order to reload the site. My medium complex ones cleans out stale files, run collectstatic and then reloads the site. The really complex ones creates a timestamped virtualenv, cloned database and remote code tree, a new server-setup that points to this, runs connection tests on the remote and if they succeed, switches the main site to point to the new versioned site, then emails me the version that is now in production, with the git hash and timestamp.
Lots of good solutions. Heroku has a good tutorial: https://devcenter.heroku.com/articles/getting-started-with-django
Check out a general guide for deploying to multiple PaaS providers here: http://www.paascheatsheet.com
I'm trying to use PostgreSQL in my application to manage some records.
With SQLite, nothing complicated was needed because it's a file db.
However, with PostgreSQL, some settings should be done before use it even if I want to connect localhost or local socket file.
Of course, initialization is only required once but the problem is, I don't want to bother any use of my application with this system setting problem.
I want to make it possible for users to just run and use my application out-of-box state.
I've read a wiki article to use PostgreSQL in Arch Linux(https://wiki.archlinux.org/index.php/PostgreSQL), and no one would like to do these things to use my application.
Furthermore, it's not possible to run those commands in wiki from application because the specific settings are depends on distros and also requires root-privileges.
Is there any way to connect PostgreSQL without such complicated initialization, like SQLite? Or, simple way to make the user's system prepared?
Just for your information, my application is written in C++ and Qt.
And, I cannot use SQLite for its limitations.
If you want an embedded database PostgreSQL isn't a great choice. It's usable, but it's a bit clunky compared to DBs designed for embedding like Firebird, SQLite, etc.
Bundling PostgreSQL
If you want to use PostgreSQL, I suggest bundling PostgreSQL binaries with your program, and starting a PostgreSQL server up when your program is used - assuming you only need a single instance of your program at a time. You can get handy pre-built Pg binaries from EDB. You can offer your users a choice on startup - "Use existing PostgreSQL" (in which case they must create the user and db themselves, make any pg_hba.conf changes, etc) or "Start private PostgreSQL server" (when you run your own using the binaries you bundled).
If they want to use an existing DB, all you need to do is tell them to enter the host / socket_directory, database name, port, username, and password. Don't make "password" required, it might be blank if their hba configuration doesn't require one.
If they want to use a private instance, you fire one up - see the guidance below.
Please do not bundle a PostgreSQL installer and run it as a silent install. This is a nightmare for the poor user, who has no idea where this "postgresql" thingy came from and is likely to uninstall it. Or, worse, it'll conflict with their own PostgreSQL install, or confuse them when they go to install PostgreSQL themselves later, finding that they already have it and don't even know the password. Yeah. Don't do that. Bundle the binaries only and control your private PostgreSQL install yourself.
Do not just use the postgres executables already on the system to make your own private instance. If the user decides to upgrade from 9.2 to 9.3, suddenly your private instance won't start up, and you won't have access to the old 9.2 binaries to do a pg_upgrade. You need to take full responsibility if you're using a private instance of Pg and bundle the binaries you need in your program.
How to start/control a private instance of Pg
On first run, you run PostgreSQL's initdb -D /path/to/datadir, pointing at an empty subdir of a private data directory for your program. Set some environment variables first so you don't conflict with any normal PostgreSQL install on the system. In particular, set PGPORT to some random high-ish port, and specify a different unix_socket_directory configuration parameter to the default.
Once initdb has run, your program will probably want to modify postgresql.conf and pg_hba.conf to fit its needs. Personally I don't bother, I just pass any parameters I want as overrides on the PostgreSQL server start-up command line, including overriding the hba_file to point to one I have pre-created.
If the data dir already exists during program startup you then need to check whether the datadir matches the current PostgreSQL major version by examining the PG_VERSION file. If it doesn't, you need to make a copy of the datadir and run pg_upgrade to upgrade it to the current version you have bundled. You need to retain a copy of the binaries for the old version for this, so you'll need to special-case your update processes or just bundle old versions of the binaries in your update installer as well as the new ones.
When your program is started, after it's checked that the datadir already exists and is the correct version, set the PGPORT env var to the same value you used for initdb then start PostgreSQL directly with postgres -D /path/to/datadir and suitable parameters for log file output, etc. The postmaster you start will have your program as the parent executable, and will be terminated if your program quits suddenly. That's OK, it'll get time to clean up, and even if it doesn't PostgreSQL is crash-safe by design. Your program should still politely ask PostgreSQL to shut down before exiting by sending an appropriate signal, though.
Your application can now connect to the PostgreSQL instance it owns and controls using libpq or whatever, as normal, by specifying the port you're running it on and (if making a unix socket connection) passing host as /path/to/whatever/unix/socket/dir.
Instead of directly controlling postgres you might instead choose to use pg_ctl to drive it. That way you can leave the database running when your program exits and only start it if you find it's not already running. It doesn't really matter if the user shuts the system down without shutting down PostgreSQL - Pg will generally get some shutdown warning from the operating system, but doesn't need it or care much about it, it's quite happy to just crash and recover when next started up.
It is definitily not as easy as using SQLite.
In SQLite you have only a single file which contains the whole database and if you "connect" to the database you simply open the file.
In PostgreSQL, MySQL, ... you have a database daemon which keeps lots of files open, you have multiple files for the database itself, then transaction logs, log files .. and of course the configuration files which specifify where all these files are, resource usage, how to connect and much more. And the database needs also be maintained regularly for optimal performance and there are lots of tunables. That's all because the usual use of all these databases is to serve multiple clients at once as fast as possible and not to make setup simple.
It is still possible to setup PostgreSQL just for a single user. To setup the database you need to create a database directory, find an unused port on localhost for TCP connections or better use UNIX domain sockets, write the config with the needed parameters (like the used socket, database directory, user, resource restrictions, permissions so that no other users on the same machine can connect), start the database daemon, initialize the database(s) (postgresql can serve multiple databases within the same daemon setup) and finally connect to the database from your program. And don't forget to regularly run the maintainance tasks and to shutdown the database in an ordered way.
I'm interested in figuring out the best practice way of organising Django apps on a server.
Where do you place Django code? The (old now) Almanac says /home/django/domains/somesitename.com/ but I've also seen things placed in /opt/apps/somesitename/ . I'm thinking that the /opt/ idea sounds better as it's not global, but I've not seen opt before, and presumably it might be better for apps to go in a site specific deployer users home dir.
Would you recommend having one global deployer user, one user per site, or one per site-env (eg, sitenamelive, sitenamestaging). I'm thinking one per site.
How do you version your config files? I currently put them in an /etc/ folder at top level of source control. eg, /etc/nginc/somesite-live.conf.
How do you do provision your servers and do the deployment? I've resisted Chef and Puppet for years on the hope of something Python based. Silver Lining doesn't seem ready yet, and I have big hopes for Patchwork (https://github.com/fabric/patchwork/). Currently we're just using some custom Fabric scripts to deploy, but the "server provisioning" is handled by a bash script and some manual steps for adding keys and creating users. I'm about to investigate Silk Deployment (https://bitbucket.org/btubbs/silk-deployment) as it seems closest to our setup.
Thanks!
I think there would have to be more information on what kinds of sites you are deploying: there would be differences based on the relations between the sites, both programatically and 'legally' (as in a business relation):
Having an system account per 'site' can be handy if the sites are 'owned' by different people - if you are a web designer or programmer with a few clients, then you might benefit from separation.
If your sites are related, i.e. a forum site, a blog site etc, you might benefit from a single deployment system (like ours).
for libraries, if they're hosted on reputable sources (pypy, github etc), its probably ok to leave them there and deploy from them - if they're on dodgy hosts which are up or down, we take a copy and put them in a /thirdparty folder in our git repo.
FABRIC
Fabric is amazing - if its setup and configured right for you:
We have a policy here which means nobody ever needs to log onto a server (which is mostly true - there are occasions where we want to look at the raw nginx log file, but its a rarity).
We've got fabric configured so that there are individual functional blocks (restart_nginx, restart_uwsgi etc), but also
higher level 'business' functions which run all the little blocks in the right order - for us to update all our servers we meerly type 'fab -i secretkey live deploy' - the live sets the settings for the live servers, and deploy ldeploys (the -i is optional if you have your .ssh keys set up right)
We even have a control flag that if the live setting is used, it will ask 'are you sure' before performing the deploy.
Our code layout
So our code base layout looks a bit like this:
/ <-- folder containing readme file etc
/bin/ <-- folder containing nginx & uwsgi binaries (!)
/config/ <-- folder containing nginx config and pip list but also things like pep8 and pylint configs
/fabric/ <-- folder containing fabric deployment
/logs/ <-- holding folder that nginx logs get written into (but not committed)
/src/ <-- actual source is in here!
/thirdparty/ <-- third party libs that we didn't trust the hosting of for pip
Possibly controversial because we load our binaries into our repo, but it means that if i upgrade nginx on the boxes, and want to roll back, i just do it by manipulation of git. I know what works against what build.
How our deploy works:
All our source code is hosted on a private bitbucket repo (we have a lot of repos and a few users, thats why bitbucket is better for us then github). We have a user account for the 'servers' with its own ssh key for bitbucket.
Deploy in fabric performs the following on each server:
irc bot announce beginning into the irc channel
git pull
pip deploy (from a pip list in our repo)
syncdb
south migrate
uwsgi restart
celery restart
irc bot announce completion into the irc channel
start availability testing
announce results of availability testing (and post report into private pastebin)
The 'availability test' (think unit test, but against live server) - hits all the webpages and API's on the 'test' account to make sure it gets back sane data without affecting live stats.
We also have a backup git service so if bitbucket is down, it falls over to that gracefully, and we even have jenkins integration that on a commit to the 'deploy' branch, it causes the deployment to go through
The scary bit
Because we use cloud computing and expect a high throughput, our boxes auto spawn. Theres a default image which contains a a copy of the git repo etc, but invariably it will be out of date, so theres a startup script which does a deployment to itself, meaning new boxes added to the cluster are automatically up-to-date.
I have few simple django-based sites and I their number increasing all the time. Every time I deploy the site I need to:
Manually create bash-script that start Django FastCGI server.
Adding it to etc/init.d to run after server reboot.
Creating separate config for Lighttpd to work with FastCGI server and serving static files.
I know how to do it, but I'd like to automate this task if possible.
My dream setup process could look like this:
I have a folder somewhere in my /var/ directory. For example: /var/django/
I clone one of my projects to the subdirectory of this directory.
After that happening one of the following: Some software automatically detects folder creation, and creates all necessary configs and then restart Lighttpd. OR I manually run some kind of script in my new folder to do it.
I tried to look for existing tools for such automation or something similar in the internet, but couldn't find one.
So I'd like to ask is there tools like this out there? Maybe not exactly to install Django apps, but to this kind of process automation in general. Or everybody just writes their own bash script to do such things?
have you had a look at fabric and puppet?
I think fabric will do the job. I've just started reading through the docs, seems very simple to get started on. Also it has nice Python-ic way of doing things locally and on remote servers.
I asked a previous question getting a django command to run on a schedule. I got a solution for that question, but I still want to get my commands to run from the admin interface. The obstacle I'm hitting is that my custom management commands aren't getting recognized once I get to the admin interface.
I traced this back to the __init__.py file of the django/core/management utility. There seems to be some strange behavior going on. When the server first comes up, a dictionary variable _commands is populated with the core commands (from django/core/management/commands). Custom management commands from all of the installed apps are also pushed into the _commands variable for an overall dictionary of all management commands.
Somehow, though between when the server starts and when django-chronograph goes to run the job from the admin interface, the _commands variable loses the custom commands; the only commands in the dictionary are the core commands. I'm not sure why this is. Could it be a path issue? Am I missing some setting? Is it a django-chronograph specific problem? So forget scheduling. How might I run a custom management command from the django admin graphical interface to prove that it can indeed be done? Or rather, how can I make sure that custom management commands available from said interface?
I'm also using django-chronograph
and for me it works fine. I did also run once into the problem that my custom commands were not regognized by the auto-discovery feature. I think the first reason was because the custom command had an error in it. Thus it might be an idea to check whether your custom commands run without problems from the command line.
The second reason was indeed some strange path issue. I might check back with my hosting provider to provide you with a solution. Will post back to you in a few days..
i am the "unix-guy" mentioned above by tom tom.
as far as i remember there were some issues in the cronograph code itself, so it would be a good idea to use the code tom tom posted in the comments.
where on the filesystem is django-cronograph stored (in you app-folder, in an extra "lib-folder" or in your site-packages?
when you have it in site-packages or another folder that is in your "global pythonpath" pathing should be no issue.
the cron-process itself DOES NOT USE THE SAME pythonpath, as your django app. remember: you start the cron-process via your crontab - right? so there are 2 different process who do not "know" each other: the cron-process AND the django-process (initialized by the webserver) so i would suggest to call the following script via crontab and export pythonpath again:
#!/bin/bash
PYTHONPATH=/path/to/libs:/path/to/project_root:/path/to/other/libs/used/in/project
export PYTHONPATH
python /path/to/project/manage.py cron
so the cron-started-process has the same pythonpath-information as your project.
greez from vienna/austria
berni