Merging an independent Git repo with another Git repo that is a conduit with Subversion: avoiding duplication when merging - django

I am happily developing a Django project in my own Git repo on my localhost. I am creating branches, committing and merging happily. The path is something like:
/path/to/git/django/
and the structure is:
project
├── README
├── REQUIREMENTS
├── __init__.py
├── fabfile.py
├── app1
├── manage.py
├── app2
├── app3
├── app4
└── project
The rest of my development team still use Subversion, which is one giant repo with multiple projects. When I am working with that on my localhost I am still using Git (via git-svn). The path is something like
/path/to/giant-svn-repo/
Projects live under this like:
giant-svn-repo
|── project1
|── project2
|── project3
└── project4
When I want to work with the latest changes from the remote svn repo I just do a git-svn rebase. Then for new features I create a new branch, develop, commit, checkout master, merge branch with master, delete branch, and then a final git-svn dcommit. Cool. Everything works well.
These two repositories (lets call them git-django and git-svn) are completely independent right now.
Now I want to add git-django into the git-svn repo as a new project (ie. in a child directory called djangoproject). I have this working pretty well, using the following workflow:
cd into git-svn repo
Create a new branch in the git-svn repo
Make a new directory to host my django project
Add a new remote that links to my original Django project
Merge the remote into my local directory
Read-tree with the prefix of relative path djangoproject so it puts the codebase into the correct location based on the root of git-svn repo
Commit the changes so everything gets dumped into the correct place
From the command line this looks like:
> cd /path/to/giant-svn-repo
> git checkout -b my_django_project
> mkdir /path/to/giant-svn-repo/djangoproject
> git remote add -f local_django_dev /path/to/git/django/project
> git merge -s ours --no-commit local_django_dev/master
> git read-tree --prefix=djangoproject -u local_django_dev/master
> git commit -m 'Merged local_django_dev into subdirectory djangoproject'
This works, but in addition to the contents of the django git repo being in /path/to/giant-svn-repo/djangoproject it is also in the main root of the repository tree!
project
├── README
├── REQUIREMENTS
├── __init__.py
├── fabfile.py
├── djangoproject
│   ├── README
│   ├── REQUIREMENTS
│   ├── __init__.py
│   ├── fabfile.py
│   ├── app1
│   ├── manage.py
│   ├── app2
│   ├── app3
│   ├── app4
│   └── project
├── app1
├── manage.py
├── app2
├── app3
├── app4
└── project
I seem to have polluted the parent directory where all the projects of the giant-svn-repo are located.
Is there any way I can stop this happening?
(BTW this has all been done in a test directory structure - I haven't corrupted anything yet. Just trying to figure out the best way to do it)
I am sure it is just (re)defining one more argument to either git merge, git read-tree or git commit but I am pretty much at my limit of git kung-fu.
Thanks in advance.

Related

Deploy private github repo golang app on elasticbeanstalk

I have been struggling trying to deploy my Golang app to AWS EB for couple days.
I am trying to deploy my app on an EB server Preconfigured Docker - Go 1.4 running on 64bit Debian/2.9.2 using the eb cli via the command eb deploy in my app folder.
After couple seconds, I've got an error message saying that my app wasn't deployed because of an error.
Looking at the eb-activity.log, here what I can see:
/var/log/eb-activity.log
-------------------------------------
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag main.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/randomuser/private-repo (download)
# cd .; git clone https://github.com/randomuser/private-repo /go/src/github.com/randomuser/private-repo
Cloning into '/go/src/github.com/randomuser/private-repo'...
fatal: could not read Username for 'https://github.com': No such device or address
package github.com/Sirupsen/logrus
imports golang.org/x/crypto/ssh/terminal
imports golang.org/x/sys/unix
imports github.com/randomuser/private-repo/apis: exit status 128
package github.com/Sirupsen/logrus
imports golang.org/x/crypto/ssh/terminal
imports golang.org/x/sys/unix
imports github.com/randomuser/private-repo/app
imports github.com/randomuser/private-repo/app
imports github.com/randomuser/private-repo/app: cannot find package "github.com/randomuser/private-repo/app" in any of:
/usr/src/go/src/github.com/randomuser/private-repo/app (from $GOROOT)
/go/src/github.com/randomuser/private-repo/app (from $GOPATH)
I suppose there is an issue when the server tries to install the app, it seems it's trying to retrieve from my private repo on github ...
I referenced my app sub packages as github.com/randomuser/private-repo/subpackage I supposed this is why it behaves like that.
Is there a way to deploy all my code, forcing my private repo to be populated within the GOROOT src/github.com/randomuser/private-repo/ so the server doesn't have to try to get it?
I didn't found any proper example from Amazon docs (multi-packages apps) nor from Github.
Am I missing anything? Is there a better solution?
On a side note, I also tried to deploy my compiled binary directly (create a folder where I put only the binary, zip it and upload it on the ebs env) but it didn't worked neither ... Maybe this option requires yet another env config (if so, which one?).
Thanks for your help :)
Configuration
Golang app having the following folders:
├── Dockerfile
├── server.go
├── Gopkg.lock
├── Gopkg.toml
├── Makefile
├── apis
│   ├── auth.go
│   ├── auth_test.go
│   ├── ...
├── app
│   ├── config.go
│   ├── init.go
│   ├── logger.go
│   ├── scope.go
│   ├── transactional.go
│   └── version.go
├── config
│   ├── dev.app.yaml
│   ├── errors.yaml
│   └── prod.app.yaml
├── daos
│   ├── auth.go
│   ├── auth_test.go
│   ├── ...
├── errors
│   ├── api_error.go
│   ├── api_error_test.go
│   ├── errors.go
│   ├── errors_test.go
│   ├── template.go
│   └── template_test.go
├── models
│   ├── identity.go
│   ├── ...
├── services
│   ├── auth.go
│   ├── auth_test.go
│   ├── ...
├── util
│   ├── paginated_list.go
│   └── paginated_list_test.go
Here is the content of my server.go
package main
import (
"flag"
"fmt"
"net/http"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
"github.com/randomuser/private-repo/apis"
"github.com/randomuser/private-repo/app"
"github.com/randomuser/private-repo/daos"
"github.com/randomuser/private-repo/errors"
"github.com/randomuser/private-repo/services"
)
func main() {
// getting env from command line
// env is either prod, preprod or dev
// by default, env is prod
env := flag.String("env", "prod", "environment: prod, preprod or dev")
flag.Parse()
...
router.To("GET,HEAD", "/ping", func(c *routing.Context) error {
c.Abort() // skip all other middlewares/handlers
return c.Write("OK " + app.Version)
})
...
// Serve on port 5000
My Dockerfile content:
FROM golang:1.4.2-onbuild
ADD . /go/src/github.com/randomuser/private-repo
RUN go install github.com/randomuser/private-repo
EXPOSE 5000
ENTRYPOINT /go/bin/private-repo
I finally managed to make it works.
So I created a brand new eb app (without docker).
I then figured out that my app wasn't able to retrieve env vars set in the console somehow ...
So what I did is that I forced the env variables to be passed to my app at startup time from my production config file using the build.sh script see below :
#!/bin/bash -xe
# See http://tldp.org/LDP/abs/html/options.html
# -x -> Print each command to stdout before executing it, expand commands
# -e -> Abort script at first error, when a command exits with non-zero status
# (except in until or while loops, if-tests, list constructs)
# $GOPATH isn't set by default, nor do we have a usable Go workspace :'(
GOPATH="/var/app/current"
APP_BUILD_DIR="$GOPATH/src/to-be-defined" # We will build the app here
APP_STAGING_DIR="/var/app/staging" # Current directory
DEP_VERSION="v0.3.2" # Use specific version for stability
ENV_VAR_PREFIX="TO_BE_DEFINED_"
# Install dep, a Go dependency management tool, if not already installed or if
# the version does not match.
if ! hash dep 2> /dev/null ||\
[[ $(dep version | awk 'NR==2{print $3}') != "$DEP_VERSION" ]]; then
# /usr/local/bin is expected to be on $PATH.
curl -L \
-s https://github.com/golang/dep/releases/download/$DEP_VERSION/dep-linux-amd64 \
-o /usr/local/bin/dep
chmod +x /usr/local/bin/dep
fi
# Remove the $APP_BUILD_DIR just in case it was left behind in a failed build.
rm -rf $APP_BUILD_DIR
# Setup the application directory
mkdir -p $APP_BUILD_DIR
# mv all files to $APP_BUILD_DIR
# https://superuser.com/questions/62141/how-to-move-all-files-from-current-directory-to-upper-directory
mv * .[^.]* $APP_BUILD_DIR
cd $APP_BUILD_DIR
# Pull in dependencies into vendor/.
dep ensure
# Build the binary with jsoniter tag.
go build -o application -tags=jsoniter .
# Modify permissons to make the binary executable.
chmod +x application
# Move the binary back to staging dir.
# Along with the configuration files.
mkdir $APP_STAGING_DIR/bin
# By default, `bin/application` is executed. This way, a Procfile isn't needed.
mv application $APP_STAGING_DIR/bin
cp -r config $APP_STAGING_DIR
# TODO: Fix the viper not working with env var
# Generate prod config from env variables
/opt/elasticbeanstalk/bin/get-config environment --output YAML | sed s/${ENV_VAR_PREFIX}//g > $APP_STAGING_DIR/config/prod.app.yaml
# Copy .ebextensions back to staging directory.
# cp -r .ebextensions $APP_STAGING_DIR
# Clean up.
rm -rf $APP_BUILD_DIR
echo "Build successful!!"
My build.sh file is called by EBS using this Buildfile:
make: ./build.sh
Et voilà ! Everything works properly now :)

Ideal Architecture for Django Projects [duplicate]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I know there is actually no single right way. However I've found that it's hard to create a directory structure that works well and remain clean for every developer and administrator. There is some standard structure in most projects on github. But it does not show a way to organize another files and all projects on pc.
What is the most convenient way to organize all these directories on development machine? How do you name them, and how do you connect and deploy this to server?
projects (all projects that your are working on)
source files (the application itself)
working copy of repository (I use git)
virtual environment (I prefer to place this near the project)
static root (for compiled static files)
media root (for uploaded media files)
README
LICENSE
documents
sketches
examples (an example project that uses the application provided by this project)
database (in case sqlite is used)
anything else that you usually need for successful work on project
The problems that I want to solve:
Good names of directories so that their purpose is clear.
Keeping all project files (including virtualenv) in one place, so I can easily copy, move, archive, remove whole project or estimate disk space usage.
Creating multiple copies of some selected file sets such as entire application, repository or virtualenv, while keeping single copy of another files that I don't want to clone.
Deploying right set of files to the server simply by rsyncing selected one dir.
There're two kind of Django "projects" that I have in my ~/projects/ directory, both have a bit different structure.:
Stand-alone websites
Pluggable applications
Stand-alone website
Mostly private projects, but doesn't have to be. It usually looks like this:
~/projects/project_name/
docs/ # documentation
scripts/
manage.py # installed to PATH via setup.py
project_name/ # project dir (the one which django-admin.py creates)
apps/ # project-specific applications
accounts/ # most frequent app, with custom user model
__init__.py
...
settings/ # settings for different environments, see below
__init__.py
production.py
development.py
...
__init__.py # contains project version
urls.py
wsgi.py
static/ # site-specific static files
templates/ # site-specific templates
tests/ # site-specific tests (mostly in-browser ones)
tmp/ # excluded from git
setup.py
requirements.txt
requirements_dev.txt
pytest.ini
...
Settings
The main settings are production ones. Other files (eg. staging.py,
development.py) simply import everything from production.py and override only necessary variables.
For each environment, there are separate settings files, eg. production,
development. I some projects I have also testing (for test runner), staging
(as a check before final deploy) and heroku (for deploying to heroku) settings.
Requirements
I rather specify requirements in setup.py directly. Only those required for
development/test environment I have in requirements_dev.txt.
Some services (eg. heroku) requires to have requirements.txt in root directory.
setup.py
Useful when deploying project using setuptools. It adds manage.py to PATH, so I can run manage.py directly (anywhere).
Project-specific apps
I used to put these apps into project_name/apps/ directory and import them
using relative imports.
Templates/static/locale/tests files
I put these templates and static files into global templates/static directory, not inside each app.
These files are usually edited by people, who doesn't care about project code
structure or python at all. If you are full-stack developer working alone or
in a small team, you can create per-app templates/static directory. It's really just a matter of taste.
The same applies for locale, although sometimes it's convenient to create separate locale directory.
Tests are usually better to place inside each app, but usually there is many
integration/functional tests which tests more apps working together, so global
tests directory does make sense.
Tmp directory
There is temporary directory in project root, excluded from VCS. It's used to
store media/static files and sqlite database during development. Everything in
tmp could be deleted anytime without any problems.
Virtualenv
I prefer virtualenvwrapper and place all venvs into ~/.venvs directory,
but you could place it inside tmp/ to keep it together.
Project template
I've created project template for this setup, django-start-template
Deployment
Deployment of this project is following:
source $VENV/bin/activate
export DJANGO_SETTINGS_MODULE=project_name.settings.production
git pull
pip install -r requirements.txt
# Update database, static files, locales
manage.py syncdb --noinput
manage.py migrate
manage.py collectstatic --noinput
manage.py makemessages -a
manage.py compilemessages
# restart wsgi
touch project_name/wsgi.py
You can use rsync instead of git, but still you need to run batch of commands to update your environment.
Recently, I made django-deploy app, which allows me to run single management command to update environment, but I've used it for one project only and I'm still experimenting with it.
Sketches and drafts
Draft of templates I place inside global templates/ directory. I guess one can create folder sketches/ in project root, but haven't used it yet.
Pluggable application
These apps are usually prepared to publish as open-source. I've taken example
below from django-forme
~/projects/django-app/
docs/
app/
tests/
example_project/
LICENCE
MANIFEST.in
README.md
setup.py
pytest.ini
tox.ini
.travis.yml
...
Name of directories is clear (I hope). I put test files outside app directory,
but it really doesn't matter. It is important to provide README and setup.py, so package is easily installed through pip.
My answer is inspired on my own working experience, and mostly in the book Two Scoops of Django which I highly recommend, and where you can find a more detailed explanation of everything. I just will answer some of the points, and any improvement or correction will be welcomed. But there also can be more correct manners to achieve the same purpose.
Projects
I have a main folder in my personal directory where I maintain all the projects where I am working on.
Source Files
I personally use the django project root as repository root of my projects. But in the book is recommended to separate both things. I think that this is a better approach, so I hope to start making the change progressively on my projects.
project_repository_folder/
.gitignore
Makefile
LICENSE.rst
docs/
README.rst
requirements.txt
project_folder/
manage.py
media/
app-1/
app-2/
...
app-n/
static/
templates/
project/
__init__.py
settings/
__init__.py
base.py
dev.py
local.py
test.py
production.py
ulrs.py
wsgi.py
Repository
Git or Mercurial seem to be the most popular version control systems among Django developers. And the most popular hosting services for backups GitHub and Bitbucket.
Virtual Environment
I use virtualenv and virtualenvwrapper. After installing the second one, you need to set up your working directory. Mine is on my /home/envs directory, as it is recommended on virtualenvwrapper installation guide. But I don't think the most important thing is where is it placed. The most important thing when working with virtual environments is keeping requirements.txt file up to date.
pip freeze -l > requirements.txt
Static Root
Project folder
Media Root
Project folder
README
Repository root
LICENSE
Repository root
Documents
Repository root. This python packages can help you making easier mantaining your documentation:
reStructuredText
Sphinx
Sketches
Examples
Database
I don't like to create a new settings/ directory. I simply add files named settings_dev.py and settings_production.py so I don't have to edit the BASE_DIR.
The approach below increase the default structure instead of changing it.
mysite/ # Project
conf/
locale/
en_US/
fr_FR/
it_IT/
mysite/
__init__.py
settings.py
settings_dev.py
settings_production.py
urls.py
wsgi.py
static/
admin/
css/ # Custom back end styles
css/ # Project front end styles
fonts/
images/
js/
sass/
staticfiles/
templates/ # Project templates
includes/
footer.html
header.html
index.html
myapp/ # Application
core/
migrations/
__init__.py
templates/ # Application templates
myapp/
index.html
static/
myapp/
js/
css/
images/
__init__.py
admin.py
apps.py
forms.py
models.py
models_foo.py
models_bar.py
views.py
templatetags/ # Application with custom context processors and template tags
__init__.py
context_processors.py
templatetags/
__init__.py
templatetag_extras.py
gulpfile.js
manage.py
requirements.txt
I think this:
settings.py
settings_dev.py
settings_production.py
is better than this:
settings/__init__.py
settings/base.py
settings/dev.py
settings/production.py
This concept applies to other files as well.
I usually place node_modules/ and bower_components/ in the project directory within the default static/ folder.
Sometime a vendor/ directory for Git Submodules but usually I place them in the static/ folder.
As per the Django Project Skeleton, the proper directory structure that could be followed is :
[projectname]/ <- project root
├── [projectname]/ <- Django root
│ ├── __init__.py
│ ├── settings/
│ │ ├── common.py
│ │ ├── development.py
│ │ ├── i18n.py
│ │ ├── __init__.py
│ │ └── production.py
│ ├── urls.py
│ └── wsgi.py
├── apps/
│ └── __init__.py
├── configs/
│ ├── apache2_vhost.sample
│ └── README
├── doc/
│ ├── Makefile
│ └── source/
│ └── *snap*
├── manage.py
├── README.rst
├── run/
│ ├── media/
│ │ └── README
│ ├── README
│ └── static/
│ └── README
├── static/
│ └── README
└── templates/
├── base.html
├── core
│ └── login.html
└── README
Refer https://django-project-skeleton.readthedocs.io/en/latest/structure.html for the latest directory structure.
Here is what I follow on My system.
All Projects: There is a projects directory in my home folder i.e. ~/projects. All the projects rest inside it.
Individual Project: I follow a standardized structure template used by many developers called django-skel for individual projects. It basically takes care of all your static file and media files and all.
Virtual environment: I have a virtualenvs folder inside my home to store all virtual environments in the system i.e. ~/virtualenvs . This gives me flexibility that I know what all virtual environments I have and can look use easily
The above 3 are the main partitions of My working environment.
All the other parts you mentioned are mostly dependent on project to project basis (i.e. you might use different databases for different projects). So they should reside in their individual projects.
You can use https://github.com/Mischback/django-project-skeleton repository.
Run below command:
$ django-admin startproject --template=https://github.com/Mischback/django-project-skeleton/archive/development.zip [projectname]
The structure is something like this:
[projectname]/ <- project root
├── [projectname]/ <- Django root
│ ├── __init__.py
│ ├── settings/
│ │ ├── common.py
│ │ ├── development.py
│ │ ├── i18n.py
│ │ ├── __init__.py
│ │ └── production.py
│ ├── urls.py
│ └── wsgi.py
├── apps/
│ └── __init__.py
├── configs/
│ ├── apache2_vhost.sample
│ └── README
├── doc/
│ ├── Makefile
│ └── source/
│ └── *snap*
├── manage.py
├── README.rst
├── run/
│ ├── media/
│ │ └── README
│ ├── README
│ └── static/
│ └── README
├── static/
│ └── README
└── templates/
├── base.html
├── core
│ └── login.html
└── README

Transform a list of md files to html in Jekyll

I am building a console application that go out to GitHub via octokit and fetch all the matched readme.md files. Then I saved these .md files to _posts folder in my Jekyll project.
I used the jekyll build command to build at the _posts dir level. It created a _site folder only contains .md files but .html. What am I missing here?
jekyll build shouldn't run inside _posts folder, it should be executed at root level, i.e., tipically where your _config.yml is.
.
├── 404.html
├── about.md
├── _config.yml
├── Gemfile
├── Gemfile.lock
├── index.md
└── _posts
   └── 2017-08-06-welcome-to-jekyll.markdown
Then your generated website will be located at /_site.

Best practice for Django project working directory structure [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I know there is actually no single right way. However I've found that it's hard to create a directory structure that works well and remain clean for every developer and administrator. There is some standard structure in most projects on github. But it does not show a way to organize another files and all projects on pc.
What is the most convenient way to organize all these directories on development machine? How do you name them, and how do you connect and deploy this to server?
projects (all projects that your are working on)
source files (the application itself)
working copy of repository (I use git)
virtual environment (I prefer to place this near the project)
static root (for compiled static files)
media root (for uploaded media files)
README
LICENSE
documents
sketches
examples (an example project that uses the application provided by this project)
database (in case sqlite is used)
anything else that you usually need for successful work on project
The problems that I want to solve:
Good names of directories so that their purpose is clear.
Keeping all project files (including virtualenv) in one place, so I can easily copy, move, archive, remove whole project or estimate disk space usage.
Creating multiple copies of some selected file sets such as entire application, repository or virtualenv, while keeping single copy of another files that I don't want to clone.
Deploying right set of files to the server simply by rsyncing selected one dir.
There're two kind of Django "projects" that I have in my ~/projects/ directory, both have a bit different structure.:
Stand-alone websites
Pluggable applications
Stand-alone website
Mostly private projects, but doesn't have to be. It usually looks like this:
~/projects/project_name/
docs/ # documentation
scripts/
manage.py # installed to PATH via setup.py
project_name/ # project dir (the one which django-admin.py creates)
apps/ # project-specific applications
accounts/ # most frequent app, with custom user model
__init__.py
...
settings/ # settings for different environments, see below
__init__.py
production.py
development.py
...
__init__.py # contains project version
urls.py
wsgi.py
static/ # site-specific static files
templates/ # site-specific templates
tests/ # site-specific tests (mostly in-browser ones)
tmp/ # excluded from git
setup.py
requirements.txt
requirements_dev.txt
pytest.ini
...
Settings
The main settings are production ones. Other files (eg. staging.py,
development.py) simply import everything from production.py and override only necessary variables.
For each environment, there are separate settings files, eg. production,
development. I some projects I have also testing (for test runner), staging
(as a check before final deploy) and heroku (for deploying to heroku) settings.
Requirements
I rather specify requirements in setup.py directly. Only those required for
development/test environment I have in requirements_dev.txt.
Some services (eg. heroku) requires to have requirements.txt in root directory.
setup.py
Useful when deploying project using setuptools. It adds manage.py to PATH, so I can run manage.py directly (anywhere).
Project-specific apps
I used to put these apps into project_name/apps/ directory and import them
using relative imports.
Templates/static/locale/tests files
I put these templates and static files into global templates/static directory, not inside each app.
These files are usually edited by people, who doesn't care about project code
structure or python at all. If you are full-stack developer working alone or
in a small team, you can create per-app templates/static directory. It's really just a matter of taste.
The same applies for locale, although sometimes it's convenient to create separate locale directory.
Tests are usually better to place inside each app, but usually there is many
integration/functional tests which tests more apps working together, so global
tests directory does make sense.
Tmp directory
There is temporary directory in project root, excluded from VCS. It's used to
store media/static files and sqlite database during development. Everything in
tmp could be deleted anytime without any problems.
Virtualenv
I prefer virtualenvwrapper and place all venvs into ~/.venvs directory,
but you could place it inside tmp/ to keep it together.
Project template
I've created project template for this setup, django-start-template
Deployment
Deployment of this project is following:
source $VENV/bin/activate
export DJANGO_SETTINGS_MODULE=project_name.settings.production
git pull
pip install -r requirements.txt
# Update database, static files, locales
manage.py syncdb --noinput
manage.py migrate
manage.py collectstatic --noinput
manage.py makemessages -a
manage.py compilemessages
# restart wsgi
touch project_name/wsgi.py
You can use rsync instead of git, but still you need to run batch of commands to update your environment.
Recently, I made django-deploy app, which allows me to run single management command to update environment, but I've used it for one project only and I'm still experimenting with it.
Sketches and drafts
Draft of templates I place inside global templates/ directory. I guess one can create folder sketches/ in project root, but haven't used it yet.
Pluggable application
These apps are usually prepared to publish as open-source. I've taken example
below from django-forme
~/projects/django-app/
docs/
app/
tests/
example_project/
LICENCE
MANIFEST.in
README.md
setup.py
pytest.ini
tox.ini
.travis.yml
...
Name of directories is clear (I hope). I put test files outside app directory,
but it really doesn't matter. It is important to provide README and setup.py, so package is easily installed through pip.
My answer is inspired on my own working experience, and mostly in the book Two Scoops of Django which I highly recommend, and where you can find a more detailed explanation of everything. I just will answer some of the points, and any improvement or correction will be welcomed. But there also can be more correct manners to achieve the same purpose.
Projects
I have a main folder in my personal directory where I maintain all the projects where I am working on.
Source Files
I personally use the django project root as repository root of my projects. But in the book is recommended to separate both things. I think that this is a better approach, so I hope to start making the change progressively on my projects.
project_repository_folder/
.gitignore
Makefile
LICENSE.rst
docs/
README.rst
requirements.txt
project_folder/
manage.py
media/
app-1/
app-2/
...
app-n/
static/
templates/
project/
__init__.py
settings/
__init__.py
base.py
dev.py
local.py
test.py
production.py
ulrs.py
wsgi.py
Repository
Git or Mercurial seem to be the most popular version control systems among Django developers. And the most popular hosting services for backups GitHub and Bitbucket.
Virtual Environment
I use virtualenv and virtualenvwrapper. After installing the second one, you need to set up your working directory. Mine is on my /home/envs directory, as it is recommended on virtualenvwrapper installation guide. But I don't think the most important thing is where is it placed. The most important thing when working with virtual environments is keeping requirements.txt file up to date.
pip freeze -l > requirements.txt
Static Root
Project folder
Media Root
Project folder
README
Repository root
LICENSE
Repository root
Documents
Repository root. This python packages can help you making easier mantaining your documentation:
reStructuredText
Sphinx
Sketches
Examples
Database
I don't like to create a new settings/ directory. I simply add files named settings_dev.py and settings_production.py so I don't have to edit the BASE_DIR.
The approach below increase the default structure instead of changing it.
mysite/ # Project
conf/
locale/
en_US/
fr_FR/
it_IT/
mysite/
__init__.py
settings.py
settings_dev.py
settings_production.py
urls.py
wsgi.py
static/
admin/
css/ # Custom back end styles
css/ # Project front end styles
fonts/
images/
js/
sass/
staticfiles/
templates/ # Project templates
includes/
footer.html
header.html
index.html
myapp/ # Application
core/
migrations/
__init__.py
templates/ # Application templates
myapp/
index.html
static/
myapp/
js/
css/
images/
__init__.py
admin.py
apps.py
forms.py
models.py
models_foo.py
models_bar.py
views.py
templatetags/ # Application with custom context processors and template tags
__init__.py
context_processors.py
templatetags/
__init__.py
templatetag_extras.py
gulpfile.js
manage.py
requirements.txt
I think this:
settings.py
settings_dev.py
settings_production.py
is better than this:
settings/__init__.py
settings/base.py
settings/dev.py
settings/production.py
This concept applies to other files as well.
I usually place node_modules/ and bower_components/ in the project directory within the default static/ folder.
Sometime a vendor/ directory for Git Submodules but usually I place them in the static/ folder.
As per the Django Project Skeleton, the proper directory structure that could be followed is :
[projectname]/ <- project root
├── [projectname]/ <- Django root
│ ├── __init__.py
│ ├── settings/
│ │ ├── common.py
│ │ ├── development.py
│ │ ├── i18n.py
│ │ ├── __init__.py
│ │ └── production.py
│ ├── urls.py
│ └── wsgi.py
├── apps/
│ └── __init__.py
├── configs/
│ ├── apache2_vhost.sample
│ └── README
├── doc/
│ ├── Makefile
│ └── source/
│ └── *snap*
├── manage.py
├── README.rst
├── run/
│ ├── media/
│ │ └── README
│ ├── README
│ └── static/
│ └── README
├── static/
│ └── README
└── templates/
├── base.html
├── core
│ └── login.html
└── README
Refer https://django-project-skeleton.readthedocs.io/en/latest/structure.html for the latest directory structure.
Here is what I follow on My system.
All Projects: There is a projects directory in my home folder i.e. ~/projects. All the projects rest inside it.
Individual Project: I follow a standardized structure template used by many developers called django-skel for individual projects. It basically takes care of all your static file and media files and all.
Virtual environment: I have a virtualenvs folder inside my home to store all virtual environments in the system i.e. ~/virtualenvs . This gives me flexibility that I know what all virtual environments I have and can look use easily
The above 3 are the main partitions of My working environment.
All the other parts you mentioned are mostly dependent on project to project basis (i.e. you might use different databases for different projects). So they should reside in their individual projects.
You can use https://github.com/Mischback/django-project-skeleton repository.
Run below command:
$ django-admin startproject --template=https://github.com/Mischback/django-project-skeleton/archive/development.zip [projectname]
The structure is something like this:
[projectname]/ <- project root
├── [projectname]/ <- Django root
│ ├── __init__.py
│ ├── settings/
│ │ ├── common.py
│ │ ├── development.py
│ │ ├── i18n.py
│ │ ├── __init__.py
│ │ └── production.py
│ ├── urls.py
│ └── wsgi.py
├── apps/
│ └── __init__.py
├── configs/
│ ├── apache2_vhost.sample
│ └── README
├── doc/
│ ├── Makefile
│ └── source/
│ └── *snap*
├── manage.py
├── README.rst
├── run/
│ ├── media/
│ │ └── README
│ ├── README
│ └── static/
│ └── README
├── static/
│ └── README
└── templates/
├── base.html
├── core
│ └── login.html
└── README

How can I correctly set DJANGO_SETTINGS_MODULE for my Django project (I am using virtualenv)?

I am having some trouble setting the DJANGO_SETTINGS_MODULE for my Django project.
I have a directory at ~/dev/django-project. In this directory I have a virtual environment which I have set up with virtualenv, and also a django project called "blossom" with an app within it called "onora". Running tree -L 3 from ~/dev/django-project/ shows me the following:
.
├── Procfile
├── blossom
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── fixtures
│   │   └── initial_data_test.yaml
│   ├── manage.py
│   ├── onora
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── admin.py
│   │   ├── admin.pyc
│   │   ├── models.py
│   │   ├── models.pyc
│   │   ├── tests.py
│   │   └── views.py
│   ├── settings.py
│   ├── settings.pyc
│   ├── sqlite3-database
│   ├── urls.py
│   └── urls.pyc
├── blossom-sqlite3-db2
├── requirements.txt
└── virtual_environment
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── django-admin.py
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── gunicorn
│   ├── gunicorn_django
│   ├── gunicorn_paster
│   ├── pip
│   ├── pip-2.7
│   ├── python
│   └── python2.7 -> python
├── include
│   └── python2.7 -> /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
└── lib
└── python2.7
I am trying to dump my data from the database with the command
django-admin.py dumpdata
My approach is to run cd ~/dev/django-project and then run source virtual_environment/bin/activate and then run django-admin.py dumpdata
However, I am getting the following error:
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
I did some googling and found this page: https://docs.djangoproject.com/en/dev/topics/settings/#designating-the-settings
which tell me that
When you use Django, you have to tell it which settings you're using.
Do this by using an environment variable, DJANGO_SETTINGS_MODULE. The
value of DJANGO_SETTINGS_MODULE should be in Python path syntax, e.g.
mysite.settings. Note that the settings module should be on the Python
import search path.
Following a suggestion at Setting DJANGO_SETTINGS_MODULE under virtualenv? I appended the lines
export DJANGO_SETTINGS_MODULE="blossom.settings"
echo $DJANGO_SETTINGS_MODULE
to virtual_environment/bin/activate. Now, when I run the activate command in order to activate the virtual environment, I get output reading:
DJANGO_SETTINGS_MODULE set to blossom.settings
This looks good to me, but now the problem I have is that running
django-admin.py dumpdata
returns the following error:
ImportError: Could not import settings 'blossom.settings' (Is it on sys.path?): No module named blossom.settings
What am I doing wrong? How can I check thesys.path? How is this supposed to work?
Thanks.
Don't run django-admin.py for anything other than the initial project creation. For everything after that, use manage.py, which takes care of the finding the settings.
I just encountered the same error, and eventually managed to work out what was going on (the big clue was (Is it on sys.path?) in the ImportError).
You need add your project directory to PYTHONPATH — this is what the documentation means by
Note that the settings module should be on the Python import search path.
To do so, run
$ export PYTHONPATH=$PYTHONPATH:$PWD
from the ~/dev/django-project directory before you run django-admin.py.
You can add this command (replacing $PWD with the actual path to your project, i.e. ~/dev/django-project) to your virtualenv's source script. If you choose to advance to virtualenvwrapper at some point (which is designed for this kind of situation), you can add the export PY... line to the auto-generated postactivate hook script.
mkdjangovirtualenv automates this even further, adding the appropriate entry to the Python path for you, but I have not tested it myself.
On unix-like machine you can simply alias virtualenv like this and use alias instead of typing everytime:
.bashrc
alias cool='source /path_to_ve/bin/activate; export DJANGO_SETTINGS_MODULE=django_settings_folder.settings; cd path_to_django_project; export PYTHONPATH=$PYTHONPATH:$PWD'
My favourite alternative is passing settings file as runtime parameter to manage.py in a python package syntax, e.g:
python manage.py runserver --settings folder.filename
more info django docs
I know there are plenty answers, but this one worked for me just for the record.
Navigate to your .virtual_env folder where all the virtual environments are.
Go to the environment folder specific to your project
Append export DJANGO_SETTINGS_MODULE=<django_project>.settings
or export DJANGO_SETTINGS_MODULE=<django_project>.settings.local if you are using a separate settings file stored in a settings folder.
Yet another way to do deal with this issue is to use the python dotenv package and include PYTHONPATH and DJANGO_SETTINGS_MODULE in the .env file along with your other environment variables. Then modify your manage.py and wsgi.py to load them as stated in the instructions.
from dotenv import load_dotenv
load_dotenv()
I had similar error while working on windows machine. My problem was using wrong debug configuration. Use Python:django as your debug config option.
First ensure you've exported/set django_settings_module correctly here.