I am relatively new to Django and this is a more general 'concept' question.
For a client I need to construct an expansive database holding data returned from a series of questionnaires as well as some basic biological data. The idea is to move away from the traditional tools (i.e. Microsoft Access) and manage the data in a mysql database using a basic CRUD interface. Initially the project doesn't need to live on the web, but the next phase will to be to have a centralized db with login and admin page.
I have started building the db with Django models which is great, and I want to use the Django admin for the management of the data.
My question is: Is this a good use of Django? Is there anything I should consider before relying on django for the whole process? And is it advisable to us the Django runserver for db admin on a client's local machine (before we get to the web phase).
Any advice would be much appreciated.
Actually, your description sounds exactly like the sort of thing for which Django is an ideal solution. It sounds more complex and customized than a CMS, and if it's as straightforward as your description then the ORM is definitely a good tool for this. Then again, this sounds exactly like an appserver-ready problem, so Rails, Express for Node.js, or even ChicagoBoss (if you're brave) would be good platforms for this kind of application.
And sure, Django is solid enough you can run it with the test server for local clients before you go whole-hog and run the thing on the web. For that, though, I recommend Apache/mod_wsgi, and if you're going to be fault tolerant there are diamond architectures (one front end proxy with monitoring failover, two or more appserver machines, one database with hot spare) and more complex (see: sharding) architectural layouts you can approach later.
If you're going to run it in a client's local setting, and you're not running Windows, I recommend looking into the screen program. It will allow you to detach the running job into the background while making diagnostics accessible in an ongoing fashion.
Related
I've read articles and posts about what a project and an app is for Django, and basically end up using the typical example of Pool and Users, however a real program generally use a complex relational database, therefore its design gravitates around this RDB; and the eternal conflict raises once again about: which ones to consider an application and which one to consider components of that application?
Let's take as an example this RDB (courtesy of Visual Paradigm):
I could consider the whole set as an application or to consider every entity as an application, the outlook looks gray. The only thing I'm sure is about this:
$ django-admin startproject movie_rental
So I wish to learn from the expertise of all of you: What approach (not necessarily those mentioned before) would you use to create applications based on this RDB for a Django project?
Thanks in advance.
PS1: MORE DETAILS RELATED ABOUT MY REQUEST
When programming something I follow this steps:
Understand the context what you are going to program about,
Identify the main actors and objects in this context,
If needed, make an UML diagram,
Design a solid-relational-database diagram, (solid=constraints, triggers, procedures, etc.)
Create the relational database,
Start coding... suffer and enjoy
When I learn something new I hope they follow these same steps to understand where they want to go with their actions.
When reading articles and posts (and viewing videos), almost all of them omit the steps 1 to 5 (because they choose simple demo apps), and when programming they take the easy route, and don't show other situations or the many supposed features that Django offers (reusability, pluggability, etc).
When doing this request, I wish to know what criteria is used for experienced programmers in Django to determine what applications to create based on this sample RDB diagram.
With the (2) answers obtained so far, "application" for...
brandonris1 is about features/services
Jeff Hui is about implementing entities of a DB
James Bennett is about every action on a object, he likes doing a lot of apps
Conclusion so far: Django application is a personal creed.
My initial request was about creating applications, but as models are mentioned, I have this another question: is with a legacy relational database (as showed in the picture) possible to create a Django project with multiple apps? this is because in every Django demo project showed, every app created has a model with their own tables, giving the impression that tables do not interact with those of other applications.
I hope my request is more clear. Thanks again for your help.
It seems you are trying to decide between building a single monolithic application vs microservices. Both approaches have their pros and cons.
For example, a single monolithic application is a good solution if you have a small amount of support resources and do not need to be able to develop new features in fast sprints across the different areas of the application (i.e. Film Management Features vs Staff Management Features)
One major downside to large monolithic applications is that eventually their feature sets grow too large and with each new feature, you have a significant amount of regression testing which will need to be done to ensure there aren't any negative repercussions in other areas of the application.
Your other option is to go with a microservice strategy. In this case, you would divide these entities amongst a series of smaller services and provide them each methods to integrate/communicate with each other (APIs).
Example:
- Film Service
- Customer Service
- Staff Service
The benefits of this approach is it allows you to separate capabilities and features by specific service areas thus reducing risk and regression testing across the application when new features are deployed or there is a catastrophic issue (i.e. DB goes down).
The downside to this approach is that under true microservice architecture, all resources are separated therefore you need to have unique resources (ie Databases, servers) for each service thus increasing your operating cost.
Either of these options is a good option but is totally dependent on your support model and expected volumes. Hope this helps.
ADDITIONAL DETAIL:
After reading through your additional details, since this DB already exists and my assumption is that you cannot migrate it, you still have the same choice as to whether or not you follow a monolithic application or a microservices architecture.
For both approaches, you would need to connect your django webapp the the specific DB you are already using. I can't speak for every connector out there but I know that the MySQL connector allows django to read from the pre-existing db to systematically generate the models.py file for the application. As a part of that connector, there is a model variable which allows you to define whether or not Django is responsible for actually managing the DB tables themselves.
The only thing this changes from an architecture perspective is how many times do you want to code this connection?
If you only want to do it once and completely comply with the DRY method, you can build a monolithic application knowing that as new features become required, application wide regression testing will be an absolute requirement.
If you want ultimate flexibility for future changes with this collection of features and don't mind recoding the migration across multiple apps while reducing the need for application wide regression testing as new features become required, a microservice architecture strategy is more appropriate.
I have a Coldfusion application, developed without any framework and almost no architecture.
I'd like slowly migrate some part of the application to some kind of java based web application framework.
The original application must still be the main application all the way until the end of the migration.
The application has user and sessions and a lot of functionalities, not easy to decouple.
I'm looking for different ideas.
For example I could try to develop a REST API backe nd and start to use it from the ColdFusion application until I'll have Coldfusion only as front end. This process must go hand by hand with database refactoring and migrations, ie. new API must use its own database, so I guess that decoupling of database will be necessary (and probably database synchronization issues will arise)
But I would like to switch also to a different front end technology.
The situation would be like: Users login in coldfusion, enter main dashboard in Coldfusion and next "some" functionalities will be handled by another framework. That means a template engine is able to understand the user, his roles, his permissions, reproduce the same graphical layout, but it should serve the content with another technology.
Final result must be that all functionalities are migrated to the new technology.
I mention Java as it is in some way related to ColdFusion, but any web application framework could be used in principle.
I also think that Coldfusion is used in the original application is not relevant. The fact that no framework is used, probably gives me more flexibility.
Any architectural suggestion is welcome.
I recently did this very thing. A spaghetti code app developed by a graphic designer with a CFML book, a MSSQL database so denormalized not even Bizarro could have wrestled it and won.
What we did was we left the old app in place, built a new one and tested it extensively, perfected it, made sure it was capable of evolving (and it has). Then we flipped a DNS switch and sent out an announcement that the new system is the only option going forward.
I then built an archive feature that provided read-only access to the mess that we left behind. Some day, if the app owners want to migrate that data, we can try that (it won't go well), but generally the vast improvements have convinced everyone that the old stuff is only needed in that read-only mode.
Obviously that means reporting only goes back to the first day of the new tool, but it was either that (with a 4 week dev time) or they get it all and it takes 8 months doing what you described.
It's worked very well for us, and it's probably been the best-received work I've done here.
I am looking for some advice before I start on a new project.
I am creating a Web Application using Django 1.10. I have experience with Django and creating general "content-based" websites with it. However, since this project is going to be a web-based application, I plan on doing more "complex" things than just rendering HTML templates and doing some basic CRUD operations. When I say "complex" things, the most specific example I can give at this point is to leverage more asynchronous requests so my web-application can remain responsive to the user and provide that "real-time" experience that comes with an application that might be installed on their local machine or whatever. Plus, since this is a web-application and not just a website, the project is definitely going to be more data-driven which could potentially mean requesting large amounts of data that would be best served in say a paginated manner, etc.
So, my though was this.....Since I am familiar with Django and have read such good things about the Django REST Framework, I could create a RESTful API to perform all of my CRUD operations and basically interface with my web application's core database.
At that point, I could essentially have two "layers": (1) A presentation layer that will render my web-application's pages and (2) an application layer that will do all my backend CRUD operations. Since the two are separated, I will also get the added benefit of being able to leverage the API from other avenues (other than my Django web-app) if that is ever required down the road.
I guess my first question is whether or not this makes sense and if so, thoughts on the best way to implement it. I believe I have two options.
Create a single Django project and include the API as a separate application. This seems like it would work fine but it would couple my API with my presentation since they would both be hosted by the same server. Which...if the only "consumer" of the API is my one Django site, then this may not be an issue in the near term but could cause problems later on.
Separate the two into two distinct Django projects. This offers the most flexibility and is probably the best answer, I think.
There may be other options as well. At the end of the day, I am looking for some advice/guidance from those who have done this before and what other things I should consider before starting on this effort.
Basically I would not recommend the way number 2. This is because you really fast will start to struggle with issues like that: where to store models? Project A, Project B, Both?? Also I am almost sure that you will have other code that can be shared between both of the django projects.
Worth to note here that in two separate django projects - you will have troubles with synchronization of the migrations (just consider simultaneously change of the models in both projects). Personally - I've never was able to solve this in some nice, acceptable way.
Maybe I will share with you my personal experience - hope this will be helpful.
First is that if your are making a big application - consider resignation from django templates - use some modern js framework for frontend: react, angular; and make a separate project for fontend. This is tricky because you need to have right competences in the team. If this is not possible - well do separate django app for front.
Use DRF - you can create rest app inside your project and use all of the models defined elsewhere - it will provide you simple CRUD out of the box and with some work it can be really powerful. If you will be able to create front in modern js framework - this REST can be also used to feed data to the front. And also any other client you can imagine. This basically shift the amount of work needed to be done in the backend (only REST instead of REST + templates and rendering) to the frontend.
Asynchronous tasks. Well nothing new here - use celery. Define your tasks inside application; make sure that you have correct number of workers. And let the magic begin.
As for real-time experience, you may consider using django-channels: https://github.com/django/channels/ It is nice way to handle websocket connections.
Things to consideration:
In big application usually you will end up with something like this:
myapp.com -> point to frontend; api.myapp.com -> point to the api
You should make a clear separation in the code between this two. In the case when your front is js-based - this is not a problem, but rest + rendering it is important to be able to run only-api-node and only-rendering-node. It's hard to say which one of them will be used more heavily.
So basically you can end up with something like this:
core
models
users.py
my_app_frontend
users
views.py
forms.py
tasks.py
urls.py
my_app_rest
users
serializers.py
views.py
signals.py
urls.py
my_app
settings.py
rest_settings.py
manage.py
wsgi.py
rest_wsgi.py
urls.py
rest_urls.py
This have some advantages that I like:
Models are in one place;
There is a possibility to run two separates nodes - based on needs.
Your development settings should be able to run dev server as a whole. Combing the settings from front and rest.
There is a clear code separation - and we know what is what in above structure.
If you have any more question - please ask.
Also I am curious what the others have to say - and how they handle issues pointed by me :)
Happy coding!
I've been looking at Rietveld as a solution for the lack of code reviews at my company. Can it be set up on a server in-house without using App Engine? It seems to have a bit of App Engine specific code, and I'm not sure it could be set up on a plain old Django/Apache install. I've looked around, but haven't found any information about this.
Check out http://django-gae2django.googlecode.com/svn/trunk/examples/rietveld/README
The gae2django project lets GAE apps run against django instead of the GAE development environment.
That means you can run rietveld under django directly, using (by default) an SQLite backend. You can also use mysql or any other DB backend django supports.
That, plus a web server (e.g. Apache) with WSGI integration, makes a local rietveld install run nicely.
What about using one of these projects that provide the same backend services as GAE?
Typhoon AE
Appscale
There may be more, these are just the ones I know about off the top of my head.
A bit of App Engine specific code? It's supposed to be an example App Engine app, so yeah it's pretty well tied to it. But, you're right, it does use Django which could make it somewhat more feasible to port. I'll second #cope360 recommendation, but from the sounds of your question, it doesn't sound like you've done much with App Engine. If it's only used by a few people, try running it on the GAE SDK itself.
Beyond that, I'd think you could take most of the code in the "codereview" directory and build you're own Django/apache app from that.
Rather than fussing around with a port or other GAE emulation, I would consider using ReviewBoard.
Review Board is a powerful web-based
code review tool that offers
developers an easy way to handle code
reviews. It scales well from small
projects to large companies and offers
a variety of tools to take much of the
stress and time out of the code review
process.
For too long, code reviews have been
too much of a chore. This is largely
due to the lack of quality tools
available, leaving developers to
resort to e-mail and bug tracker-based
solutions.
We've seen a lot of time and energy
wasted doing code reviews both in open
source projects and at companies. In
both cases, code reviews were
typically done over e-mail. A
significant amount of time was spent
in forming review requests, switching
between the diff and the e-mail, and
trying to understand what parts of the
code the reviewer was referring to.
So in an effort to keep our sanity and
improve the process both in our open
source projects and at companies, we
wrote Review Board. We hope it will be
useful to your team too so you can
focus on what's important: writing
great products.
I have my first app, not that big, but it is the first step. (next big one on the way)
Now if I want to put it on my own Linode VPS, I have to configure mod_python or mod_wsgi, as well as memcache, Ngix, mySQL or Postgresql, etc. to make it work. If I put it GAE, All I have to do is convert the models to use GAE's API.
What I like about GAE is scaling. (if they can really do it)
Then I'd only worry about developing my apps and doing SEO work on them instead of worrying about load share/balance, cache, db / IO redundancy, etc.
I don't want to do any porting later on. (I have to decide now and stick with it)
So, if you have any experience on this, what do you recommend:
1- Use VPS(s) for everthing
2- Use VPS(s) plus Amazon S3
3- Use VPS(s) plus Amazon S3 & SimpleDB
4- Use GAE
Also: Would I be able to get away with not having JOIN rights when using the BigTable?
Note: I don't have any spatial need now, but for a location table I might need that later on.
I'd like to know what do you think!
There's business risk and technical risk.
Business risk is that you might have to move hosts later for some external reason. VPS's, EC2, etc require more upfront investment, but keep you independent. Tools like Chef can help with the configuration effort.
Technical risk is that your application may not be easily implemented on the platform. Since most VPS options allow you to install arbitrary software, they minimize this, again at the cost of more configuration effort on your part. AFAIK, the largest constraint GAE enforces on you is it's difficult to do long running background tasks. (Working without JOINs and other aspects of de-normalized data requires a different way of thinking, but this approach is fairly common in web applications no matter where they run once the SQL database is larger than a single host can support.)
If you can live with both these risks, GAE would appear to save you a substantial amount of effort. If you cannot live with these risks, you should tailor your own environment.
As an aside, I find S3 to be worth it no matter your environment. It's far simpler than ensuring your local server static file storage is reliably backed up, and you never have to worry about capacity. It's best if you use it for data that is uploaded but rarely overwritten or deleted (think facebook photo albums).
I don't want to do any porting later on. (I have to decide now and stick with it)
If that's the case, wouldn't you prefer to control deployment from the outset? It could be a great pain to port back from GAE later down the line if you hit its limits (whether they be technological limits or simply business decisions by Google that run counter to your plans for the future of your app).
Also configuring mod_wsgi, installing postgres etc. isn't particularly difficult, and you don't have to worry about things like load balancing and db redundancy for a while yet.
If it were me, I'd prefer the long-term certainty of a traditional server over the quick win of GAE. It all depends on your vision for the app, however.
I may be biased, but if you can live with GAE's limitations it really saves you a lot of work and worry about system administration issues (and to some extent scaling) -- plus, it's free as long as your resource consumption is low (basically meaning your traffic is low).
Can you do without joins? I don't know, as I don't know your app -- I'm a SQL fanatic, myself, yet for simple enough needs I haven't found it too hard to adapt. As I see it, the main limitation of non-relational DBs is that they're nowhere as nice as relational ones for "ad hoc" queries... you typically have to write a lot of procedural code instead of a nice SELECT or two:-(. But, that's more of a "data mining later" issue than one connected with serving your web app -- probably best solved by regularly bulk-downloading data from the web app's online storage to a "data warehouse" kind of setup, anyway, even if such storage was relational in the first place;-).
Before deciding, it might be worth a quick prototype adaptation of your app to GAE. You might run into stoppers that force the decision. Possible stopper issues include
Your schema doesn't make the transition to BigTable
You're depending on some C-based library that GAE doesn't support
You have a few long-running requests that exceed the thresholds that GAE imposes
The answer depends on the complexity and nature of your model layer, really. If it's complex or tightly bound to the rest of your code, porting is likely to be a significant effort. If it's fairly straightforward, or easy to tear out and replace, I would say go for it.
These days, I mostly write new code for GAE, but the fact that I can simply deploy with a single command has really lowered the barrier I feel towards writing cool new apps. Not having to worry about deployment and hosting is quite liberating.
All I have to do is convert the models to use GAE's API.
I am sorry, you are totally mistaken.
You also need to rewrite all the views code that uses the ORM. There are no joins. So you have to deal with and write a lot of procedural code instead of the nifty SQL that provides U whatever you want.
Querying is slow. You need to override save method of each model to store additional information of that model which may take a lot of time to compute when need. You also need to work on memcache to make the queries fast enough.
And then, Guido has said Django 1.1 is going to be included in a future version of Appengine. I am hoping they will have an out of the box generic ORM to BigTable mapper.
That said, if your app is simple without many joins needed, you could use the appengine patch project to use the current version of django on Appengine. Here is how.