I'm doing some i18n on a web-based app using Django, which uses gettext as its i18n foundation. It seems like an obvious idea that translations should be stored in the database, and not difficult to do, but po files on the filesystem are still being used. Why is this?
My current suspicion is that the benefits of developing a db backaged are simply outweighed by the reliability/familiarity of gettext as a well-established package. Are there other significant reasons for continuing to store the translations on the filesystem?
Performance is the main reason. Gettext is not using a database because a database will always be considerably slower than a file. The load time of the dictionary is very important and for this reason almost everyone is using files for that.
Also, the compiled gettext files (.mo) are optimized for loading in memory and for this reason they are more appropriate than plain text files (like not-compiled .po files).
You can always use translation platform, probably that uses a database backend, for doing the translation and export the results to text files. Examples: Pootle, Narro, Launchpad Rosetta, Transifex (hosted only).
Do not confuse your application language files with the localization database. Your application should use file based dictionaries that are fast to load and your localization system probably will have to use a database and logically be able to export data to files.
By the way, using gettext is probably the best technological decision you may be able to make regarding localization. I never seen any commercial solution or in-house developed to be able to compete with it on features, tools and even support.
It's a very common way to do translations that has been around for a long time allowing any issues to be ironed out over the years. I imagine writing something like gettext it would be all too easy to make incorrect generalisations about how languages work. Why should the Django development team spending time researching that and developing it when it's already been done in a tried and tested system? Furthermore professional translators probably know what to do with PO files where as a home-brew translation database may prevent them from working in ways they're used to.
Why would you prefer translations in a database? I guess you might prefer it as you could make a translation interface to the database. If that's the case have a look at Pootle it's a powerful web-based translation interface that works directly with PO files and can even integrate with common version control systems. Add some post-commit hooks and you can have such a system with little work and without the overhead of a translations database.
Hope that helps.
This seems like an obvious idea for you, I don't think everybody will agree. AFAIK django uses .po files for following reasons:
Version control - you will have to create additional ".po to database" tools, because you still need to maintain different people working on translations, and you can't get away from having .po files for that purpose
gettext is a standart way of doing translations in .nix world, there are many tools for working with it and it's simple to edit, diff, etc.
No need to hit database if you need to translate anything. Some views can work without any db requests, so no need to tie them to database just to get translation. (I may be wrong, but in case of mod_wsgi - translations will be loaded once and stored in memory for every thread).
Btw, if you need to have different translations for fields, it's a bit different question and you should check http://www.muhuk.com/2010/01/dynamic-translation-apps-for-django/ and choose app that best fit your needs.
Related
We have a MySQL database with very big number of tables. Unfortunately in 2018 we still use Perl CGI. So loading time of a script is essential.
DBIx::Class was ruled out by me because it loads about 1.6 sec (so long because it loads Perl definitions for all tables of the DB) what is clearly too much.
How quickly Class::DBI loads? My main question: Does Perl load information about all available tables (like DBIx::Class does) when we use Class::DBI or does it load Perl definitions for only these tables which we actually use?
The following is a DBIx::Class code which loads 1.6 sec:
#!/usr/bin/perl
package MyApp::Schema;
use lib '.../ORMs/dbix-class';
use base qw/DBIx::Class::Schema/;
__PACKAGE__->load_namespaces();
1;
(The schema is autogenerated.)
Is there any way to make it faster? How to use it without loading all tables?
I really wouldn't recommend Class::DBI. It's been unmaintained for twelve years - and there were good reasons why everyone switched to DBIx::Class.
I would highly recommend working on the problem that leads to you still using CGI. What is preventing you from, for example, using CGI::Emulate::PSGI to trivially convert your CGI code to PSGI apps which you can then deploy in a persistent environment like FastCGI or, better, as a standalone service which you can then access using nginx? Any of these solutions would mean that the DBIx::Class load time is no longer problem.
Obviously, I have no idea what is keeping you tied to CGI. But, in my experience, moving to PSGI solution is often easier than people expect it to be and it will undoubtedly leave you in a better position.
I've been tasked with porting personal face recognition software to iOS and Mac OS X as well as helping keep the basic SDK and much of the software as cross-platform as possible. One of the things one of my associates and I want to do is store data on the user's face in an SQL database (probably SQLite). We would also like to allow users to put their data on iCloud so they don't have to train each of their devices separately to recognize them. What's bugging me is how to do both these tasks, and I'm confronted with enough choices to feel overwhelmed. (I am still new to some of the technologies involved.)
For implementing SQL, I could embed SQLite directly in my program and write code for it, or I could use Core Data and have it talk to SQLite for me. (The database is not meant to be shared, so this is OK. And SQL is not fun.) However Core Data is anything but portable (not to mention not intended for a model encoded as C++ objects), while writing directly for SQL would mean we could reuse more code on other platforms.
Things get messier when factoring in iCloud, which has something like five or six possible ways of integrating it with a program. The only method I have definitively ruled out so far is iCloud key-value storage. (At the very least, there's a good chance a user would get into trouble with the 1 MB limit, and it is clearly not intended for anything as complex as I'm dealing with.) Core Data can integrate with iCloud through UIManagedDocument or NSPersistentStore, but, again, that means less in the way of reusable code. I can use SQLite together with UIDocument or NSDocument, but what I am trying to do seems to be not quite what these objects were intended for. The files I am dealing with are essentially large preference files, not meant for end-users to interact with directly; UIDocument and NSDocument seem to be meant for user-viewable and -editable files. And then there are iCloud Drive and CloudKit, which are still in beta. (On the other hand, these two are due to be released fairly soon. Considering that iOS users tend to upgrade to the latest version of the system software quickly, arguments about using either of these based on how many devices they will be able to run on should quickly become weak and obsolete.)
Can anyone recommend which way is best suited for my purposes? Thanks in advance.
Aaron Solomon Adelman
First off, you don't want to try and share the SQLite file directly. That's extremely likely to corrupt the file, because SQLite wasn't built with that kind of use in mind.
However:
You could use SQLite for local on-device storage only and use a separate API to send data back and forth. Apple's CloudKit would probably be a good choice if you can require iOS 8+. Numerous third party solutions exist (for example, Parse). You'll have to write your own code to translate between SQLite and the network API. Your SQLite schema and your data files would be portable to other platforms, and maybe some of the code if you use SQLite's direct API instead of an Objective-C wrapper (and I highly recommend using either FMDB or PLDatabase if you use SQLite).
Core Data does have built-in support for iCloud, which probably makes it a viable option. Your comment that "...I could use Core Data and have it talk to SQLite for me." suggests you might have somewhat misunderstood Core Data. Core Data is not a SQLite wrapper; it presents a completely different API, and uses its own schema. You can't really take a Core Data persistent store file and use it on other platforms unless you want to spend some time reverse-engineering the schema. Also, using Core Data with iCloud does not require the use of UIManagedDocument, though it does still require a lot of other Core Data-specific classes.
If you want to be able to sync data across multiple devices which are not all Apple devices then you need a third party API. None of Apple's cloud APIs will be useful here. There are many providers that can help out with this. For local data storage, either SQLite or Core Data would work, but you should look at the third party services and see what storage option(s) they support, then try to work with them.
The best approach depends on your needs. If you expect to copy data files from an iOS app to other platforms, SQLite is good. You'll still have a lot of platform-specific code, the savings there are much less. If you don't plan to move data files around like that, Core Data is probably easier to deal with.
I'm starting to design an application and I'd like it to support several languages from the beginning to avoid having to change big amounts of code later on to enable multiple languages.
I've written early a C application for which the messages were in a struct so I'd have several header files with the translated strings which would be integrated into the application itself, so to add a new language I'd have to recompile.
What I'd like is to be able to have a localization system that allows me to translate the application and add new languages easily without having to recompile the application. Ideally with a pretty straightforward way of translating and testing the translation (like editing a text file and seeing the results back in the app), without having to depend on 3rd party tools for the translation.
I'm thinking about writing my own system for it, maybe based on XML for the files containing the translations for example, but I wonder if anybody has experience with this and would recommend a lightweight library that provides that I'd like (even if the translation is not so direct as editing a text file). I emphasize lightweight because I think the application itself is not going to weight more than a couple of megabytes.
It's going to be a Windows application, if that's relevant for the matter. As of now I still haven't decided on the graphic toolkit I use but it's probably that I'll use the default Windows offers without using a cross platform one (like wxWidgets, Qt, GTK+, ...). But it's not set in stone, if one offered significant advantages it'd consider it.
I am not sure how lightweight is, but qt has a good support for internationalization.
Then you can use QtLinguist to simply translate your text.
Don't invent your own system. Internationalization (i18n) and localization (l10n) are almost always covered in some existing library.
If you develop for Windows, this link, Globalizing and Localizing .NET Framework Applications, might prove useful.
It's finally (after years of postponing) the time to localize my app in a few other languages other than English.
The first challenge is to design the integration into my C++ / MFC application that has dozens of dialogs and countless strings. I came across two possible alternative implementations:
Compile and deploy localized resource files as DLLs
Extract and replace all strings with the localized version. For each
language there will be an XML (or simple text) file.
Personally I opt for the second alternative since it seems to me more flexible. The changes are many but not hard to make, and very importantly the XML files will be very easy to modify for the translators.
Any advise is greatly appreciated.
Regards,
Cosmin Unguru
http://www.batchphoto.com/
I did some long-living MFC projects with different languages.
I strongly recommend the first approach with resource-only DLLs.
The reasons:
(1) If the user does a XCOPY install, he always has the default language (English) in the main executables.
(2) If you don't translate everything (e.g. you're late with your release or forget some strings), the Windows resource functions if properly used return the resource in the default language automatically - you don't have to implement it on your own.
(3) My very person opinion: (a) Line breaks, tabs, whitespaces in XML files are a pain in your a**. (b) Merging XML files is even worse...
(4) Don't forget the encoding. It's okay in XML but your translators might use an unsuitable editor and damage the file.
And now for the main reason:
(5) You will have to rearrange many of your dialogs, because many strings are longer in e.g. French or German than in English. And making all statics, buttons, ... larger "just in case" looks crappy.
Another hint: Spend some bucks and buy one of the translation tools which import your projects / binaries and build up a translation database. This will be amortized after the first release.
Another hint (2): If possible make a release which doesn't contain any changes but only the multi-language feature. Also in future, if possible: Release your product in English. Then do the translation in one single step (per language) and release the other languages.
My good and friendly suggestion from somebody who worked a lot with localization:
Grab GNU Gettext,
Mark all your strings as _("English").
Extract all strings using gettext tool xgettext and compile dictionalries
Translate string using great tools like poedit.
Use gettext in your project and make your localization life simpler!
You can also use boost::locale for same purpose - it uses GNU Gettext dictionaries and approach but provides different and more powerful runtime and for windows developer it has very good addon - it supports wide strings that MFC requires to use for normal Unicode support.
Don't use resources and other "translation" tools that are total crap from linguistic point of view (and developer's point of view as well).
Further reading: http://cppcms.sourceforge.net/boost_locale/html/tutorial.html
Using a DLL resource library is a relatively straightforward operation, and allows you to manage not only strings, but other resources as well. And this is its main advantage, because i18n is not only about string translation.
However, depending on your needs, a text-based solution may be a better decision, because of its easier handling - resource scripts being more complex than xml files, especially for the average translator.
I would suggest creating your own abstraction layer, something like "LoadLocalizedString", etc.; in this way, you can start implementing it just with text files, and then change to something more complex when and if required in a transparent way - all the effort for making your software i18n aware would still be valid.
In our case we had diffrent dialogues per Language. The resource file was the same as the multiple laguages were implemented at development time. You could basically append on existing resource files the diferent languages. I hope it helps to find your way.
The DLL option is commonly used for this since the resource loading procedure (e.g. LoadLibrary) is already written - meaning you don't have to write any parsing/loading code. While XML is easier to edit, DLLs have a bit more security (users won't be able to easily edit them) and will allow the developer (meaning you) more time to work on application logic instead of writing a language loading system.
HMODULE hLangDLL = LoadLibrary("text_en.dll");
// more stuff
TCHAR mybuffer[1024] = {0};
LoadString(hLangDLL, IDS_MYSTRING, mybuffer, 1023);
If it is just the strings that are changing then I agree that XML is the way forward here for the exact reasons you outline. Easy for other people to edit, easy to change language at runtime, etc.
The only reason (in my eyes) that you'd choose option 1 is if things other than strings are being localized such as needing different icons.
If it's just text? I say go with the XML.
First, some background information... I'm coming up on a medium-scale website for a non-profit that will require both English and Korean translations. Feature-set includes: CMS for normal content, a blog, some form submission/handling (including CSV/PDF exports), a job posting board, a directory of related businesses and non-profits (that accepts visitor submissions), and a basic (probably blog-driven) newsroom.
I have a fairly strong development background, and I've done some sites using Drupal, built some basic custom CMSes using frameworks like CodeIgniter, and I've recently started getting into Django. These are the primary options that I am exploring, and I would consider using different tools for different portions of the project, but what I'm mainly interested in, is if anyone has any experience to share with regards to localization/internationalization. I haven't yet put together a site that supports multiple languages, so before I get in trouble by underestimating the task, or making poor assumptions, I'd like to get some input to help guide my decision-making process.
Do you have any recommendations for frameworks (Drupal, Django, CodeIgniter) that handle localization/internationalization/translation well for a CMS? I know they all support it, but I'm looking for real-world experience here (or suggestions for modules/plugins given explanations).
Sorry for the longwinded question, but I wanted to be clear as possible. Thanks in advance!
There is a distinction between "site" translation and content translation. Django handles the site translation great, out of the box. The content translation, however, requires making some decisions (there's no one right way at this point). This probably makes sense, because of the very nature of Django as a lower level framework (when compared to something like Drupal, which is intended to serve as a complete CMS).
There are applications for Django which are meant to add this functionality (in the form of translations configured at the model level):
Django-multilingual
Transmeta
Also, I found this question that is related.
The bottom line though, is that this is still being explored in the Django world, and neither approach has been decided upon for the framework. Also, although I haven't used it, Drupal has module support for this in the form of the i18n module.
I will update with more conclusions as I come to them. If you have anything to add about content translation in Django or in Drupal, feel free to add your own answer as well.
You probably already know that the native i18n support in django is quite good. As for translation, you might try the django-rosetta app which allows you to grant translation rights to a subset of users, who are then able to translate through an admin-like interface.
Zend_Translate is pretty comprehensive. And if you decide to use PHP, I suggest you take a look at it. It provides multiple interfaces (e.g. an Array, CSV, Gettext, etc.) to manage your translations, which makes it IMHO unmatched when it comes to PHP.
I'm not sure how well it plays with Drupal, since Drupal is hardly a framework but more a CMS -- or maybe a CMS framework. I'm pretty sure that Drupal either has a thing build in or that there is a plugin for it.
With CodeIgniter you would start from scratch and Zend_Translate plays well with it.
I liked Drupal over Joomla. You should also look into DotNetNuke, out of the box it has lot of things that will meet your needs.
Checkout django-blocks. Has multi-language Menu, Flatpages and even has a simple Shopping Cart!!