So I've finally gotten back to my main task - porting a rather large C++ project from Windows to the Mac.
Straight away I've been hit by the problem where wchar_t is 16-bits on Windows but 32-bits on the Mac. This is a problem because all of the strings are represented by wchar_t and there will be string data going back and forth between Windows and Mac machines (in both on-disk data and network data forms). Because of the way in which it works it wouldn't be totally straightforward to convert the strings into some common format before sending and receiving the data.
We've also really started to support a lot more languages recently and so we're starting to deal with a lot of Unicode data (as well as dealing with right-to-left languages).
Now, I could be conflating multiple ideas here and causing more problems for myself than needed which is why I'm asking this question. We're thinking that storing all of our in-memory string data as UTF-8 makes a lot of sense. It solves the wchar_t being different sizes problem, it means we can easily support multiple languages and it also dramatically reduces our memory footprint (we have a LOT of - mostly English - strings loaded) - but it doesn't seem like many people are doing this. Is there something we're missing? There's the obvious problem you have to deal with where string length can be less than the memory size storing that string data.
Or is using UTF-16 a better idea? Or should we stick to wchar_t and write code to convert between wchar_t and, say, Unicode in places where we read/write to the disk or the network?
I realize this is dangerously close to asking for opinions - but we're nervous that we're overlooking something obvious because it doesn't seem like there are many Unicode string classes (for example) - but yet there's plenty of code for converting to/from Unicode like in boost::locale, iconv, utf-cpp and ICU.
Always use a protocol defined to the byte when a file or network connection is involved. Do not rely on how a C++ compiler stores anything in memory. For Unicode text, this means choosing both an encoding and a byte order (okay, UTF-8 doesn't care about byte order). Even if the platforms you currently want to support have similar architectures, another popular platform with different behavior or even a new OS for one of your existing platforms will likely come along, and you'll be glad you wrote portable code.
I tend to use UTF-8 as the internal representation. You only lose string length checking, with isn't really useful anyways. For Windows API conversion, I use my own Win32 conversion functions I devised here. As Mac and linux are (for the most part standard UTF-8-aware, no need to convert anything there). Free bonuses you get:
use plain old std::string.
byte-wise network/stream transport.
For most languages, nice memory footprint.
For more functionality: utf8cpp
As a rule of thumb: UTF-16 for processing, UTF-8 for communication & storage.
Sure, any rule can be broken and this one is not carved in stone.
But you have to know when it is ok to break it.
For instance it might be a good idea to use something else if the environment you are using wants something else. But Mac OS X APIs use UTF-16, same as Windows. So UTF-16 makes more sense.
It is more straightforward to convert before you put/get things on the net (because you probably do it in 2-3 routines) than doing all the conversions to call OS APIs.
It also matter the type of application you develop.
If it is something with very little text processing, and very little calls to the system (something like an email server that mostly moves things around without changing them), then UTF-8 might be a good choice.
So, as much as you might hate this answer, "it depends".
ICU has a C++ string class, UnicodeString
Related
I only have experience with processing ASCII (single byte characters) and have read a number of posts on how people process Unicode differently which present their own set of issues.
At this point of my very limited exposure to Unicode, I’ve read that internal processing with UTF-16 presents portability and other issues.
I feel that UTF-32 makes more sense than UTF-16 since all Unicode characters fit within 4 bytes but would consume more resources, especially if you are mainly dealing with ISO-8859-1 characters.
I humbly feel that UTF-8 could be an ideal format to work with internally (especially for case where you deal mainly with English and Latin based characters) since the ASCII range of characters would be handled byte by byte very efficiently. Characters from the Latin alphabet would consume two bytes and other characters would consume more bytes of course.
Another advantage that I see is that UTF-8 strings could be stored within regular C++ std::string or C string arrays which seems so natural.
The disadvantage for using UTF-8 for me at least is that I have not found any libraries to support UTF-8 internally. For example, I have not found any libraries for UTF-8 case conversion and substring operations.
Another disadvantage for me is that I have not found functions to parse bytes within a UTF-8 string for character processing.
Would it be feasible to work with UTF-8 internally and are there any support libraries available for this purpose? I do hope so but if not, I think that my best option would be to forget using UTF-8 internally and use Boost::Locale since I’ve read that ICU is a mature library used by many to handle Unicode.
I would really like to hear your opinions on this matter.
I bumped into my very old answer and I'll tell you what I ended up doing. I decided to stick with UTF-8 and store my data in std::string or single byte char arrays. There was never a need for me to use multi-byte characters!
The first library that I used was UTF8-CPP which is very easy to bring into your app and use. But you soon find that you need more and more capability.
I really wanted to avoid using ICU because it is such a large library, but once you build it and get it installed, you begin to wish that you had done it in the first place because it has everything you need and much, much more.
What are my benefits you may wonder:
I write truly portable code that builds under VC++ for Windows or GCC for Linux.
ICU has everything, and I mean everything you need concerning unicode.
I am able to stick with my beloved std::string and char arrays.
I use many open source libraries in my apps with zero issues. For example, I use RapidJson for my JSON to create in-memory JSON objects containing UTF-8 data. I'm able to pass them to a web server or write them to disk, etc. Really simple.
I store my data into Firebird SQL but you need to specify your varchar and char field types as UTF8. This means that your strings will be stored as mutli-byte in the database. But this is totally transparent to you, the developer. I am certain that this applies to other SQL databases as well.
Drawbacks:
Large library, very scary and confusing at first.
The C++ was not written by C++ experts (like the Boost developers). But the code is totally stable and fast. You may not like the syntax used though. What I've done is to "wrap" common procedures with my code. This pretty much means that I include my own UTF-8 library which wraps the ICU uglies. Don't let this bother you because ICU is totally stable and fast.
I personally dynamically link ICU into my applications. This means that I first built ICU dynamically for my Win and Linux 64 bit environments. In the case of Windows, I store the dlls in a folder somewhere and add that to my Windows path so that any app that requires ICU can find the dlls.
When I looked at built-in language features, I found several lacking such as lower/upper case conversion, word boundaries, counting characters, accent sensitivity, string manipulation such as substrings, etc. Local support is also totally amazing.
I guess that summarizes entire exercise in UTF-8.
We are specifically eyeing Windows and Linux development, and have come up with two differing approaches that both seem to have their merits. The natural unicode string type in Windows is UTF-16, and UTF-8 in linux.
We can't decide whether the best approach:
Standardise on one of the two in all our application logic (and persistent data), and make the other platforms do the appropriate conversions
Use the natural format for the OS for application logic (and thus making calls into the OS), and convert only at the point of IPC and persistence.
To me they seem like they are both about as good as each other.
and UTF-8 in linux.
It's mostly true for modern Linux. Actually encoding depends on what API or library is used. Some hardcoded to use UTF-8. But some read LC_ALL, LC_CTYPE or LANG environment variables to detect encoding to use (like Qt library). So be careful.
We can't decide whether the best approach
As usual it depends.
If 90% of code is to deal with platform specific API in platform specific way, obviously it is better to use platform specific strings. As an example - a device driver or native iOS application.
If 90% of code is complex business logic that is shared across platforms, obviously it is better to use same encoding on all platforms. As an example - chat client or browser.
In second case you have a choice:
Use cross platform library that provides strings support (Qt, ICU, for example)
Use bare pointers (I consider std::string a "bare pointer" too)
If working with strings is a significant part of your application, choosing a nice library for strings is a good move. For example Qt has a very solid set of classes that covers 99% of common tasks. Unfortunately, I has no ICU experience, but it also looks very nice.
When using some library for strings you need to care about encoding only when working with external libraries, platform API or sending strings over the net (or disk). For example, a lot of Cocoa, C# or Qt (all has solid strings support) programmers know very little about encoding details (and it is good, since they can focus on their main task).
My experience in working with strings is a little specific, so I personally prefer bare pointers. Code that use them is very portable (in sense it can be easily reused in other projects and platforms) because has less external dependencies. It is extremely simple and fast also (but one probably need some experience and Unicode background to feel that).
I agree that bare pointers approach is not for everyone. It is good when:
You work with entire strings and splitting, searching, comparing is a rare task
You can use same encoding in all components and need a conversion only when using platform API
All your supported platforms has API to:
Convert from your encoding to that is used in API
Convert from API encoding to that is used in your code
Pointers is not a problem in your team
From my a little specific experience it is actually a very common case.
When working with bare pointers it is good to choose encoding that will be used in entire project (or in all projects).
From my point of view, UTF-8 is an ultimate winner. If you can't use UTF-8 - use strings library or platform API for strings - it will save you a lot of time.
Advantages of UTF-8:
Fully ASCII compatible. Any ASCII string is a valid UTF-8 string.
C std library works great with UTF-8 strings. (*)
C++ std library works great with UTF-8 (std::string and friends). (*)
Legacy code works great with UTF-8.
Quite any platform supports UTF-8.
Debugging is MUCH easier with UTF-8 (since it is ASCII compatible).
No Little-Endian/Big-Endian mess.
You will not catch a classical bug "Oh, UTF-16 is not always 2 bytes?".
(*) Until you need to lexical compare them, transform case (toUpper/toLower), change normalization form or something like this - if you do - use strings library or platform API.
Disadvantage is questionable:
Less compact for Chinese (and other symbols with large code point numbers) than UTF-16.
Harder (a little actually) to iterate over symbols.
So, I recommend to use UTF-8 as common encoding for project(s) that doesn't use any strings library.
But encoding is not the only question you need to answer.
There is such thing as normalization. To put it simple, some letters can be represented in several ways - like one glyph or like a combination of different glyphs. The common problem with this is that most of string compare functions treat them as different symbols. If you working on cross-platform project, choosing one of normalization forms as standard is a right move. This will save your time.
For example if user password contains "йёжиг" it will be differently represented (in both UTF-8 and UTF-16) when entered on Mac (that mostly use Normalization Form D) and on Windows (that mostly likes Normalization Form C). So if user registered under Windows with such password it will a problem for him to login under Mac.
In addition I would not recommend to use wchar_t (or use it only in windows code as a UCS-2/UTF-16 char type). The problem with wchar_t is that there is no encoding associated with it. It's just an abstract wide char that is larger than normal char (16 bits on Windows, 32 bits on most *nix).
I'd use the same encoding internally, and normalize the data at entry point. This will involve less code, less gotchas, and will allow you to use the same cross platform library for string processing.
I'd use unicode (utf-16) because it's simpler to handle internally and should perform better because of the constant length for each character. UTF-8 is ideal for output and storage because it's backwards compliant with latin ascii, and unly uses 8 bits for English characters. But inside the program 16-bit is simpler to handle.
C++11 provides the new string types u16string and u32string. Depending on the support your compiler versions deliver, and the expected life expectancy, it might be an idea to stay forward-compatible to those.
Other than that, using the ICU library is probably your best shot at cross-platform compatibility.
This seems to be quite enlightening on the topic. http://www.utf8everywhere.org/
Programming with UTF-8 is difficult as lengths and offsets are mixed up. e.g.
std::string s = Something();
std::cout << s.substr(0, 4);
does not necessarily find the first 4 chars.
I would use whatever a wchar_t is. On Windows that will be UTF-16. On some *nix platforms it might be UTF-32.
When saving to a file, I would recommend converting to UTF-8. That often makes the file smaller, and removes any platform dependencies due to differences in sizeof(wchar_t) or to byte order.
I'm targeting Windows but I don't see any reason why some API code I'm writing cannot use basic C++ types. What I want to do is expose methods that return strings and ints. In the C# world I'd just use string, and have a unicode string, but in VC++ I've got the option of using std::string, std::wstring, or MFC/ATL CStrings.
Should I just use std::wstring exclusively to support unicode, or can I use std::string which would be compiled to unicode based on my build settings? I'm leaning toward the latter. I'd prefer to provide Get[Item]AsCString() methods on my objects for other string types.
Also should I be using size_t instead of integer?
The API is going to be used by me, and perhaps a future developer working on the C++ GUI. It is a way to separate concerns. My preferences:
Intuitiveness for other developers.
Forward compatibility with VC++
Compatibility with other C++ compilers
Performance (this is a lesser concern for me, but need the startup time for rest of my app)
Any guides would be appreciated.
You should probably stick to the STL string type. The MFC CString class is built on top of that nowadays anyway.
As has been noted before, using wstring is not a magic bullet to address Unicode issues since there are many Unicode characters that still require multiple wchars to encode.
Using Utf-8 instead has potential benefits (you don't have to worry about endianness for example).
On Windows, all modern kernels are wchar based, so there is a (minimal) performance overhead involved if you use the 8bit char versions of APIs.
In your situation it would take me few hours / days to develop an opinion and decide. First of all, I very much prefer C_API to C++_API, even for C++ code. Then the answer would be char*, or wchar*, or TCHAR*. Now, try to guess if you REALLY expect the need for UNICODE. Great majority of my projects (including those with GUIs), had no need for UNICODE, the simplicity and familiarity of plain C-arrays is often hard to beat.
In short, try to predict what will be your needs, do not try to look too far into future (2 years is a good mark), then come up with the simplest solution to meet the needs.
Last: To answer your question more directly, I would start with std::string as my 1st choice to evaluate. Unless I would find some bid advantage in favor of the other choices, I would stay with it.
Using std::wstring/string instead of the MFC CString will allow you to port your code to other frameworks (e.g. Qt for Windows).
Even when using std::string you could encode the strings in UTF-8, so your API will still be able to return UNICODE strings.
Keep in mind that even wstring is really UTF-16 and not the full 32 bits UNICODE (while on some operating systems wstring is UTF-32).
Suppose we have an arbitrary string, s.
s has the property of being from just about anywhere in the world. People from USA, Japan, Korea, Russia, China and Greece all write into s from time to time. Fortunately we don't have time travellers using Linear A, however.
For the sake of discussion, let's presume we want to do string operations such as:
reverse
length
capitalize
lowercase
index into
and, just because this is for the sake of discussion, let's presume we want to write these routines ourselves (instead of grabbing a library), and we have no legacy software to maintain.
There are 3 standards for Unicode: utf-8, utf-16, and utf-32, each with pros and cons. But let's say I'm sorta dumb, and I want one Unicode to rule them all (because rolling a dynamically adapting library for 3 different kinds of string encodings that hides the difference from the API user sounds hard).
Which encoding is most general?
Which encoding is supported by wchar_t?
Which encoding is supported by the STL?
Are these encodings all(or not at all) null-terminated?
--
The point of this question is to educate myself and others in useful and usable information for Unicode: reading the RFCs is fine, but there's a 'stack' of information related to compilers, languages, and operating systems that the RFCs do not cover, but is vital to know to actually use Unicode in a real app.
Which encoding is most general
Probably UTF-32, though all three formats can store any character. UTF-32 has the property that every character can be encoded in a single codepoint.
Which encoding is supported by wchar_t
None. That's implementation defined. On most Windows platforms it's UTF-16, on most Unix platforms its UTF-32.
Which encoding is supported by the STL
None really. The STL can store any type of character you want. Just use the std::basic_string<t> template with a type large enough to hold your code point. Most operations (e.g. std::reverse) do not know about any sort of unicode encoding though.
Are these encodings all(or not at all) null-terminated?
No. Null is a legal value in any of those encodings. Technically, NULL is a legal character in plain ASCII too. NULL termination is a C thing -- not an encoding thing.
Choosing how to do this has a lot to do with your platform. If you're on Windows, use UTF-16 and wchar_t strings, because that's what the Windows API uses to support unicode. I'm not entirely sure what the best choice is for UNIX platforms but I do know that most of them use UTF-8.
Have a look at the open source library ICU, especially at the Docs & Papers section. It's an extensive library dealing with all sorts of unicode oddities.
In response to your final bullet, UTF-8 is guaranteed not to have NULL bytes in its encoding of any character (except NULL itself, of course). As a result, many functions that work with NULL-terminated strings also work with UTF-8 encoded strings.
Define "real app" :)
Seriously, the decision really depends a lot on the kind of software you are developing. If your target platform is Win32 API (with or without wrappers such as MFC, WTL, etc) you would probably want to use wstring types with the text encoded as UTF-16. That's simply because all Win32 API internally uses that encoding anyway.
On another hand, if your output is something like XML/HTML and/or needs to be delivered over the internet, UTF-8 is pretty much the standard - it is usually transmitted well via protocols that make assumptions about characters having 8 bits.
As for UTF-32, I can't think of a single reason to use it, unless you need 1:1 mapping between code units and code points (that still does not mean 1:1 mapping between code units and characters!).
For more information, be sure to look at Unicode.org. This FAQ may be a good starting point.
I'm still trying to decide whether my (home) project should use UTF-8 strings (implemented in terms of std::string with additional UTF-8-specific functions when necessary) or some 16-bit string (implemented as std::wstring). The project is a programming language and environment (like VB, it's a combination of both).
There are a few wishes/constraints:
It would be cool if it could run on limited hardware, such as computers with limited memory.
I want the code to run on Windows, Mac and (if resources allow) Linux.
I'll be using wxWidgets as my GUI layer, but I want the code that interacts with that toolkit confined in a corner of the codebase (I will have non-GUI executables).
I would like to avoid working with two different kinds of strings when working with user-visible text and with the application's data.
Currently, I'm working with std::string, with the intent of using UTF-8 manipulation functions only when necessary. It requires less memory, and seems to be the direction many applications are going anyway.
If you recommend a 16-bit encoding, which one: UTF-16? UCS-2? Another one?
UTF-16 is still a variable length character encoding (there are more than 2^16 unicode codepoints), so you can't do O(1) string indexing operations. If you're doing lots of that sort of thing, you're not saving anything in speed over UTF-8. On the other hand, if your text includes a lot of codepoints in the 256-65535 range, UTF-16 can be a substantial improvement in size. UCS-2 is a variation on UTF-16 that is fixed length, at the cost of prohibiting any codepoints greater than 2^16.
Without knowing more about your requirements, I would personally go for UTF-8. It's the easiest to deal with for all the reasons others have already listed.
I have never found any reasons to use anything else than UTF-8 to be honest.
If you decide to go with UTF-8 encoding, check out this library: http://utfcpp.sourceforge.net/
It may make your life much easier.
I've actually written a widely used application (5million+ users) so every kilobyte used adds up, literally. Despite that, I just stuck to wxString. I've configured it to be derived from std::wstring, so I can pass them to functions expecting a wstring const&.
Please note that std::wstring is native Unicode on the Mac (no UTF-16 needed for characters above U+10000), and therefore it uses 4 bytes/wchar_t. The big advantage of this is that i++ gets you the next character, always. On Win32 that is true in only 99.9% of the cases. As a fellow programmer, you'll understand how little 99.9% is.
But if you're not convinced, write the function to uppercase a std::string[UTF-8] and a std::wstring. Those 2 functions will tell you which way is insanity.
Your on-disk format is another matter. For portability, that should be UTF-8. There's no endianness concern in UTF-8, nor a discussion over the width (2/4). This may be why many programs appear to use UTF-8.
On a slightly unrelated note, please read up on Unicode string comparisions and normalization. Or you'll end up with the same bug as .NET, where you can have two variables föö and föö differing only in (invisible) normalization.
I would recommend UTF-16 for any kind of data manipulation and UI.
The Mac OS X and Win32 API uses UTF-16, same for wxWidgets, Qt, ICU, Xerces, and others.
UTF-8 might be better for data interchange and storage.
See http://unicode.org/notes/tn12/.
But whatever you choose, I would definitely recommend against std::string with UTF-8 "only when necessary".
Go all the way with UTF-16 or UTF-8, but do not mix and match, that is asking for trouble.
MicroATX is pretty much a standard PC motherboard format, most capable of 4-8 GB of RAM. If you're talking picoATX maybe you're limited to 1-2 GB RAM. Even then that's plenty for a development environment. I'd still stick with UTF-8 for reasons mentioned above, but memory shouldn't be your concern.
From what I've read, it's better to use a 16-bit encoding internally unless you're short on memory. It fits almost all living languages in one character
I'd also look at ICU. If you're not going to be using certain STL features of strings, using the ICU string types might be better for you.
Have you considered using wxStrings? If I remember correctly, they can do utf-8 <-> Unicode conversions and it will make it a bit easier when you have to pass strings to and from the UI.