InternetCanonicalizeUrl fails to decode diacritic letters - c++

Question

I'm having lots of troubles while dealing with some characters into a URL, let's suppose that I have the following URL:
http: //localhost/somewere/myLibrary.dll/rest/something?parameter=An%C3%A1lisis
Which must be converted to:
http: //localhost/somewere/myLibrary.dll/rest/something?parameter=Análisis
In order to deal with the decoding of diacritic letters, I've decided to use the InternetCanonicalizeUrl function, because the application that I'm working on is going to work only in Windows and I don't want to install additional libraries, the helper function I've used is the following:
String DecodeURL(const String &a_URL)
{
String result;
unsigned long size = a_reportType.Length() * 2;
wchar_t *buffer = new wchar_t[size];
if (InternetCanonicalizeUrlW(a_URL.c_str(), buffer, &size, ICU_DECODE | ICU_NO_ENCODE))
{
result = buffer;
}
delete [] buffer;
return result;
}
That works kind of well for almost any of the URL passed through it, except for diacritic letters, my example URL is decoded as follows:
http: //localhost/somewere/myLibrary.dll/rest/something?parameter=Análisis
The IDE I'm working with is CodeGear™ C++Builder® 2009 (that's why I'm forced to use String instead of std::string), I've also tried with a AnsiString and char buffer version with the same results.
Any hint/alternative about how to deal with this error?
Thanks in advance.

Answer 1

InternetCanonicalizeUrl() is doing the right thing, you just have to take into account what it is actually doing.
URLs do not support Unicode (IRIs do), so Unicode data has to be charset-encoded into byte octets and then those octets are url-encoded using %HH sequences as needed. In this case, the data was encoded as UTF-8 (not uncommon in many URLs nowadays, but also not guaranteed), but InternetCanonicalizeUrl() has no way of knowing that as URLs do not have a syntax for describing which charset is being used. All it can do is decode %HH sequences to the relevant byte octet values, it cannot charset-decode the octets for you. In the case of the Unicode version, InternetCanonicalizeUrlW() returns those byte values as-is as wchar_t elements. But either way, you have to charset-decode the octets yourself to recover the original Unicode data.
So what you can do in this case is copy the decoded data to a UTF8String and then assign/return that as a String so it gets decoded to UTF-16. That will only work for UTF-8 encoded URLs, of course. For example:
String DecodeURL(const String &a_URL)
{
DWORD size = 0;
if (!InternetCanonicalizeUrlW(a_URL.c_str(), NULL, &size, ICU_DECODE | ICU_NO_ENCODE))
{
if (GetLastError() == ERROR_INSUFFICIENT_BUFFER)
{
String buffer;
buffer.SetLength(size-1);
if (InternetCanonicalizeUrlW(a_URL.c_str(), buffer.c_str(), &size, ICU_DECODE | ICU_NO_ENCODE))
{
UTF8String utf8;
utf8.SetLength(buffer.Length());
for (int i = 1; i <= buffer.Length(); ++i)
utf8[i] = (char) buffer[i];
return utf8;
}
}
}
return String();
}
Alternatively:
// encoded URLs are always ASCII, so it is safe
// to pass an encoded URL UnicodeString as an
// AnsiString...
String DecodeURL(const AnsiString &a_URL)
{
DWORD size = 0;
if (!InternetCanonicalizeUrlA(a_URL.c_str(), NULL, &size, ICU_DECODE | ICU_NO_ENCODE))
{
if (GetLastError() == ERROR_INSUFFICIENT_BUFFER)
{
UTF8String buffer;
buffer.SetLength(size-1);
if (InternetCanonicalizeUrlA(a_URL.c_str(), buffer.c_str(), &size, ICU_DECODE | ICU_NO_ENCODE))
{
return utf8;
}
}
}
FYI, C++Builder ships with Indy pre-installed. Indy has a TIdURI class, which can decode URL and take charsets into account, eg:
#include <IdGlobal.hpp>
#include <IdURI.hpp>
String DecodeURL(const String &a_URL)
{
return TIdURI::URLDecode(URL, enUTF8);
}
In any case, you have to know the charset used to encode the URL data. If you do not, all you can do is decode the raw octets and then use heuristic analysis to guess what the charset might be, but that is not 100% reliable for non-ASCII and non-UTF charsets.