pcre2 UTF32 usage - c++

Question

I've just spent some time figuring out the pcre2 interface and think I've got it for the most part. I want to support UTF32, pcre2 is already built with support and code point width has been set to 32.
The code below is what I've got for working with code point width set to 8.
How do I change this to work with UTF32?
#include "gtest/gtest.h"
#include <pcre2.h>
TEST(PCRE2, example) {
//iterate over all matches in a string
PCRE2_SPTR subject = (PCRE2_SPTR) string("this is it").c_str();
PCRE2_SPTR pattern = (PCRE2_SPTR) string("([a-z]+)|\\s").c_str();
int errorcode;
PCRE2_SIZE erroroffset;
pcre2_code *re = pcre2_compile(pattern, PCRE2_ZERO_TERMINATED, PCRE2_ANCHORED | PCRE2_UTF, &errorcode,
&erroroffset, NULL);
if (re) {
uint32_t groupcount = 0;
pcre2_pattern_info(re, PCRE2_INFO_BACKREFMAX, &groupcount);
pcre2_match_data *match_data = pcre2_match_data_create_from_pattern(re, NULL);
uint32_t options_exec = PCRE2_NOTEMPTY;
PCRE2_SIZE subjectlen = strlen((const char *) subject);
errorcode = pcre2_match(re, subject, subjectlen, 0, options_exec, match_data, NULL);
while (errorcode >= 0) {
PCRE2_UCHAR *result;
PCRE2_SIZE resultlen;
for (int i = 0; i <= groupcount; i++) {
pcre2_substring_get_bynumber(match_data, i, &result, &resultlen);
printf("Matched:%.*s\n", (int) resultlen, (const char *) result);
pcre2_substring_free(result);
}
// Advance through subject
PCRE2_SIZE *ovector = pcre2_get_ovector_pointer(match_data);
errorcode = pcre2_match(re, subject, subjectlen, ovector[1], options_exec, match_data, NULL);
}
pcre2_match_data_free(match_data);
pcre2_code_free(re);
} else {
// Syntax error in the regular expression at erroroffset
PCRE2_UCHAR error[256];
pcre2_get_error_message(errorcode, error, sizeof(error));
printf("PCRE2 compilation failed at offset %d: %s\n", (int) erroroffset, (char *) error);
}
Presumably subject and pattern needs to be converted somehow and result would be of the same type? I couldn't find anything in pcre2 header to indicate support for that.
And I guess subjectlen would no longer be simply strlen.
Finally, I put this example together from having gone through some of the docs and the header, is there anything else I should be doing/worth knowing.

Answer 1

I left pcre2 in the end, after evaluating RE2, PCRE2 and ICU, I chose ICU. Its unicode support (from what I've seen so far) much more complete than the other two. It also provides a very clean API and lots of utilities for manipulation. Importantly, like PCRE2 provides a perl style regex engine which, out of the box works great with unicode.

Answer 2

If you set code width properly this could be the problem:
(PCRE2_SPTR) string("this is it").c_str();
Converting the c_str() to PCRE2_SPTR doesn't make the string utf32.
If you are unsure about setting the proper code width (I didn't see it in your sorce code) you can force 32 bit by adding _32 postfix to everything e.g. pcre2_compile_32.

Answer 3

It depends on which character type you are going to use and which system you are going to target.
The basic unit for std::string is char which is generally 8 bit and supports UTF-8 (can be different depending on implementation/system). So you can't use std::string("some string") and such codes when dealing with UTF-32 in such systems.
PCRE2_CODE_UNIT_WIDTH must match the bit-size of the basic character unit you are going to use. For 8-bit char it should be defined as 8, for 16-bit char it should be defined as 16 etc...
In GNU/Linux, you can use wchar_t i.e std::wstring which is 32 bit and UTF-32 supported. In windows wchar_t is 16 bit (with UTF-16).
In >=C++11, you can use char32_t i.e std::u32string which is at least 32 bit (you will have to make sure it's exactly 32 bit in your target system)
I have a wrapper for PCRE2 in C++ which contains some examples (in src directory) on how to handle UTF-16 and UTF-32 modes.