Regexp doesn't work for specific special characters in Perl - regex

Question

I can't get rid of the special character ¤ and ❤ in a string:
$word = 'cɞi¤r$c❤u¨s';
$word =~ s/[^a-zöäåA-ZÖÄÅ]//g;
printf "$word\n";
On the second line I try to remove any non alphabetic characters from the string $word. I would expect to get the word circus printed out but instead I get:
ci�rc�us
The öäå and ÖÄÅ in the expression are just normal characters in the Swedish alphabet that I need included.

Answer 1

If the characters are in your source code, be sure to use utf8. If they are being read from a file, binmode $FILEHANDLE, ':utf8'.
Be sure to read perldoc perlunicode.

Answer 2

Short answer: add use utf8; to make sure your literal string in the source code are interepreted as utf8, that includes the content of the test string, and the content of the regexp.
Long answer:
#!/usr/bin/env perl
use warnings;
use Encode;
my $word = 'cɞi¤r$c❤u¨s';
foreach my $char (split //, $word) {
print ord($char) . Encode::encode_utf8(":$char ");
}
my $allowed_chars = 'a-zöäåA-ZÖÄÅ';
print "\n";
foreach my $char (split //, $allowed_chars) {
print ord($char) . Encode::encode_utf8(":$char ");
}
print "\n";
$word =~ s/[^$allowed_chars]//g;
printf Encode::encode_utf8("$word\n");
Executing it without utf8:
$ perl utf8_regexp.pl
99:c 201:É 158: 105:i 194:Â 164:¤ 114:r 36:$ 99:c 226:â 157: 164:¤ 117:u 194:Â 168:¨ 115:s
97:a 45:- 122:z 195:Ã 182:¶ 195:Ã 164:¤ 195:Ã 165:¥ 65:A 45:- 90:Z 195:Ã 150: 195:Ã 132: 195:Ã 133:
ci¤rc¤us
Executing it with utf8:
$ perl -Mutf8 utf8_regexp.pl
99:c 606:ɞ 105:i 164:¤ 114:r 36:$ 99:c 10084:❤ 117:u 168:¨ 115:s
97:a 45:- 122:z 246:ö 228:ä 229:å 65:A 45:- 90:Z 214:Ö 196:Ä 197:Å
circus
Explanation:
The non-ascii characters that you are typing into your source code are represented by one than more byte. Since your input is utf8 encoded. In a pure ascii or latin-1 terminal the characters would've been one byte.
When not using utf8 module, perl thinks that each and every byte you are inputting is a separate character, like you can see when doing the splitting and printing each and every individual character. When using the utf8 module, it treats the combination of several bytes as one character correctly according to the rules of utf8 encoding.
As you can see by coinscidence, some of the bytes that the swedish characters are made up of match with some of the bytes that some of the characters in your test string are made up of, and they are kept. Namely: the ö which in utf8 consists of 195:Ã 164:¤ - The 164 ends up as one of the characters you allow and it passes thru.
The solution is to tell perl that your strings are supposed to be considered as utf-8.
The encode_utf8 calls are in place to avoid warnings about wide characters being printed to the terminal. As always you need to decode input, and encode output according to the character encoding that input or output is supposed to handle/operate in.
Hope this made it clearer.

Answer 3

As pointed out by choroba, adding this in the beginning of the perl script solves it:
use utf8;
binmode(STDOUT, ":utf8");
where use utf8 lets you use the special characters correctly in the regular expression and binmode(STDOUT, ":utf8") lets you output the special characters correctly on the shell.