Converting full-width characters to half-width characters - regex

Question

I have a program for converting full width characters to half width. It works fine, except for the number zero. Full-width zero is not converting to half-width zero.
Perl
use strict;
use warnings;
use warnings qw(FATAL utf8);
use utf8;
use feature qw(unicode_strings);
use open qw(:std :utf8);
unless ( #ARGV == 2 ) {
print "Usage: script.pl input_file output_file\n";
exit;
}
my %fwhw = (
'０' => '0', '１' => '1', '２' => '2', '３' => '3', '４' => '4',
'５' => '5', '６' => '6', '７' => '7', '８' => '8', '９' => '9',
'Ａ' => 'A', 'Ｂ' => 'B', 'Ｃ' => 'C', 'Ｄ' => 'D', 'Ｅ' => 'E',
'Ｆ' => 'F', 'Ｇ' => 'G', 'Ｈ' => 'H', 'Ｉ' => 'I', 'Ｊ' => 'J',
'Ｋ' => 'K', 'Ｌ' => 'L', 'Ｍ' => 'M', 'Ｎ' => 'N', 'Ｏ' => 'O',
'Ｐ' => 'P', 'Ｑ' => 'Q', 'Ｒ' => 'R', 'Ｓ' => 'S', 'Ｔ' => 'T',
'Ｕ' => 'U', 'Ｖ' => 'V', 'Ｗ' => 'W', 'Ｘ' => 'X', 'Ｙ' => 'Y',
'Ｚ' => 'Z', 'ａ' => 'a', 'ｂ' => 'b', 'ｃ' => 'c', 'ｄ' => 'd',
'ｅ' => 'e', 'ｆ' => 'f', 'ｇ' => 'g', 'ｈ' => 'h', 'ｉ' => 'i',
'ｊ' => 'j', 'ｋ' => 'k', 'ｌ' => 'l', 'ｍ' => 'm', 'ｎ' => 'n',
'ｏ' => 'o', 'ｐ' => 'p', 'ｑ' => 'q', 'ｒ' => 'r', 'ｓ' => 's',
'ｔ' => 't', 'ｕ' => 'u', 'ｖ' => 'v', 'ｗ' => 'w', 'ｘ' => 'x',
'ｙ' => 'y', 'ｚ' => 'z', '－' => '-', '、' => ', ', '　' => ' ',
'／' => '/',);
sub slurp {
my $file = shift;
open my $fh_read, '<', $file or die "Could not open file: $!";
return do {local $/; <$fh_read>};
}
sub convert {
my $sub_string = shift;
$sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;
return $sub_string;
}
my $string = slurp($ARGV[0]);
$string =~ s/<target>\s*<g id="\d+">\K(.*?)(?=<\/g>\s*<\/target>)/convert($1)/seg;
open my $fh_write, ">", $ARGV[1] or die "Could not open file: $!";
print $fh_write $string;
close $fh_write;
Here is what I have tried
I have made sure that the number 0 (zero) and the letter O (oh) are indeed different by checking their code points. Full width 0 is \x{ff10}. Full width letter O is \x{ff2f}. I checked this using this code:
use Encode;
sub codepoint_hex {
sprintf "%04x", ord Encode::decode("UTF-8", shift);
}
my $codepoint = codepoint_hex('０');
print $codepoint, "\n";
I have checked that the hash is indeed loading all of the keys and values correctly.
What I haven't tried yet:
I haven't tried to duplicate this oddity on Linux yet. I am using ActiveState Perl 5.24 on Windows 10.
If anyone has any suggestions or sees my mistake, I would be very grateful for the guidance.

Answer 1

Since $fwhw{'０'} returns 0, and since 0 is false, the replacement doesn't occur. Replace
$sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;
with
$sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;
If that still doesn't work, use sprintf "%vX", $str to see what you really have.
By the way,
sub convert {
my $sub_string = shift;
$sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;
return $sub_string;
}
would be much faster if replaced with
sub convert {
state $chars = join '', keys(%fwhw);
state $re = qr/([\Q$chars\E])/;
return $_[0] =~ s/$re/$fwhw{$1}/gr;
}
Faster yet,
sub convert {
state $s = join '', keys(%fwhw);
state $r = join '', values(%fwhw);
state $tr = eval("sub { $_[0] =~ tr/\Q$s\E/\Q$r\E/r }");
return $tr->($_[0]);
}

Answer 2

You don't need such a huge dictionary with lots of supporting functions like that. Just a simple sed is enough
halfwidth='!"#$%&'\''()*+,-.\/0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ '
fullwidth='！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　'
sed -ie "y/$fullwidth/$halfwidth/" your_file
If you want to do that in perl it's pretty simple too
perl -Mutf8 -i -C -pe 'BEGIN{ use open qw/:std :utf8/; } tr#！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　#!"\#$%&'\''()*+,-.\/0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ # your_file'