bradchoate.com

Available here.

The weird problem that this update addresses is a glitch in a particular edition of Perl. Specifically 5.8.0 with the 18379 maintenance patch. For example, that version of Perl causes this to break:

use Encode;
my $str = 'a';
$str = Encode::decode('iso-8859-1', $str);
if ($str =~ m/^([^\s]*?)$/) {
  print "found $1\n";
} else {
  print "no match\n";
}

Under most versions of Perl (that have utf8 string support), this will print:

found a

But for Perl 5.8.0, with the 18379 maintenance patch, it prints:

no match

Very odd indeed. Thats a very basic regex failure for utf8 strings, don't you think?

In fact, you can eliminate the decode operation and produce the same failure by changing it to this:

Encode::_utf8_on($str);

Which just flips the little 'utf8' flag on $str. This should have no adverse effect to 7-bit ASCII data, but it does alter how that regex operation matches.

In any case, this particular issue has been a real annoyance. In MT-Textile, what I want to do is this:

1. Take the raw byte data as provided by Movable Type
2. Decode the data to native a Perl wide-character string (only if the user has Encode and if their PublishCharset is defined in mt.cfg)
3. Do everything Textile would do (safely operating against multibyte characters)
4. If step 2 happened, encode the data returned by Textile back to the character set identified by PublishCharset.
5. Return string back to Movable Type

But with this bug, I've been forced to eliminate steps 2 and 4 which means that Textile may or may not properly handle multibyte data. That being said, I have not seen any problems with my (limited) tests with Japanese, Chinese, Korean, etc. But if you do, please report any problems you may encounter to the bugtracker.