Keith Devens .com

A regular expression to check for valid UTF-8

This is great. A regular expression that allows you to check if text is valid UTF-8. Via Sam Ruby.

I'd previously used a function I found in the PHP manual and reproduced here. I like the regular expression better for aesthetic reasons, because it appears more straightforward (and therefore it's easier to verify its correctness), and because it's easier to port to any language you need. It's possibly even faster than the other code. I'll reproduce it here in case that page ever goes away:

$field =~

  m/^(

     [\09\0A\0D\x20-\x7E]               # ASCII

   | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte

   |  \xE0[\xA0-\xBF][\x80-\xBF]        # excluding overlongs

   | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte

   |  \xED[\x80-\x9F][\x80-\xBF]        # excluding surrogates

   |  \xF0[\x90-\xBF][\x80-\xBF]{2}     # planes 1-3

   | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15

   |  \xF4[\x80-\x8F][\x80-\xBF]{2}     # plane 16

  )*$/x;

Excellent.

Update: Making that a non-capturing group would probably be better.

Permalink | 7 Comments | Tags: Programming | id 4982

18:52 (utc-4)
edited: 2004-06-30 13:57 (utc-4)

Comments

actually, after looking at rfc3629 and talking to sam ruby, it appears that this regex is, in fact, not valid. the rfc specifies that octets 0x00-0x7F are valid, being identical to their ASCII counterparts, but the regex excludes much of that range. it looks as though the utf-8 definition was conflated with the xml 1.0 character list to the detriment of the stated purpose ('is this valid utf-8?' determination).

Quick Q if someone can tell me. Is Â valid utf-8 or must it be   ? Trying to insert XML (unencoded) into a DB that was reportedly set for UTF-8. It is failing and I am trying to understand if I need to encode or the DBA should fix the DB.

Run the above regular expression against your XML to see if the Â is actually encoded in UTF-8 or whether it's Latin-1 or something. If the db is supposed to accept UTF-8 and your XML passes the regex, then the db is wrong. More likely, your XML isn't actually encoded in UTF-8.

Feel free to post a comment below. Please see my comment policy.

Formatting Rules (No HTML):

**bold**, *italic*, _underlined_, --strikeout--
"text"="url" creates a link, and URLs are auto-highlighted
Blockquote: Like e-mail, begin paragraph with > (greater-than sign)
Lists: begin paragraph with *,-, or + (unordered), or # (ordered)
Code block: ?!code:language=perl|php|sql|javascript|etc.{\n}...{\n}?!/code

Tuesday, June 29, 2004

A regular expression to check for valid UTF-8

Comments

Recent comments