Multibyte character processing in SigScheme

1. Overview

SigScheme’s multibyte character handling interface is basically based on R6RS Unicode character handlings. See also "R6RS conformance" section and "Characters" subsection of "R5RS conformance" of Specifications of SigScheme.

In addition to R6RS Unicode character handlings, SigScheme supports EUC-JP, EUC-CN, EUC-KR and Shift_JIS character encoding schemes and they can be used simultaneously. But no character encoding conversion method such as iconv is provided at now.

2. Current character codec

On SigScheme, characters and strings are processed in accordance with the parameter current character codec. Its initial value is specified by --enable-default-encoding option of the configure script and defaults to UTF-8. So characters and strings in SigScheme are treated as UTF-8 by default.

The value of current character codec can be checked by %%current-char-codec procedure of SigScheme extension.

sscm> (require-extension (sscm-ext))
sscm> (%%current-char-codec)
"UTF-8"

To specify another encoding as default character codec, pass -C option to sscm command as follows, or specify it in the second argument of scm_initialize().

$ sscm -C ISO-8859-1
$ sscm -C UTF-8
$ sscm -C EUC-JP
$ sscm -C EUC-CN
$ sscm -C EUC-KR
$ sscm -C Shift_JIS

provided? predicate can be used to know whether an encoding is enabled by the configuration or not.

(provided? "utf8")
(provided? "eucjp")
(provided? "euccn")
(provided? "euckr")
(provided? "sjis")

The ISO-8859-1 encoding is used as generic singlebyte character encoding and accepts any character represented in integer range 0-255, and always provided regardless of configuration.

Use with-char-codec procedure to switch to another encoding temporarily.

(define euc-A (with-char-codec "EUC-JP" (lambda () (integer->char #xa4a2))))

See also "Ports" section of this document for current character codec switching on I/O.

3. Characters

When the reader is working on an UTF-8 port:

#\あ     ==> #\あ  ;; U+3042 HIRAGANA LETTER A
#\x3042  ==> #\あ  ;; U+3042 HIRAGANA LETTER A

Conversion between character and integer is performed based on the value of %%current-char-codec. When %%current-char-codec is "UTF-8", integer value of an Unicode character corresponds to the Unicode code point.

When %%current-char-codec is "UTF-8":

(integer->char #x3042)  ==> #\あ   ;; U+3042 HIRAGANA LETTER A
(char->integer #\あ)    ==> 12354  ;; #x3042

If an integer value is invalid for the current codec, an error is caused.

(with-char-codec "UTF-8"      (lambda () (integer->char #x3042)))  ==> #\あ
(with-char-codec "ISO-8859-1" (lambda () (integer->char #x3042)))  ==> error

Since a character in SigScheme is internally represented as an integer value marked with character-object tag without any character encoding information, user is responsible to manage the character encoding scheme of each character object.

And no character encoding conversion is performed on integer<→char conversion regaradless of %%current-char-codec.

;; U+3042 HIRAGANA LETTER A in Unicode
(define ucs-A    (with-char-codec "UTF-8"  (lambda () (integer->char #x3042))))

;; HIRAGANA LETTER A in EUC-JP
(define eucjp-A  (with-char-codec "EUC-JP" (lambda () (integer->char #xa4a2))))

(eqv? ucs-A eucjp-A)  ==> #f

;; no conversion is performed
(with-char-codec "UTF-8"
  (lambda () (char->integer eucjp-A)))  ==> 42146  ;; U+A4A2 YI RADICAL ZUP

4. Strings

When both reader’s port and %%current-char-codec is UTF-8:

"\x3042;a\x3044;"        ==> "あaい"
(string->list "あaい")   ==> (#\あ #\a #\い)
(string-length "あaい")  ==> 3

A string in SigScheme is internally represented as a C string with its logical character length without character encoding information. User is responsible to manage the character encoding scheme of each string object. Though strings have no encoding information, they have logical character length counted in %%current-char-codec on its object creation. So processing a string in another encoding such as UTF-8 string as byte string cannot fully be performed.

;; U+3042 HIRAGANA LETTER A, with string length 1 counted in UTF-8
(define utf8-A "あ")

;; string length is not re-counted even if %%current-char-codec is changed
(with-char-codec "UTF-8"      (lambda () (string-length utf8-A)))  ==> 1
(with-char-codec "ISO-8859-1" (lambda () (string-length utf8-A)))  ==> 1

;; character reference of string is based on %%current-char-codec
(with-char-codec "UTF-8"      (lambda () (string-ref utf8-A 0)))  ==> #\あ
(with-char-codec "ISO-8859-1" (lambda () (string-ref utf8-A 0)))  ==> #\ã

;; character reference that exceeds logical length is an error even if its
;; physical length is enough
(with-char-codec "UTF-8"      (lambda () (string-ref utf8-A 1)))  ==> error
(with-char-codec "ISO-8859-1" (lambda () (string-ref utf8-A 1)))  ==> error

5. Identifiers

Any Unicode characters can be used as identifiers in Scheme as R6RS allows.

'Français-symbole
'日本語シンボル
(define ひらがな->カタカナ (lambda (イ . ロ) ...))

But since SigScheme’s Unicode handling is incomplete, all non-ASCII Unicode characters are treated as ordinary letter. Though it allows using any whitespace characters and punctuations as identifier, it should not be done.

Non-ASCII identifier in SigScheme is only allowed for Unicode. In other words, allowed only if the reading port is UTF-8. This limitation is intended to avoid character identity problem between different character encodings. For example, WAVE DASH and FULLWIDTH TILDE may be altered to another unexpectedly if the source code is converted to/from non-Unicode Japanese encoding. Inhibiting non-Unicode identifiers is the simplest way to avoid such problems.

To use Unicode identifiers, prepend following line to the source code. See also "Ports" section to understand its mechanism.

# /usr/bin/env sscm -C UTF-8

6. Ports

Each port is associated with a single character encoding when it is open

sscm> (current-input-port)
#<iport mb UTF-8 file stdin>

The encoding of a port is not switched to another once it is open (NOTE: a C extension that operates on low level port can alter it)
The encoding of a port is determined by %%current-char-codec on its opening

$ sscm -C UTF-8
sscm> (%%current-char-codec)
"UTF-8"

sscm> (open-output-file "/tmp/sigscheme.tmp")
#<oport mb UTF-8 file /tmp/sigscheme.tmp>

sscm> (with-char-codec "ISO-8859-1"
        (lambda () (open-output-file "/tmp/sigscheme.tmp")))
#<oport mb ISO-8859-1 file /tmp/sigscheme.tmp>

%%current-char-codec does not affect already open ports

sscm> (current-input-port)
#<iport mb UTF-8 file stdin>

sscm> (with-char-codec "ISO-8859-1"
        (lambda ()
          (list (%%current-char-codec) (current-input-port))))
("ISO-8859-1" #<iport mb UTF-8 file stdin>)

Characters output to a port is encoded to a byte stream according to the port’s own character encoding regardless of %%current-char-codec value

$ sscm -C ISO-8859-1
sscm> (current-output-port)
#<oport mb ISO-8859-1 file stdout>

sscm> (with-char-codec "UTF-8" (lambda () (write #\x3042)))
Error: ScmMultibyteCharPort: invalid character

To specify per-file character encoding, prepend following line into the file. The load procedure detects this line and temporarily switches %%current-char-codec and the encoding of the reading port until read all expressions from the file.

#! /usr/bin/env sscm -C UTF-8