1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
|
<!-- vim: set sw=2 et sts=2 ft=xml: -->
<!-- Last content review: 2024-01-21T08:10:39 UTC -->
<!--
https://wiki.archlinux.org/title/Xorg/Keyboard_configuration
https://www.x.org/releases/X11R7.5/doc/input/XKB-Config.html
ibus no XKB
-->
<chapter id="_i18n_and_l10n">
<title>I18N and L10N</title>
<para><ulink url="https://en.wikipedia.org/wiki/Internationalization_and_localization">Multilingualization (M17N) or Native Language Support</ulink> for an application software is done in 2 steps.</para>
<itemizedlist>
<listitem> <para> Internationalization (I18N): To make a software potentially handle multiple locales. </para> </listitem>
<listitem> <para> Localization (L10N): To make a software handle an specific locale. </para> </listitem>
</itemizedlist>
<tip> <para>There are 17, 18, or 10 letters between "m" and "n", "i" and "n", or "l" and "n" in multilingualization, internationalization, and localization which correspond to M17N, I18N, and L10N. See <ulink url="https://en.wikipedia.org/wiki/Internationalization_and_localization">Internationalization and localization</ulink> for details.</para> </tip>
<section id="_the_locale">
<title>The locale</title>
<para>The behavior of programs supporting internationalization are configured by the environment variable "<literal>$LANG</literal>" to support localization. Actual support of locale dependent features by the <literal>libc</literal> library requires to install <literal>locales</literal> or <literal>locales-all</literal> packages. The <literal>locales</literal> package requires to be initialized properly.</para>
<para>If neither <literal>locales</literal> or <literal>locales-all</literal> package are installed, support of locale features are lost and system uses US English messages and handles data as <emphasis role="strong">ASCII</emphasis>. This behavior is the same way as "<literal>$LANG</literal>" is set by "<literal>LANG=</literal>", "<literal>LANG=C</literal>", or "<literal>LANG=POSIX</literal>".</para>
<para>The modern software such as GNOME and KDE are multilingualized. They are internationalized by making them handle <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink> data and localized by providing their translated messages through the <literal>gettext</literal>(1) infrastructure. Translated messages may be provided as separate localization packages.</para>
<para>The current Debian desktop GUI system normally sets the locale under GUI environment as "<literal>LANG=xx_YY.UTF-8</literal>". Here, "<literal>xx</literal>" is <ulink url="https://en.wikipedia.org/wiki/ISO_639-1">ISO 639 language codes</ulink> and "<literal>YY</literal>" is <ulink url="https://en.wikipedia.org/wiki/ISO_3166-1">ISO 3166 country codes</ulink>. These values are set by the desktop configuration GUI dialogue and change the program behavior. See <xref linkend="_the_literal_lang_literal_variable"/></para>
<section id="_rationale_for_utf_8_locale">
<title>Rationale for UTF-8 locale</title>
<para>The simplest representation of the text data is <emphasis role="strong">ASCII</emphasis> which is sufficient for English and uses less than 127 characters (representable with 7 bits).</para>
<para>Even plain English text may contain non-ASCII characters, e.g. slightly curly left and right quotation marks are not available in ASCII.</para>
<screen>“double quoted text” is not "double quoted ASCII"
‘single quoted text’ is not 'single quoted ASCII'</screen>
<para>In order to support more characters, many character sets and encoding systems have been used to support many languages (see <xref linkend="list-of-encoding-values"/>).</para>
<para><ulink url="https://en.wikipedia.org/wiki/Unicode">Unicode</ulink> character set can represent practically all characters known to human with 21 bit code point range (i.e., 0 to 10FFFF in hexadecimal notation).</para>
<para>Text encoding system <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink> fits Unicode code points into a sensible 8 bit data stream mostly compatible with the ASCII data processing system. This makes <emphasis role="strong">UTF-8</emphasis> the modern preferred choice. <emphasis role="strong">UTF</emphasis> stands for Unicode Transformation Format. When <ulink url="https://en.wikipedia.org/wiki/ASCII">ASCII</ulink> plain text data is converted to <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink> one, it has exactly the same content and size as the original ASCII one. So you loose nothing by deploying UTF-8 locale.</para>
<para>Under <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink> locale with the compatible application program, you can display and edit any foreign language text data as long as required fonts and input methods are installed and enabled. For example under "<literal>LANG=fr_FR.UTF-8</literal>" locale, <literal>gedit</literal>(1) (text editor for the GNOME desktop) can display and edit Chinese character text data while presenting menus in French.</para>
<tip> <para>Both the new standard "<literal>en_US.UTF-8</literal>" locale and the old standard "<literal>C</literal>"/"<literal>POSIX</literal>" locale use the standard US English message, they have subtle differences in sorting order etc. If you want to handle not only ASCII characters but also handle all UTF-8 encoded characters gracefully while maintaining the old "<literal>C</literal>" local behavior, use the non-standard "<literal>C.UTF-8</literal>" locale on Debian.</para> </tip>
<note> <para>Some programs consume more memory after supporting I18N. This is because they are coded to use <ulink url="https://en.wikipedia.org/wiki/UTF-32/UCS-4">UTF-32(UCS4)</ulink> internally to support Unicode for speed optimization and consume 4 bytes per each ASCII character data independent of locale selected. Again, you loose nothing by deploying UTF-8 locale.</para> </note>
</section>
<section id="_the_reconfiguration_of_the_locale">
<title>The reconfiguration of the locale</title>
<para>In order for the system to access a particular locale, the locale data must be compiled from the locale database.</para>
<para>The <literal>locales</literal> package does <emphasis role="strong">not</emphasis> come with pre-compiled locale data. You need to configure it as:</para>
<screen># dpkg-reconfigure locales</screen>
<para>This process involves 2 steps.</para>
<orderedlist>
<listitem> <para> Select all required locale data to be compiled into the binary form. (Please make sure to include at least one UTF-8 locale)</para> </listitem>
<listitem> <para> Set the system wide default locale value by creating "<literal>/etc/default/locale</literal>" for use by PAM (see <xref linkend="_pam_and_nss"/>). </para> </listitem>
</orderedlist>
<para>The system wide default locale value set in "<literal>/etc/default/locale</literal>" may be overridden by the GUI configuration for GUI applications.</para>
<note> <para>Actual traditional encoding system can be identified by "<literal>/usr/share/i18n/SUPPORTED</literal>". Thus, the "<literal>LANG=en_US</literal>" is "<literal>LANG=en_US.ISO-8859-1</literal>". </para> </note>
<para>The <literal>locales-all</literal> package comes with pre-compiled locale data for all locale data. Since it doesn't create "<literal>/etc/default/locale</literal>", you may still need to install the <literal>locales</literal> package, too.</para>
<tip><para>The <literal>locales</literal> package of some Debian derivative distributions come with pre-compiled locale data for all locale data. You need to install both <literal>locales</literal> and <literal>locales-all</literal> packages on Debian to emulate such system environment.</para></tip>
</section>
<section id="_filename_encoding">
<title>Filename encoding</title>
<para>For cross platform data exchanges (see <xref linkend="_removable_storage_device"/>), you may need to mount some filesystem with particular encodings. For example, <literal>mount</literal>(8) for <ulink url="https://en.wikipedia.org/wiki/File_Allocation_Table">vfat filesystem</ulink> assumes <ulink url="https://en.wikipedia.org/wiki/Code_page_437">CP437</ulink> if used without option. You need to provide explicit mount option to use <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink> or <ulink url="https://en.wikipedia.org/wiki/Code_page_932">CP932</ulink> for filenames.</para>
<note> <para>When auto-mounting a hot-pluggable <ulink url="https://en.wikipedia.org/wiki/USB_flash_drive">USB flash drive</ulink> under modern desktop environment such as GNOME, you may provide such mount option by right clicking the icon on the desktop, click "Drive" tab, click to expand "Setting", and entering "utf8" to "Mount options:". The next time this USB flash drive is mounted, mount with UTF-8 is enabled.</para> </note>
<note> <para>If you are upgrading system or moving disk drives from older non-UTF-8 system, file names with non-ASCII characters may be encoded in the historic and deprecated encodings such as <ulink url="https://en.wikipedia.org/wiki/ISO/IEC_8859-1">ISO-8859-1</ulink> or <ulink url="https://en.wikipedia.org/wiki/Extended_Unix_Code">eucJP</ulink>. Please seek help of text conversion tools to convert them to <ulink url="https://en.wikipedia.org/wiki/UTF-8">UTF-8</ulink>. See <xref linkend="_text_data_conversion_tools"/>.</para> </note>
<para><ulink url="https://en.wikipedia.org/wiki/Samba_(software)">Samba</ulink> uses Unicode for newer clients (Windows NT, 200x, XP) but uses <ulink url="https://en.wikipedia.org/wiki/Code_page_850">CP850</ulink> for older clients (DOS and Windows 9x/Me) as default. This default for older clients can be changed using "<literal>dos charset</literal>" in the "<literal>/etc/samba/smb.conf</literal>" file, e.g., to <ulink url="https://en.wikipedia.org/wiki/Code_page_932">CP932</ulink> for Japanese.</para>
</section>
<section id="_localized_messages_and_translated_documentation">
<title>Localized messages and translated documentation</title>
<para>Translations exist for many of the text messages and documents that are displayed in the Debian system, such as error messages, standard program output, menus, and manual pages. <ulink url="https://en.wikipedia.org/wiki/Gettext">GNU gettext(1) command tool chain</ulink> is used as the backend tool for most translation activities.</para>
<para>Under "Tasks" → "Localization" <literal>aptitude</literal>(8) provides an extensive list of useful binary packages which add localized messages to applications and provide translated documentation.</para>
<para>For example, you can obtain the localized message for manpage by installing the <literal>manpages-<emphasis>LANG</emphasis></literal> package. To read the Italian-language manpage for <emphasis>programname</emphasis> from "<literal>/usr/share/man/it/</literal>", execute as the following.</para>
<screen>LANG=it_IT.UTF-8 man <emphasis>programname</emphasis></screen>
<para>GNU gettext can accommodate priority list of translation languages with <literal>$LANGUAGE</literal> environment variable. For example:</para>
<screen> $ export LANGUAGE="pt:pt_BR:es:it:fr"</screen>
<para>For more, see <literal>info gettext</literal> and read the section "The LANGUAGE variable".</para>
</section>
<section id="_effects_of_the_locale">
<title>Effects of the locale</title>
<para>The sort order of characters with <literal>sort</literal>(1) and <literal>ls</literal>(1) are affected by the locale. Exporting <literal>LANG=en_US.UTF-8</literal> sorts in the dictionary <literal>A->a->B->b...->Z->z</literal> order, while exporting <literal>LANG=C.UTF-8</literal> sorts in ASCII binary <literal>A->B->...->Z->a->b...</literal> order.</para>
<para>The date format of <literal>ls</literal>(1) is affected by the locale (see <xref linkend="_customized_display_of_time_and_date"/>).</para>
<para>The date format of <literal>date</literal>(1) is affected by the locale. For example:</para>
<screen> $ unset LC_ALL
$ LANG=en_US.UTF-8 date
Thu Dec 24 08:30:00 PM JST 2023
$ LANG=en_GB.UTF-8 date
Thu 24 Dec 20:30:10 JST 2023
$ LANG=es_ES.UTF-8 date
jue 24 dic 2023 20:30:20 JST
$ LC_TIME=en_DK.UTF-8 date
2023-12-24T20:30:30 JST</screen>
<para>Number punctuation are different for locales. For example, in English locale, one thousand point one is displayed as "<literal>1,000.1</literal>" while in German locale, it is displayed as "<literal>1.000,1</literal>". You may see this difference in spreadsheet program.</para>
<para>Each detail feature of "<literal>$LANG</literal>" environment variable may be overridden by setting "<literal>$LC_*</literal>" variables. These environment variables can be overridden again by setting "<literal>$LC_ALL</literal>" variable. See <literal>locale</literal>(7) manpage for the details. Unless you have strong reason to create complicated configuration, please stay away from them and use only "<literal>$LANG</literal>" variable set to one of the UTF-8 locales.</para>
</section>
</section>
<section id="_the_keyboard_input">
<title>The keyboard input</title>
<section id="_the_keyboard_input_for_linux_console_and_x_window">
<title>The keyboard input for Linux console and X Window</title>
<para>The Debian system can be configured to work with many international keyboard arrangements using the <literal>keyboard-configuration</literal> and <literal>console-setup</literal> packages.</para>
<screen># dpkg-reconfigure keyboard-configuration
# dpkg-reconfigure console-setup</screen>
<para>For the Linux console and the X Window system, this updates configuration parameters in "<literal>/etc/default/keyboard</literal>" and "<literal>/etc/default/console-setup</literal>". This also configures the Linux console font. Many non-ASCII characters including accented characters used by many European languages can be made available with <ulink url="https://en.wikipedia.org/wiki/Dead_key">dead key</ulink>, <ulink url="https://en.wikipedia.org/wiki/AltGr_key">AltGr key</ulink>, and <ulink url="https://en.wikipedia.org/wiki/Compose_key">compose key</ulink>.</para>
</section>
<section id="_the_keyboard_input_for_wayland">
<title>The keyboard input for Wayland</title>
<para>For GNOME on Wayland desktop system, <xref linkend="_the_keyboard_input_for_linux_console_and_x_window"/> can't support non-English European languages. <ulink url="https://en.wikipedia.org/wiki/Intelligent_Input_Bus">IBus</ulink> was made to support not only Asian languages but also European languages. The package dependency of GNOME desktop Environment recommends "<literal>ibus</literal>" via "<literal>gnome-shell</literal>". The code of "<literal>ibus</literal>" has been updated to integrate <literal>setxkbmap</literal> and XKB option functionalities. You need to configure <literal>ibus</literal> from "GNOME Settings" or "GNOME Tweaks" for the multilingualized keyboard input.</para>
<note><para>If ibus is active, your classic X keyboard configuration by the <literal>setxkbmap</literal> may be overridden by <literal>ibus</literal> even under classic X-based desktop environment. You can disable installed <literal>ibus</literal> using <literal>im-config</literal> to set input method to "None". For more, see <ulink url="https://wiki.debian.org/Keyboard">Debian Wiki on keyboard</ulink>.</para></note>
</section>
<section id="_the_input_method_support_with_ibus">
<title>The input method support with IBus</title>
<para>Since GNOME desktop Environment recommends "<literal>ibus</literal>" via "<literal>gnome-shell</literal>", "<literal>ibus</literal>" is the good choice for input method.</para>
<para>Multilingual input to the application is processed as:</para>
<screen>Keyboard Application
| ^
| |
+-> Linux kernel -> Input method (ibus) -> Gtk, Qt, X, Wayland
+-- Engine--+</screen>
<para>The list of IBus and its engine packages are the following.</para>
<table pgwide="0" frame="topbot" rowsep="1" colsep="1">
<title>List of IBus and its engine packages</title>
<tgroup cols="4">
<colspec colwidth="124pt" align="left"/>
<colspec colwidth="76pt" align="left"/>
<colspec colwidth="70pt" align="left"/>
<colspec colwidth="211pt" align="left"/>
<thead>
<row>
<entry> package </entry>
<entry> popcon </entry>
<entry> size </entry>
<entry> supported locale </entry>
</row>
</thead>
<tbody>
<row>
<entry> ibus </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> input method framework using dbus </entry>
</row>
<row>
<entry> ibus-mozc </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Japanese </entry>
</row>
<row>
<entry> ibus-anthy </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , </entry>
</row>
<row>
<entry> ibus-skk </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , </entry>
</row>
<row>
<entry> ibus-kkc </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , </entry>
</row>
<row>
<entry> ibus-libpinyin </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Chinese (for zh_CN) </entry>
</row>
<row>
<entry> ibus-chewing </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , (for zh_TW) </entry>
</row>
<row>
<entry> ibus-libzhuyin </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , (for zh_TW) </entry>
</row>
<row>
<entry> ibus-rime </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , (for zh_CN/zh_TW) </entry>
</row>
<row>
<entry> ibus-cangjie </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> , , (for zh_HK) </entry>
</row>
<row>
<entry> ibus-hangul </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Korean </entry>
</row>
<row>
<entry> ibus-libthai </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Thai </entry>
</row>
<row>
<entry> ibus-table-thai </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Thai </entry>
</row>
<row>
<entry> ibus-unikey </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Vietnamese </entry>
</row>
<row>
<entry> ibus-keyman </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Multilingual: <ulink url="https://keyman.com/">Keyman</ulink> engine for over 2000 languages </entry>
</row>
<row>
<entry> ibus-table </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> table engine for IBus </entry>
</row>
<row>
<entry> ibus-m17n </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> Multilingual: Indic, Arabic and others </entry>
</row>
<row>
<entry> plasma-widgets-addons </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> additional widgets for Plasma 5 containing Keyboard Indicator </entry>
</row>
<!--
<row>
<entry> fcitx5 </entry>
<entry> @-@popcon1@-@ </entry>
<entry> @-@psize1@-@ </entry>
<entry> input method framework using dbus (IBus compatible) </entry>
</row>
-->
</tbody>
</tgroup>
</table>
<note>
<para>For Chinese, "<literal>fcitx5</literal>" may be an alternative input method framework. For Emacs aficionados, "<literal>uim</literal>" may be an alternative. Either cases, you may need to do extra manual configuration with <literal>im-config</literal>. Some old classic <ulink url="https://en.wikipedia.org/wiki/Input_method">input methods</ulink> such as "<literal>kinput2</literal>" may still exist in Debian repository but are not recommended for the modern environment.</para>
</note>
</section>
<section id="_an_example_for_japanese">
<title>An example for Japanese</title>
<para>I find the Japanese input method started under English environment ("<literal>en_US.UTF-8</literal>") very useful. Here is how I did this with IBus for GNOME on Wayland:</para>
<orderedlist>
<listitem> <para> Install the Japanese input tool package <literal>ibus-mozc</literal> (or <literal>ibus-anthy</literal>) with its recommended packages such as <literal>im-config</literal>. </para> </listitem>
<listitem> <para> Select "Settings" → "Keyboard" → "Input Sources" → click "<literal>+</literal>" in "Input Sources" → "Japanese" → "Japanese mozc (or anthy)" and click "Add" if it hasn't been activated. </para> </listitem>
<listitem> <para> You may chose as many input sources. </para> </listitem>
<listitem> <para> Relogin to user's account. </para> </listitem>
<listitem> <para> Setup each input source by right clicking the GUI toolbar icon. </para> </listitem>
<listitem> <para> Switch among installed input sources by SUPER-SPACE. (SUPER is normally the Windows key.) </para> </listitem>
</orderedlist>
<tip> <para>If you wish to have access to alphabet only keyboard environment with the physical Japanese keyboard on which shift-<literal>2</literal> has <literal>"</literal> (double quotation mark) engraved, you select "Japanese" in the above procedure. You can enter Japanese using "Japanese mozc (or anthy)" with physical "US" keyboard on which shift-<literal>2</literal> has <literal>@</literal> (at mark) engraved.</para> </tip>
<itemizedlist>
<listitem> <para> The GUI menu entry for <literal>im-config</literal>(8) is "Input method". </para> </listitem>
<listitem> <para> Alternatively, execute "<literal>im-config</literal>" from user's shell. </para> </listitem>
<listitem> <para> <literal>im-config</literal>(8) behaves differently if command is executed from root or not. </para> </listitem>
<listitem> <para> <literal>im-config</literal>(8) enables the best input method on the system as default without any user actions. </para> </listitem>
</itemizedlist>
</section>
</section>
<section id="_the_display_output">
<title>The display output</title>
<para>Linux console can only display limited characters. (You need to use special terminal program such as <literal>jfbterm</literal>(1) to display non-European languages on the non-GUI console.)</para>
<para>GUI environment (<xref linkend="_gui_system"/>) can display any characters in the UTF-8 as long as required fonts are installed and enabled. (The encoding of the original font data is taken care and transparent to the user.)</para>
</section>
<section id="_east_asian_ambiguous_character_width_characters">
<title>East Asian Ambiguous Character Width Characters</title>
<para>Under the East Asian locale, the box drawing, Greek, and Cyrillic characters may be displayed wider than your desired width to cause the unaligned terminal output (see <ulink url="https://unicode.org/reports/tr11/#Ambiguous">Unicode Standard Annex #11, 4.2 Ambiguous Characters</ulink>).</para>
<para>You can work around this problem:</para>
<itemizedlist>
<listitem> <para><literal>gnome-terminal</literal>: Preferences → Profiles → <emphasis>Profile name</emphasis> → Compatibility → Ambiguous-wide characters → Narrow </para> </listitem>
<listitem> <para><literal>ncurses</literal>: Set environment <literal>export NCURSES_NO_UTF8_ACS=0</literal>. </para> </listitem>
</itemizedlist>
</section>
</chapter>
|
