1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
|
<html>
<head>
<title> UTF-8 in HDF5</title>
</head>
<body>
<!--
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* Copyright by The HDF Group. *
* Copyright by the Board of Trustees of the University of Illinois. *
* All rights reserved. *
* *
* This file is part of HDF5. The full HDF5 copyright notice, including *
* terms governing use, modification, and redistribution, is contained in *
* the files COPYING and Copyright.html. COPYING can be found at the root *
* of the source code distribution tree; Copyright.html can be found at the *
* root level of an installed copy of the electronic HDF5 document set and *
* is linked from the top-level documents page. It can also be found at *
* http://hdfgroup.org/HDF5/doc/Copyright.html. If you do not have *
* access to either file, you may request a copy from help@hdfgroup.org. *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
-->
<br />
<p>
Return to “<a href="../../Advanced.html">Advanced Topics</a>.”</p>
<hr />
<h1>Using UTF-8 Encoding in HDF5 Applications</h1>
<h2>Introduction</h2>
<dir>
Text and character data are often discussed as though
<i>text</i> means <i>ASCII text</i>. We even go so far
as to call a file containing only ASCII text a <i>plain text</i> file.
This works reasonably well for English (though better for
American English than British English),
but what if that plain text file is in French, German, Chinese,
or any of several hundred other languages?
This document introduces the use of UTF-8 encoding
(see <a href="#UTF8_Note1">note 1</a>),
enabling the use of a much more extensive and flexible character set
that can faithfully represent any of those languages.
<p>
This document assumes a working familiarity with UTF-8 and Unicode.
Any reader who is unfamiliar with UTF-8 encoding should read the
<a href="https://en.wikipedia.org/wiki/UTF-8">Wikipedia UTF-8 article</a>
(<code>https://en.wikipedia.org/wiki/UTF-8</code>)
before proceeding; it provides an excellent primer.
<p>
For our context, the most important UTF-8 concepts are:
<ul>
<li>Multi-byte and variable-size character encodings
<li>Limitations of the ASCII character set
<li>Risks associated with the use of the term <i>plain text</i>
<li>Representation of multiple language alphabets or characters
in a single document
</ul>
<p>
More specific technical details will only become important if they
affect the specifics of your application design or implementation.
</dir>
<p>
<h2>How and Where Is UTF-8 Supported in HDF5?</h2>
<dir>
HDF5 uses characters in object names (which are actually link names,
but that’s a story for a different article), dataset raw data,
attribute names, and attribute raw data.
Though the mechanisms differ, you can use either
ASCII or UTF-8 character sets in all of these situations.
<h3>Object and Attribute Names</h3>
By default, HDF5 creates object and attribute names with ASCII character
encoding. An object or attribute creation property list setting is
required to create object names with UTF-8 characters.
This uses the function <code>H5Pset_char_encoding</code>,
which sets the character encoding used for object and attribute names.
<p>
For example, the following call sequence could be used to create a dataset
with its name encoded with the UTF-8 character set:
<pre>
lcpl_id = H5Pcreate(H5P_LINK_CREATE) ;
error = H5Pset_char_encoding(lcpl_id, H5T_CSET_UTF8) ;
dataset_id = H5Dcreate2(group_id, "datos_ñ", datatype_id, dataspace_id,
lcpl_id, H5P_DEFAULT, H5P_DEFAULT) ; </pre>
<p>
If the character encoding of an object or attribute name is unknown,
the combination of an <code>H5A/H5D/H5G/H5Tget_create_plist</code> call
(that is, <code>H5Aget_create_plist</code>, <code>H5Dget_create_plist</code>,
etc., as appropriate)
and an <code>H5Pget_char_encoding</code> call will reveal that
information.
<h3>Character Datatypes in Datasets and Attributes</h3>
Like object names, HDF5 character data in datasets and attributes
is encoded as ASCII by default.
Setting up attribute or dataset character data to be UTF-8-encoded is
accomplished while defining the attribute or dataset datatype.
This makes use of the function <code>H5Tset_cset</code>,
which sets the character encoding to be used in building a character
datatype.
<p>
For example, the following commands could be used to create an
8-character, UTF-8 encoded, string datatype for use in either
an attribute or dataset:
<pre>
datatype_id = H5Tcopy(H5T_C_S1) ;
error = H5Tset_cset(datatype_id, H5T_CSET_UTF8) ;
error = H5Tset_size(datatype_id, "8") ; </pre>
<p>
If a character or string datatype’s character encoding is unknown,
the combination of an <code>H5Aget_type</code> or <code>H5Dget_type</code>
call and an <code>H5Tget_cset</code> call can be used to determine that.
</dir>
<p>
<h2>Caveats, Pitfalls, and Things to Watch For</h2>
<dir>
Programmers who are accustomed to using ASCII text without accommodating
other text encodings will have to be aware of certain common issues as
they begin using UTF-8 encodings.
</dir>
<dir>
<h3>Cross-platform Portability</h3>
Since the HDF5 Library handles datatypes directly,
UTF-8 encoded text in dataset and attribute datatypes in a
well-designed HDF5 application and file should work transparently
across platforms.
The same should be true of handling names of groups, datasets,
committed datatypes, and attributes within a file.
<p>
Be aware, however, of system or application limitations
once data or other information has been extracted from an HDF5 file.
The application or system must be designed to accommodate
UTF-8 encodings if the information is then used elsewhere in the
application or system environment.
<p>
Data from a UTF-8 encoded HDF5 datatype, in either a dataset or an
attribute, that has been established within an HDF5 application
should “just work” within the HDF5 portions of the
application.
</dir>
<dir>
<h3>Filenames</h3>
Since file access is a system issue, filenames do not fall within
the scope of HDF5’s UTF-8 capabilities;
filenames are encoded at the system level.
<p>
Linux and Mac OS systems normally handle UTF-8 encoded filenames
correctly while Windows systems generally do not.
</dir>
<dir>
<h3>The <i>Plain Text</i> Illusion</h3>
Beware the use of the term <i>plain text</i>.
<i>Plain text</i> is at best ambiguous, but often misleading.
Many will assume that <i>plain text</i> means ASCII,
but plain text German or French, for example,
cannot be represented in ASCII.
Plain text is only unambiguous in the context of English (and even
then can be problematic!).
</dir>
<dir>
<h3>Storage Size</h3>
Programmers and data users accustomed to working strictly with
ASCII data generally make the reasonable assumption that 1 character,
be it in an object name or in data, requires 1 byte of storage.
This equation does not work when using UTF-8 or any other Unicode
encoding.
With Unicode encoding, <i>number of characters</i>
is not synonymous with <i>number of bytes</i>.
One must get used to thinking in terms of <i>number of characters</i>
when talking about content, reserving <i>number of bytes</i>
for discussions of storage size.
<p>
When working with Unicode text, one can no longer assume
a 1:1 correspondence between the number of characters and
the data storage requirement.
</dir>
<dir>
<p>
<h3>System Dependencies</h3>
<strong>Linux</strong>, <strong>Unix</strong>, and similar systems
generally handle UTF-8 encodings in correct and predictable ways.
There is an apparent consensus in the Linux community that
“UTF-8 is just the right way to go.”
</p>
<p>
<strong>Mac OS</strong> systems generally handle UTF-8 encodings
correctly.
</p>
<p>
<strong>Windows</strong> systems use a different Unicode encoding,
UCS-2 (discussed in this
<a href="https://en.wikipedia.org/wiki/UTF-16">UTF-16</a> article)
at the system level.
Within an HDF5 file and application on a Windows system,
UTF-8 encoding should work correctly and as expected.
Problems may arise, however, when that UTF-8 encoding is exposed
directly to the Windows system. For example:
<ul>
<li>File open and close calls on files with UTF-8 encoded names
are likely to fail as the HDF5 open and close operations
interact directly with the Windows file system interface.
<li>Anytime an HDF5 command-line utility (<code>h5ls</code>
or <code>h5dump</code>, for example) emits text output,
the Windows system must interpret the character encodings.
If that output is UTF-8 encoded, Windows will correctly
interpret only those characters in the ASCII subset of UTF-8.
</ul>
</p>
</dir>
<p>
<h2>Common Characters in UTF-8 and ASCII</h2>
<dir>
One interesting feature of UTF-8 and ASCII is that the ASCII
character set is a discrete subset of the UTF-8 character set.
And where they overlap, the encodings are identical.
This means that a character string consisting entirely of
members of the ASCII character set
can be encoded in either ASCII or UTF-8,
the two encodings will be indistinguishable, and
the encodings will require exactly the same storage space.
</dir>
<p>
<h2>See Also</h2>
<dir>
<table width="100%">
<tr valign="top" align="left">
<td width="50%">
For object and attribute names:
<br>
<a href="../../RM/RM_H5P.html#Property-SetCharEncoding"
><code>H5Pset_char_encoding</code></a>
<br>
<a href="../../RM/RM_H5P.html#Property-GetCharEncoding"
><code>H5Pget_char_encoding</code></a>
</td>
<td width="50%">
For dataset and attribute datatypes:
<br>
<a href="../../RM/RM_H5T.html#Datatype-SetCset"
><code>H5Tset_cset</code></a>
<br>
<a href="../../RM/RM_H5T.html#Datatype-GetCset"
><code>H5Tget_cset</code></a>
</td>
</tr>
<tr valign="top" align="left">
<td colspan="2">
<br>
<a href="https://en.wikipedia.org/wiki/UTF-8"
>UTF-8 article on Wikipedia</a>
</td>
</tr>
</table>
</dir>
<hr /><br />
<strong>Notes</strong>
<br />
<a name="UTF8_Note1">1.</a>
UTF-8 is the only <a href="https://en.wikipedia.org/wiki/Unicode">Unicode</a>
standard encoding supported in HDF5.
</p>
<hr /><br />
<p>
Return to “<a href="../../Advanced.html">Advanced Topics</a>.”</p>
<br />
<hr>
<!-- #BeginLibraryItem "/ed_libs/Footer-THGonly.lbi" -->
<address>
<table width="100%" border="0">
<tr valign="top">
<td align="left">
<address>
The HDF Group Help Desk: <img src="../../Graphics/help.png" align=top height=16>
<br>
Describes HDF5 Release 1.8.13, May 2014.
</address>
</td><td width="5%"> </td>
<td align="right">
<a href="../../Copyright.html">Copyright</a> by
<a href="http://www.hdfgroup.org">The HDF Group</a>
</td>
</tr>
</table>
</address>
<!-- #EndLibraryItem --><html><SCRIPT LANGUAGE="JAVASCRIPT">
<!--
document.writeln("Last modified: 17 February 2014");
-->
</SCRIPT>
</body>
</html>
|
