     [file ebzip-ver.texi not included]

ebzip: Utility for compressing and uncompressing CD-ROM books, by Motoyuki Kasahara.

Copyright © 1998, 99, 2000, 01  Motoyuki Kasahara

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by Free Software Foundation, Inc.

This is edition EDITION, for EB Library version VERSION.

Introduction
************

ebzip is a command to compress/uncompress CD-ROM books for EB Library applications. EB Library applications can access compressed CD-ROM books without any functional restriction. (But, their access speed to compressed books may be slower than that to original books.) Since the compression data format is independent of CPU type, operating system, and file system, files compressed by ebzip can be interchanged. The ebzip command is distributed with EB Library.

ebzip is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.

ebzip is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.  See the GNU General Public License for more details.

In addition, you must follow the licenses of your CD-ROM books. Though ebzip is free software, your books may not be free. If the publishers of your CD-ROM don't permit to copy the book onto a hard disk (nor other strage devices), please give up compress the book.

This is edition EDITION, for EB Library version VERSION.

Action and Command Name
***********************

ebzip has three action modes; compression, uncompression and information.

  * compression Compress a CD-ROM book.

  * uncompression Uncompress a CD-ROM book compressed by ebzip.

  * information Output information about compression status of a CD-ROM book compressed by ebzip.

When ebzip is invoked, the action is determined by invoked name of the ebzip command and command line options to ebzip. The action is never changed during an execution of ebzip.

ebzip has installed with three different command names; ebzip, ebunzip and ebzipinfo. (On DOS and Windows systems, ebzipinfo is renamed to ebzipinf). Usually, they are hard-linked like as follows:

     % ls -li ebzip ebunzip ebzipinfo
     38787 -rwxr-xr-x  3 bin   bin     73728 Jun 27 22:48 ebunzip
     38787 -rwxr-xr-x  3 bin   bin     73728 Jun 27 22:48 ebzip
     38787 -rwxr-xr-x  3 bin   bin     73728 Jun 27 22:48 ebzipinfo

When it is invoked as ebunzip, the default action is uncompression. When it is invoked as ebzipinfo, information is the default action. Otherwise, the default action is compression.

In addition, you can specify the action by the command line options; --compress (-z), --uncompress (-u) and --information (-i). They are prior to invoked names. In the following example, "compress" is the action of all the lines:

     ebunzip
     ebzip --uncompress
     ebzipinfo --uncompress

Invoking ebzip
**************

Its usual way to invoke ebzip, ebunzip and ebzipinfo are as follows:

     % ebzip BOOK-PATH
     % ebunzip BOOK-PATH
     % ebzipinfo BOOK-PATH

BOOK-PATH points to the top directory of a CD-ROM book where the file CATALOG or CATALOGS resides. ebzip compresses a book at BOOK-PATH, ebunzip uncompresses a book at BOOK-PATH, and ebzipinfo outputs information about a book at BOOK-PATH.

Though you cannot specify two or more book-paths, you can omit book-path like as follows:

     % ebzip
     % ebunzip
     % ebzipinfo

In this case, it is assumed that the current working directory (.) is specified.

Compression
***********

ebzip (equivalent to ebunzip --compress and ebzipinfo --compress) compresses a CD-ROM book.

Example of Compression (1)
==========================

This section explains how to compress your CD-ROM book which has already copied on a hard disk. Supporse that you have already copied a CD-ROM book to /dict on a hard disk, but it is not compressed. The book has the following files under /dict.

     /dict/catalog
     /dict/language
     /dict/kanji/start
     /dict/english/start

You can compress the book by the following way:

     % cd /dict
     % ebzip

This is equivalent to:

     % ebzip --output-directory /dict /dict

Compression may take a while. ebzip outputs status messages during the compression like as follows:

     ==> compress /dict/kanji/start <==
     output to ./dict/kanji/start.ebz
      1.0% done (2097152 / 220069888 bytes)
      1.9% done (4194304 / 220069888 bytes)
      2.9% done (6291456 / 220069888 bytes)
             : (snip)
     completed (220069888 / 220069888 bytes)
     220069888 -> 78163751 bytes (35.5%)

When the compression has done, the directory structure under /dict is slightly changed.

     /dict/catalog
     /dict/language.ebz
     /dict/kanji/start.ebz
     /dict/english/start.ebz

The suffix .ebz is added to the some files (/dict/language.ebz, /dict/kanji/start.ebz and /dict/english/start.ebz). They are compressed files. Other files are unchanged.

ebzip compresses the following files; START, HONMON, LANGUAGE, and font files under the GAIJI directories.

Example of Compression (2)
==========================

This section shows another example. This section explains how to compress files in a CD-ROM disc and put them onto a hard disk. Suppose that and original CD-ROM book is mounted on /cdrom and that the book has the following files under /cdrom.

     /cdrom/catalog
     /cdrom/language
     /cdrom/german/start
     /cdrom/french/start
     /cdrom/italian/start

Execute the following commands to compress the book an put it on /dict:

     % cd /dict
     % ebzip --keep /cdrom

This is equivalent to:

     % ebzip --keep --output-directory /dict /cdrom

Please note that the output directory /dict may be empty, but it must have been created before compression. In this example, the --keep option is used. By default, ebzip deletes original files after compression, but the --keep option disables this behavior.

The directory strcture under /dict and /cdrom are almost the same, but the suffix .ebz is added to the compressed files under /dict.

     /dict/catalog
     /dict/language.ebz
     /dict/german/start.ebz
     /dict/french/start.ebz
     /dict/italian/start.ebz

The intermediate directories, /dict/german, /dict/french and /dict/italian, are created by ebzip if missing. CATALOG and CATALOGS are simply copied to the output directory. ebzip ignores files and directories not supported by EB Library (i.e. not compressed or copied to the output directory).

Compression Level
=================

ebzip supports six compression levels, from 0 to 5. The compression level 0 is the fastest compression/uncompression method, but it is the worst compression method. The compression level 5 is the best compression method but it is the slowest compression/uncompression method. The compression level is set by the --level (-l) option.

     % ebzip --level 2 --output-directory /dict /dict

The default level is 0 (fastest but worst).

Test
====

Invoking ebzip with the --test (-t) option, ebzip doesn't write compressed data to actual files. This is usuful when you want to know the sizes of compressed files before compression.

     % ebzip --test /dict /dict

Partial Compression
===================

ebzip compresses an entire book by default, but the --subbook (-S) and --skip-content (-s) options allow you to compress part of a book.

The --subbook option takes an argument which is a list of directory names of subbooks to be compressed, separeted by a comma (,). The ebinfo command tells you what subbooks your book contains and what their directory names are.

     % ebinfo /mnt/cdrom
     disc type: EB/EBG/EBXA/EBXA-C/S-EBXA
     character code: JIS X 0208
     the number of subbooks: 3

     subbook 1:
       title: New English Japanese Dictionary (4th edition)
       directory: english
       search methods: word endword keyword menu
       font sizes: 16 24 30 48
       narrow font characters: 0xa121 -- 0xa24e
       wide font characters: 0xa321 -- 0xa27e

     subbook 2:
       title: CD-ROM French Japanese Dictionary (3rd edition)
       directory: french
       search methods: word endword keyword menu
       font sizes: 16 24 30 48
       narrow font characters:
       wide font characters: 0xa321 -- 0xa27e

     subbook 3:
       title: Pocket German Japanese Dictionary (3rd edition)
       directory: german
       search methods: menu
       font sizes: 16 24 30 48
       narrow font characters:
       wide font characters: 0xa321 -- 0xa27e

(see Invoking ebinfo (ebinfo), for more details about the ebinfo command). In this example, there are three subbooks in the book, and their directory names are english, french and german. By the following, you can compress english and french:

     % ebzip --subbook english,french /dict

Case of the directory names are not sensitve. An argument to --subbook can be splitted like as follows:

     % ebzip --subbook english --subbook french /dict

Files in the rest subbooks (german in this example) are not compressed nor copied, so that the compression procedure for the book is incompleted at this point. In order to use the book, you must compress the rest subbooks in a further operation, or deletes the rest subbooks from a subbook list in the CATALOG(S) file by using the ebrefile command (see Combination with ebzip (ebrefile), for more details). You can compress subbooks with different compression levels (e.g. compress english and french with level 0, and compress german with level 2).

The --skip-content (-s) is similar to --subbook, but --skip-content specifies data types of files to be ignored.

     % ebzip --skip-content sound,movie --subbook french /dict

In this example, ebzip skips sound and movie files. The data type names currently recognized by ebzip are: font, graphic, sound and movie.

Overwrite Old Files
===================

ebzip asks you whether a file may be overwritten or not when the file already exists.

     ==> compress /dict/science/data/honmon <==
     output to /dict/science/data/honmon.ebz

     the file already exists: /dict/science/data/honmon.ebz
     do you wish to overwrite (y or n)?

Please answer y or n. If you answer as n, ebzip skips the file.

When the --force-overwrite option is specified, ebzip overwrites existed files without confirmation. Similary, when the --no-overwrite option is specified, ebzip skips the file without confirmation.

Compress Compressed Book
========================

Also S-EBXA and EPWING define compression formats, and some CD-ROM books of S-EBXA and EPWING are compressed with the formats. The formats are different from the ebzip compression format.

ebzip (ebunzip) can also uncompress CD-ROM book compressed with the EPWING or S-EBXA compression fomrat, but it cannot compress book with the format.

If you command ebzip to compress a CD-ROM book which has been compressed with a compression format, ebzip automatically uncompresses the book and compresses it with the ebzip compression.

Compression ratio of the ebzip compression format is generally better than those of EPWING and S-EBXA formats.

Uncompression
*************

ebunzip (equivalent to ebzip --uncompress and ebzipinfo --uncompress) uncompresses a CD-ROM book compressed by ebzip.

Example of Uncompression
========================

Uncompression procedure is closely similar to compression. Suppose that a compressed subbook resides under /dict. The book has the following files under /dict.

     /dict/catalog
     /dict/language.ebz
     /dict/kanji/start.ebz
     /dict/english/start.ebz

Execute the following command to uncompress the book:

     % cd /dict
     % ebunzip

This is equivalent to:

     % ebunzip --output-directory /dict /dict

Uncompression may take a while. ebunzip outputs status messages during the uncompression like as follows:

     ==> uncompress /dict/kanji/start.ebz <==
     output to ./kanji/start.ebz
      5.1% done (4194304 / 82460672 bytes)
     10.2% done (8388608 / 82460672 bytes)
     15.3% done (12582912 / 82460672 bytes)
             : (snip)
     completed (82460672 / 82460672 bytes)
     25681439 -> 82460672 bytes

During the uncompression, ebunzip asks you whether an existed file may be overwritten or not:

     ==> uncompress /dict/kanji/start.ebz <==
     output to /dict/kanji/start

     the file already exists: /dict/kanji/start
     do you wish to overwrite (y or n)?

When the uncompression has done, the directory structure under dict is slightly changed.

     /dict/catalog
     /dict/language
     /dict/kanji/start
     /dict/english/start

Uncompressed files change their filenames; the suffix .ebz is removed from them.

Options for Uncompression
=========================

Most options have the same meanings at compression and uncompression.

When the --keep option is specified, ebunzip doesn't delete compressed files after uncompression.

Invoking ebunzip with the --test option, ebunzip doesn't write uncompressed data to acutual files. This option is usuful for checking validity of compressed files. See Test, for more details about the --test option.

When the --subbook and/or --skip-content options are specified, ebunzip only uncompresses files matched to the given condition only. See Partial Compression, for more details about these options.

When the --force-overwrite or --no-overwrite option is specified, ebunzip overwrites or skips files without confirmation for overwriting. See Overwrite Old Files, for more details about the --force-overwrite and --no-overwrite options.

ebunzip ignores the --level option. You don't have to specify the --level option at uncompression.

Uncompress Compressed Book
==========================

Also S-EBXA and EPWING define compression formats, and some CD-ROM books of S-EBXA and EPWING are compressed with the formats. The formats are different from the ebzip compression format.

ebzip (ebunzip) can also uncompress CD-ROM book compressed with the EPWING or S-EBXA compression fomrat, but it cannot compress book with the format.

Information
***********

ebzipinfo (equivalent to ebzip --information and ebunzip --information) outputs information about compression status of a CD-ROM book.

Example of Information
======================

Please type the following commands to output information of the compressed CD-ROM book at /dict.

     % ebzipifo /dict

ebzipinfo outputs information to standard out like as following:

     ==> /dict/spanish/spanish/start.ebz <==
     154290176 -> 54881862 bytes (35.6%, ebzip level 0 compression)

     ==> /dict/spanish/language.ebz <==
     compressed: 12288 -> 4121 bytes (33.5%, ebzip level 0 compression)

     ==> /dict/spanish/catalog <==
     2048 bytes (not compressed)

Options for Information
=======================

When you invoke ebzipinfo with the --subbook and/or --skip-content options, ebzipinfo outputs information about files matched to the given condition only. See Partial Compression, for details about these options.

ebzipinfo ignores the following options; --force-overwrite, --keep, --level, --no-overwrite, --output-directory, --quiet, --silence and --test.

Delete a Book
*************

ebzip doesn't support deletion of a compressed book on a hard disk. Please use the rm command, instead.

     % rm -r /dict

Please be careful in this operation.

Summary of ebzip Options
************************

The ebzip command supports both traditional single-letter options and mnemonic long option names. Long option names are indicated with -- instead of -. Abbreviations for option names are allowed as long as they are unique.

-f
--force-overwrite
     Overwrite existed files without confirmation. This option is ignored when the action is information, or when you specify the --test (-t) option.

     (see Overwrite Old Files, for more details).

-h
--help
     Output help message to standard out, then exit.

-i
--information
     Output compression status of the specified CD-ROM book.

     (see Information, for more details).

-k
--keep
     Don't delete an input file when the corresponding output file has been written. This option is ignored when the action is information, or when you specify the --test (-t) option.

     (see Compression Example 2, for more details).

-l INTEGER
--level INTEGER
     Specify compression level. INTEGER must be 0, 1, 2, 3, 4 or 5. The level 0 is fastest but least compression method. The level 5 is slowest but best compression method. The defalt level is 0. This option is ignored when the action is uncompression or information.

     (see Compression Level, for more details).

-n
--no-overwrite
     Skip existed files without confirmation for overwriting. This option is ignored when the action is information, or when you specify the --test (-t) option.

     (see Overwrite Old Files, for more details).

-o DIRECTORY
--output-directory DIRECTORY
     Specify a directory of output files. The default is the current directory (.). This option is ignored when the action is information, or when you specify the --test (-t) option.

-q
--quiet
--silence
     Supress all information and warning messages. However, when an output file has already exists, during compression/uncompresion, ebzip still asks you whether the output file may be overwritten or not. To supress this question, specify the --force-overwrite (-f) or --no-overwrite (-n) option. This option is ignored when the action is uncompression or information.

-s TYPES
--skip-content TYPES
     Specify data types of files to be ignored. TYPES must be a list of data type names separated by a comma (,). The data type names ebzip currently recognizes are: font, graphic, sound and movie. By default, none is skipped.

-S SUBBOOKS
--subbook SUBBOOKS
     Specify target subbooks to compress, uncompress or output information. SUBBOOKS must be a list of directory names of subbooks separeted by a comma (,). By default, all subbooks are specified as target.

     (see Partial Compression, for more details).

-t
--test
     Don't write compression/uncompression data to the actual files. This option is ignored when the action is information.

     (see Test, for more details).

-u
--uncompress
     Uncompress a CD-ROM book.

     (see Uncompression, for more details).

-v
--version
     Output version number to standard out, then exit.

-z
--compress
     Compress a CD-ROM book.

     (see Compression, for more details).

Comparision of Compression Results
**********************************

Compression result of START file in a certain English-Japanese and Japanese-English dictionary (EBXA):

compression level              file size           ratio
(original)               154476544 bytes     100.0%
level 0                  54922039 bytes      35.6%
level 1                  49615791 bytes      32.1%
level 2                  45983464 bytes      29.8%
level 3                  43431397 bytes      28.1%
(reference: gzip -6)     38712247 bytes      25.0%

Compression result of START file in a certain English dictionary (EBG):

compression level        file size          ratio
(original)               55511040 bytes     100.0%
level 0                  25866491 bytes     46.6%
level 1                  24419066 bytes     44.0%
level 2                  23453212 bytes     42.2%
level 3                  22748840 bytes     41.0%
(reference: gzip -6)     20930009 bytes     37.7%

Compression result of HONMON file in a certain Japanese dictionary (EPWING V1):

compression level        file size          ratio
(original)               23642112 bytes     100.0%
level 0                   9927048 bytes     42.0%
level 1                   9213184 bytes     39.0%
level 2                   8767603 bytes     37.1%
level 3                   8495563 bytes     35.9%
(reference: gzip -6)      8114658 bytes     34.3%

Compression result of HONMON2 file in a certain Japanese dictionary (EPWING V5):

compression level        file size           ratio
(uncompressed)           220069888 bytes    100.0%
(original)               149139690 bytes     67.8%
level 0                   82973263 bytes     37.7%
level 1                   78163751 bytes     35.5%

Compression result of the START file in a certain encyclopedia (S-EBXA):

compression level        file size           ratio
(uncompressed)           225140735 bytes    100.0%
(original)               176467968 bytes     78.4%
level 0                  146045331 bytes     64.9%
level 1                  138137218 bytes     61.4%

Compressed File Format
**********************

This chapter explains details about compressed file format used by ebzip.

Overview about Compression File Format
======================================

The compressed file format has the following features.

  * independent of CPU type, operating system, and file system.

  * loss-less compression.

     You can recover an original file by uncompressing the compressed file.

  * four compression levels.

     You can choose faster or better compression.

A compressed file consists of header, index and data parts. They are placed in that order.

     +--------+-------------+-----------------------------+
     | header |    index    |            data             |
     +--------+-------------+-----------------------------+
                                                          EOF

Data Part
=========

An orignal file is compressed by the following processes.

At first, ebzip slices an original file. Each slice except for the last slice (slice N in the following picture) has the same size.

     +---------------+---------------+--   --+----------+
     |    slice 1    |    slice 2    |  ...  | slice N  |
     +---------------+---------------+--   --+----------+
                                                        EOF

Slice size is determined by the compression level (see Compression Level, about compression level):

compression level     slice size
0                      2048 bytes
1                      4096 bytes
2                      8192 bytes
3                     16384 bytes
4                     32768 bytes
5                     65536 bytes

Second, if the last slice is shorter than the slice size, ebzip extends the last slice to the slice size by padding bytes of 0x00.

                                                         pad
     +---------------+---------------+--   --+---------+-----+
     |    slice 1    |    slice 2    |  ...  |    slice N    |
     +---------------+---------------+--   --+---------+-----+
                                                             EOF

Finally, ebzip compresses each slice into the DEFLATE compressed data format, described in RFC 1951. A slice is compressed independently of another slice. Usually, each compressed slice occupies different size. If the number of bits of the compressed slice is not a multiple of 8, 1 to 7 bits are padded to the number of bits come to a multiple of 8 at the tail of the compressed slice. Thus, each compressed slice starts at byte boundary. The contents of the padded bits are undefined, but the padded bits are never used.

     +------------+----------+--   --+--------------+
     | compressed |compressed|  ...  |  compressed  |
     |   slice 1  | slice 2  |  ...  |   slice N    |
     +------------+----------+--   --+--------------+

This is a data part of the compressed file format, which consists of compressed slices.

The padding in the last slice is compressed as a part of the slice. When ebunzip recovers the last slice, it uncompresses the slice and then remove the padding.

When a compressed slice is larger than or equal to slice size, ebzip discards the compressed data of the slice. In this case, ebzip records original data as the compressed data for that slice instead.

If an original file is empty, the data part is not appered in a compressed file.

Index Part
==========

At compression, ebzip records an index for each compressed slice. An index represents a distance betweeen the beginning of the compressed file to the beginning of a compressed slice. The unit of the distance is byte.

     +---------+---------+--           --+---------+---------+
     | index 1 | index 2 |  ...........  | index N |index END|
     +---------+---------+--           --+---------+---------+
          |         |                        |         |
      +---+         +----+                   +------+  +-----------+
      V                  V                          V              V
     +------------------+------------------+--   --+--------------+
     |    compressed    |    compressed    |  ...  |  compressed  |
     |      slice 1     |     slice 2      |  ...  |   slice N    |
     +------------------+------------------+--   --+--------------+

Each index takes from 2 to 4 bytes, according with size of an original file:

original file size                index size
0 ...      65535 bytes     2 bytes
65535 ...   16777215 bytes     3 bytes
16777216 ... 4294967295 bytes     4 bytes

All multi-byte numbers in the indexes stored with the most significant byte first. For example, 0x1234 is stored as follows. First byte holds 0x12, and second byte holds 0x34.

     +---------+---------+
     |0001 0010|0011 0100|
     +---------+---------+
       (0x12)    (0x34)

The index part begins with the index for the compressed slice 1, and the index for the compressed slice 2 follows it. The index for compressed slice N is followed by the index for END; index for the next byte of the end of the compressed slice N. This index also represents the size of the compressed file.

     +---------+---------+--       --+---------+---------+
     | index 1 | index 2 |  .......  | index N |index END|
     +---------+---------+--       --+---------+---------+

If its size is equal to the slice size, the data of the slice is not compressed accutually.

If an original file is empty, the index part has only one index. The index represents the size of the compressed file.

Header Part
===========

A header part occupies 22 bytes. It consists of the following fields.

     +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
     |   magic ID   |*1| *2  |    file size    | Adler-32  |   mtime   |
     +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
      0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21

     *1: zip mode and compression level
     *2: reserved area

magic ID (5 bytes)
     It has the fixed value; 0x45, 0x42, 0x5a, 0x69, 0x70 (EBZip in ASCII string).

zip mode (4 bits of the most significant bit side)
     This identifies the compression mode. In EB Library version VERSION, only 1 (0001 in binary digits) can be prescribed as this version.

compression level (4 bits of the least-significant-bit side)
     This identifies the compression level.

reserved area (2 bytes)
     Reserved but not used. It is filled with 0x0000.

file size (6 bytes)
     This contains the size of the original (uncompressed) file.

Adler-32 (4 bytes)
     This is a checksum value of the uncompressed data computed according to Adler-32 algorithm described in RFC 1950.

mtime (4 bytes)
     This gives the most recent modification time of the original file. This value is the time in seconds since 00:00:00 GMT, Jan. 1, 1970.

Both zip mode and compression level are packed into the 5th byte in the header. zip mode includes a most siginificant bit, and compression level includes least-significant-bit. If zip mode is 1 and compression level is 2, then 5th byte of the header is 0x12.

      MSB                         LSB
     +---+---+---+---+---+---+---+---+
     | 0   0   0   1   0   0   1   0 | = 0x12
     +---+---+---+---+---+---+---+---+
        (zip mode)   | (compression level)

All multi-byte numbers in the header are stored with the most significant byte first.