1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
|
README for JTE 1.19
Steve McIntyre <steve@einval.com>
16 June 2011
License - GPL v2. See the file COPYING for more details.
JTE - Jigdo Template Export
===========================
Introduction
Jigdo is a useful tool to help in the distribution of large files like CD and
DVD images. See Richard Atterer's site for more details. Debian CDs and DVD ISO
images are published on the web in jigdo format to allow end users to download
them more efficiently.
Jigdo is generic and powerful - it can be used for any large files that are
made up of smaller files. However, to be this generic is costly. Creating jigdo
files from ISO images is quite inefficient - to work out which files are
included in the ISO image, jigdo has to calculate and compare checksums of
every possible file and every extent in the image. Essentially it has to
brute-force the image. On my home system, it can take several hours to do this
for a 4.5GB DVD image.
There are a few ways to improve this that I can see:
1. Modify jigdo so it knows about the internals of ISO images and can
efficiently scan them (bad, not very generic for jigdo)
2. Write a helper tool to dump extra information for jigdo to use alongside
the ISO image (helper tool written, but modifying jigdo to use this looks
HARD)
3. Patch ISO creation tools to write .jigdo and .template files
alongside the ISO image
I've now done the third of these, and called the code JTE (or Jigdo
Template Export). The code works fine, and runs in a very small
fraction of the time taken to run genisoimage and jigdo
separately. The output .jigdo and .template files work correctly,
i.e. jigdo-file and the wrapper script jigdo-mirror accept them and
will generate an ISO image that exactly matches the original.
However, cdrkit / genisoimage is now deprecated in my opinion. The
nice people working on xorriso (George Danchev and Thomas Schmitt)
have helped a huge amount by cleaning up some of my code and wrapping
it properly in a shared library. The code is now included here as
libjte, and re-licensed slightly as LGPL v2.1 or later.
Tools
=====
JTE also comes with some extra tools:
jigit-mkimage, a simple and very fast tool to reconstruct image files
from .jigdo and .template files. It doesn't have any logic to cope
with downloading missing files, but will list the missing files that
are needed. It is also much faster for people (like me!) who already
have full local mirrors.
iso-image.pl is a CGI script to wrap around jigit-mkimage if you'd
like to be able to offer images for HTTP download without using up
multiple gigabytes of disk space. And for added network efficiency the
perl CGI also supports HTTP v1.1 byte ranges so clients can resume
aborted downloads.
jigit is a first attempt at a user-friendly wrapper for jigit-mkimage
on a user's machine.
----------------------------------------------------------------------
How to use JTE in genisoimage
To use the jigdo creation code, specify the location of the output
.jigdo and .template files alongside the ISO image. You can also
specify the minimum size beneath which files will just be dropped into
the binary template file data rather than listed as separate files to
be found on the mirror, and exclude patterns to ignore certain files
in the same way. And paths in the original filesystem can be mapped
onto more global namespaces using the [Servers] section in the .jigdo
file. For example:
genisoimage -J -r -o /home/steve/test1.iso \
-jigdo-jigdo /home/steve/test1.jigdo \
-jigdo-template /home/steve/test1.template \
-jigdo-min-file-size 16384 \
-jigdo-ignore "README*" \
-jigdo-force-md5 /pool/ \
-jigdo-map Debian=/mirror/debian \
-md5-list /home/steve/md5.list \
/mirror/jigdo-test
If the -jigdo-* options are not used, the normal genisoimage execution
path is not affected at all. The above invocation will create 3 output
files (.iso, .jigdo and .template). Multiple -jigdo-ignore and
-jigdo-map options are accepted, for multiple ignore and map patterns.
Use the -md5-list option to specify the location of a list of files
and their md5sums in normal md5sum format. genisoimage will then compare
the checksum of each file it is asked to write against the checksum of
that file in the list. It will abort on any mismatches. The MD5 list
file must list all the files that are expected to be found and listed
in the output .jigdo file. The -jigdo-force-md5 option specifies a
path where all files are expected to have an MD5 entry
(e.g. /pool/). Then if any files do not have a match, they must have
been corrupted and genisoimage will abort.
More options have now been added in version 1.2 so that you can
specify the location of boot files within the ISO image. Previously
the four architectures alpha, hppa, mips and mipsel needed separate
tools to make an ISO image bootable. This also made life very hard
when trying to produce jigdo files. Instead, I've folded boot support
for those architectures into this patch so that genisoimage will do all
the work:
+-----------------------------------------------------------------------------+
|Alpha |
|-----------------------------------------------------------------------------|
|-alpha-boot |Specify the location of the boot image (relative to the root|
|<FILE> |of the ISO image) |
|-----------------------------------------------------------------------------|
|Hppa |
|-----------------------------------------------------------------------------|
|-hppa-cmdline |Specify the hppa boot command line. Separate elements with |
|<CMDLINE> |commas or spaces. |
|----------------+------------------------------------------------------------|
|-hppa-kernel-32 |Specify the location of the 32-bit boot image (relative to |
|<FILE> |the root of the ISO image) |
|----------------+------------------------------------------------------------|
|-hppa-kernel-64 |Specify the location of the 64-bit boot image (relative to |
|<FILE> |the root of the ISO image) |
|----------------+------------------------------------------------------------|
|-hppa-bootloader|Specify the location of the bootloader code (iplboot, |
|<FILE> |relative to the root of the ISO image) |
|----------------+------------------------------------------------------------|
|-hppa-ramdisk |Specify the location of the ramdisk (relative to the root of|
|<FILE> |the ISO image) |
|-----------------------------------------------------------------------------|
|Mips |
|-----------------------------------------------------------------------------|
|-mips-boot |Specify the location of the boot image (relative to the root|
|<FILE> |of the ISO image) |
|-----------------------------------------------------------------------------|
|Mipsel |
|Mipsel is awkward because we have to parse the ELF header of the boot loader |
|and write some locations from it into the boot sector. Ick! |
|-----------------------------------------------------------------------------|
|-mipsel-boot |Specify the location of the boot image (relative to the root|
|<FILE> |of the ISO image) |
+-----------------------------------------------------------------------------+
----------------------------------------------------------------------
How it works
I've hooked all the places in genisoimage where it will normally write image data.
All the normal data write calls (directory entries etc.) I simply pass through
and build into the template file. Any file data entries are passed through with
information about the original file. If that file is large enough, I grab the
filename and the MD5 of the file's data so I can just write a file match record
into the template file (and then the jigdo file).
----------------------------------------------------------------------
How fast is it?
On my laptop (600MHz P3, slow laptop disk) I can make a template file in
parallel with the ISO image from a typical 500MB data set in about 2 minutes.
By simply not creating the ISO (-o /dev/null), this time halves again. The data
set I'm using here is a copy of the woody i386 r2 update CD, as it's a handy
image I had lying around.
On my faster home server machine (1.7GHz Athlon, 512MB RAM, fast SCSI disks), I
can produce a 7GB DVD iso image with the jigdo and template files in about 8
minutes. A debian-cd run from start to finish to create DVD images takes about
25 minutes per architecture.
Genisoimage is normally I/O-bound on this system, but when running the jigdo
creation code it's now CPU bound - it's now running 2 MD5 checksums on each
data block that it sees. To boost performance when creating images, running 2
copies of debian-cd in parallel on a single system should parallelise nicely -
ideally run the CD and DVD versions of each architecture together to get
maximum benefit from the dentry and page cache.
----------------------------------------------------------------------
How to use jigit-mkimage
jigit-mkimage is a faster, local-only version of "jigdo-file
make-image", again written in portable C. It takes a few options:
+-----------------------------------------------------------------------------+
|-f <MD5 |Specify a file containing MD5sums for files we should attempt to |
|file> |use when rebuilding the image |
|---------+-------------------------------------------------------------------|
|-j <jigdo|Specify the input jigdo file |
|file> | |
|---------+-------------------------------------------------------------------|
|-t | |
|<template|Specify the input template file |
|file> | |
|---------+-------------------------------------------------------------------|
|-m <item=|Map <item> to <path> to find the files in the mirror |
|path> | |
|---------+-------------------------------------------------------------------|
|-M |Don't attempt to build the image; just verify that all the |
|<Missing |components needed are available. If some are missing, list them in |
|file> |the specified file. |
|---------+-------------------------------------------------------------------|
|-v |Make the output logging more verbose. |
|---------+-------------------------------------------------------------------|
|-l <log |Specify a logfile. If not specified, will log to stderr just like |
|file> |genisoimage |
|---------+-------------------------------------------------------------------|
| |Don't bother checking md5sums of the input files, or of the output |
|-q |image. |
| |WARNING: this may lead to corrupt images, but is much faster. |
|---------+-------------------------------------------------------------------|
|-s <start|Specify where to start in the image (in bytes). If not specified, |
|offset> |will start at the beginning (offset 0). Added for iso-image.pl use |
|---------+-------------------------------------------------------------------|
|-e <end |Specify where to end in the image (in bytes). If not specified, |
|offset> |will run all the way to the end of the image. Added for |
| |iso-image.pl use |
|---------+-------------------------------------------------------------------|
| |Don't attempt to reassemble the image; simply parse the image |
|-z |descriptor in the template file and print the image size. Added for|
| |iso-image.pl use |
+-----------------------------------------------------------------------------+
Specifying a start or end offset implies -q - it's difficult to check MD5 sums
if the full image is not generated!
----------------------------------------------------------------------
How to use iso-image.pl
iso-image.pl is a small perl wrapper script written to drive
jigit-mkimage and turn it into a CGI. It will parse the incoming
request (including byte-ranges) and call jigit-mkimage to actually
generate the image pieces wanted.
Configuration is simple: place iso-image.pl in a cgi-bin directory and set
various paths in the script:
* The path to jigit-mkimage
* A log file location
* The path to the template and jigdo files
* Other options to jigit-mkimage (e.g. -q and match locations)
----------------------------------------------------------------------
How to use jigit
Jigit is a small, user-friendly wrapper around jigit-mkimage to make
life easier for end users to download or upgrade existing CDs and CD
images. Configure it by editing /etc/jigit.conf or ~/.jigit.conf, then
run jigit with just one command line option, the name of the
distribution CD. Answer the questions it asks and it will do the rest.
----------------------------------------------------------------------
External integration
The current released version of debian-cd in etch supports JTE out of
the box. Wodim ships with integrated JTE code too.
----------------------------------------------------------------------
What's left to do?
1. Testing! :-) This is where you lot come in! Please play with this some more
and let me know if you have any problems, especially with data corruption.
2. More documentation.
3. Support for non-local mirrors in jigit-mkimage.
|
