1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
|
*******************************************************************************
******* WDF == Wiimms Disc File == Wii Disc File *******
*******************************************************************************
This file is part of the documentation of 'WIT' (Wiimms ISO Tools).
The most current version of this document can be found under:
http://wit.wiimm.de/file/WDF.txt
*******************************************************************************
******* What is WDF? *******
*******************************************************************************
This file describes the WDF format. WDF is an abbreviation of either
"Wiimms Disc File" or shorter "Wii Disc File".
To avoid misunderstandings: WDF will not replace WBFS. It is only a way to
shrink ISO images (reduces the disk usage) on other file systems like ext2,
FAT or NTFS. And while adding/extracting a WDF image to/from WBFS the
transformation can be done on the fly without time costs.
Wii ISO images has a size of about 4.4 GiB, but much space is unused and have
to be filled with zeros. Some games need less than 100 MiB and that means
about 4.3 GiB waste. The linux ext2/3 file system can handle such files very
good because it do not allocate disk space for such (zero-)holes. Such files
called "sparse files".
For me there is no need to store Wii ISO images in the old way filled with
zeros. And the handling with large sparse files is not effective and needs
always special options like 'ls -s' or 'cp --sparse=always'. Also it costs
much time because the holes have to be scanned again.
One way is to compress the data. But compressing works only for the holes but
not for the crypt data. And random access (needed by the WBFS subsystem) is
very difficult for compressed data. The direct support of holes seems to be
much better.
Because of this all I have defined a new file format, the WDF format. There
are some main issues for me:
1.) The WDF format should handles holes effectivly to save disk space.
2.) It should support splitted files because small files can sometimes be
handled more effectivly than one whole large files.
3.) There must be a WBFS tool to handle this kind of files. I have already
implemented the WDF support in my 'WIT' (Wiimms ISO Tools).
The WWT package contains some more tools to handle WDF directly.
4.) Other WBFS manager should support the WDF too. And this means that the
implementation must be done by the well known wrapper functions
'rw_sector_callback_t' and 'read_wiidisc_callback_t'. That makes
transfering the code very easy.
*******************************************************************************
******* History *******
*******************************************************************************
2009-10-03, WDF v1 (revision 1)
WDF was announced and implemented in Wiimms WBFS Tool (wwt v0.00d). Only the
planed internal split file management was never implemented and splitting
was handled extennal by break the files into peaces.
2012-09-23, WDF v1 (revision 2)
The 4 unused split file information members of the WDF header are now used
for compatibility and aligning infos. This makes it easier to accept future
compatible versions. But it is still WDF v1.
*******************************************************************************
******* Tests and statistics *******
*******************************************************************************
I have implemented the WDF support in my toolset 'WIT' since 2009 and tested
it with many games. In the tests I have converted games from ISO to WDF and
back. I done it with the tools 'wit', 'wwt' and 'wdf'. It work well and have
not found any errors.
Additionally I have testet more than 2 million non ISO files without any
failures. This works because WDF may detect holes in any files. For tests I
use the script: http://wit.wiimm.de/file/scripts/test-wdf-tools.sh
*******************************************************************************
******* Tools that supports WDF *******
*******************************************************************************
At the moment only the tools from WIT (Wiimms ISO Tools) will support WDFS:
- Tool 'wwt' may import WDF files directly to a WBFS
and may export discs from WBFS directly to WDF files.
- Tool 'wit' converts ISO images from and into WDF files.
- Tool 'wdf' general WDF, WIA and CISO conversion tool. It can be used as
'wdf-dump', 'wdf-cat', 'unwdf' -> http://wit.wiimm.de/wdf
See http://wit.wiimm.de/ for announcements and discussions.
*******************************************************************************
******* Data structures *******
*******************************************************************************
First a magic is defined to identify a WDF clearly. The magic should never be
a possible Wii ISO image identification. Wii ISO images starts with the ID6.
And so the WDF magic contains one control character (CTRL-A) within.
#define WDF_MAGIC "WII\1DISC"
#define WDF_MAGIC_SIZE 8
#define WDF_VERSION 1
A WDF file contains 3 parts (splittet files wil be discussed later):
WDF header (struct WDF_Header_t, including WDF_MAGIC)
data chunks (pure data, unstructured)
WDF_MAGIC (the MAGIC again)
WDF chunk table (struct WDF_Chunk_t)
The WDF header has an constant size and can be written as dummy and before
closing rewritten with the needed data. The chunk data is written 1:1. The
chunk control informations are collected an will be written at the end of the
file. There are some redundant information. They can be used for plausibility
checks. Remember: All data is stored in network byte order (big endian).
WDF header: WDF v1 revision 1
-----------------------------
typedef struct WDF_Head_t
{
// The magic
char magic[WDF_MAGIC_SIZE]; // WDF_MAGIC, what else!
u32 wdf_version; // WDF_VERSION
// Split file support. The values are always [*,0,1] because the
// internal split file support was never implemented or needed.
u32 split_file_id; // random number, for plausibility checks
// (any value, most 0, ignored)
u32 split_file_index; // zero based index ot this file
// (always 0, ignored)
u32 split_file_num_of; // total number of split files
// (always 1, ignored)
// virtual file infos
u64 file_size; // the size of the virtual file
// data size of this file
u64 data_size; // the ISO data size in this file
// (without header and chunk table)
// chunks
u32 chunk_split_file; // which split file contains the chunk table
// (always 0, ignored)
u32 chunk_n; // total number of data chunks
u64 chunk_off; // the 'MAGIC + chunk_table' file offset
} WDF_Head_t;
The 'split_*' fields are described below.
The 'file_size' contains the total size of the virtual ISO image.
The 'data_size' defines the total data size in current split file.
The 'chunk_off' data points to the file offset of the table.
WDF header: WDF v1 revision 2 & WDF v>1
---------------------------------------
This header is full compatible to revision 1. Only the yet unused split members
have now new names. For WDF v1, these members are set to the old split values.
But the new definition allow easier support of future extensions
typedef struct WDF_Head_t
{
// the magic
char magic[WDF_MAGIC_SIZE]; // WDF_MAGIC, what else!
u32 wdf_version; // WDF_VERSION
u32 wdf_head_size; // size of version related WDF_Head_t
// (WDF v1: ignored)
u32 align_factor; // info: all data is aligned with a multiple of #
// (WDF v1: always 0, ignored)
u32 wdf_compatible; // this file is compatible down to version #
// (WDF v1: always 1)
// virtual file infos
u64 file_size; // the size of the virtual file
// data size of this file
u64 data_size; // the ISO data size in this file
// (without header and chunk table)
// chunks
u32 chunk_size_factor; // info: all chunk sizes are multiple of #
// (WDF v1: always 0, ignored)
u32 chunk_n; // total number of data chunks
u64 chunk_off; // the 'MAGIC + chunk_table' file offset
} WDF_Head_t;
WDF chunk info: WDF v1
----------------------
typedef struct WDF_Chunk_t
{
u32 split_file_index; // which split file conatins that chunk
// (WDF v1: always 0, ignored)
u64 file_pos; // the virtual ISO file position
u64 data_off; // the data file offset
u64 data_size; // the data size
} WDF_Chunk_t;
WDF chunk info: WDF v>1 (planned)
---------------------------------
It is planned, that the 'split_file_index' is removed, if WDF v2 is definied.
typedef struct WDF_Chunk_t
{
u64 file_pos; // the virtual ISO file position
u64 data_off; // the data file offset
u64 data_size; // the data size
} WDF_Chunk_t;
The chunk table (WDF_Chunk_t) is always sorted by the iso_pos_* in ascending
order so that a lookup can be done by a binary search.
There is always a chunk at virtual ISO file position #0 an one chunk that
reaches the end of the virtual file. If there are holes at the beginning or
end of the data than chunks with data_size==0 bytes have to be insertet.
This is an example dump of the game "Animal Crossing". The dump was created
with the tool 'wdf-dump' which is part of 'wit' (Wiimms ISO Tool).
| # wdf-dump --chunk pool/wdf/animal.wdf
|
| wdf-dump v0.02b r191M - Dirk Clemens - 2009-10-07
|
|
| WDF dump of file pool/wdf/animal.wdf
|
| Header:
|
| Magic : "WII.DISC" 57 49 49 01 44 49 53 43
| wdf_version : 1/hex = 1
| split_file_id : 0/hex = 0
| split_file_index : 0/hex = 0
| split_file_num_of : 1/hex = 1
| file_size : 118240000/hex = 4699979776
| - WDF file size : 1557cd40/hex = 358075712 7.62%
| data_size : 1557cbcc/hex = 358075340
| chunk_split_file : 0/hex = 0
| chunk_n : b/hex = 11
| chunk_off : 1557cc04/hex = 358075396
|
| File Parts:
|
| Header : 0 .. 38 [ 38]
| Data : 38 .. 1557cc04 [ 1557cbcc]
| Chunk-Magic : 1557cc04 .. 1557cc0c [ 8]
| Chunk-Table : 1557cc0c .. 1557cd40 [ 134]
|
| Chunk Table:
|
| idx WDF file address data len virtual ISO address hole size
| ------------------------------------------------------------------------
| 0. 38.. 6c 34 0.. 34 3ffcc
| 1. 6c.. 98 2c 40000.. 4002c dfd4
| 2. 98.. b8 20 4e000.. 4e020 1de0
| 3. b8.. 1164 10ac 4fe00.. 50eac 7154
| 4. 1164.. 1880 71c 58000.. 5871c 178e4
| 5. 1880.. 191880 190000 70000.. 200000 f600000
| 6. 191880.. 19272c eac f800000.. f800eac 7154
| 7. 19272c.. 19cc04 a4d8 f808000.. f8124d8 db28
| 8. 19cc04.. 97cc04 7e0000 f820000.. 10000000 f2800000
| 9. 97cc04.. 1557cc04 14c00000 102800000.. 117400000 e40000
| 10. 1557cc04.. 1557cc04 0 118240000.. 118240000 0
|
*******************************************************************************
******* Split files *******
*******************************************************************************
There is only a small need for splitted files, e.g. when transporting a large
image via a FAT file system. The WDF splits the data anyway in chunks. So it
is only a little step to devide the chunks into more than 1 file.
---------------------
Parameter overview:
---------------------
WDF_Head_t:
split_file_id : any (random) value, but same in all files.
split_file_index : unique index [0..]
split_file_num_of : Total number of split files [1..]
chunk_split_file : Index of file which contais the chunk table.
WDF_Chunk_t
split_file_index : Index of split fle where the chunk data resides.
---------------------
Here are the rules:
---------------------
* Each file of the (splitted) set contains the same header at the beginning
of the file. The only difference in the headers is the 'split_file_index',
a zero based index.
* 'split_file_num_of' gives the numer of files in the set. 'split_file_index'
is always smaller than 'split_file_num_of'.
* 'split_file_id' may have any (random) value but must be same in all files.
* The chunk table is stored in the split file with index 'chunk_split_file'.
* Each chunk has a parameter called 'split_file_index' which declares in which
file the chunk data resides.
* The filename of split file #0 is the base name. The filenames of all other
split files have the identical base name plus '.' plus an decimal index
taken from 'split_file_index' without any padding.
Example:
file #0: my_game.wdf
file #1: my_game.wdf.1
file #2: my_game.wdf.2
...
file #9: my_game.wdf.9
file #10: my_game.wdf.10
---------------------
Style guide:
---------------------
Here are some rules how to write a good splitted WDF. Assume that the user has
set a limit for each file. No file schould be larger than this limit.
* Open the first file (index #0) and write a dummy header to that file.
* While writing chunks check if the limit will be exceeded.
If so:
- Split the current chunk into 2 chunks: The first chunks contains enough
data to reach the limit. The second chunk takes the remaining data.
- Open the next split file with a filename described above and write a
dummy header to that file.
* If all data is written then:
- Calculate the chunk tables size including the magic. If the data will
execeed the limit open a new split file and write the dummy header.
- Write the chunk table to the current file (remember: network byte order).
- Calculate the header and write it to all data files. Don't forget the
individual 'split_file_index' and the network byte order.
- Close all files.
REMARK:
The WIT tools supports splitted WDF files *not* following this rules. The
files are splitted hard by breaking the files into peaces. This is done by
the file layer so that the WDF layer don't see the split.
*******************************************************************************
******* Example implementation *******
*******************************************************************************
The example implementation is part of 'WIT' (Wiimms ISO Tools):
see: http://wit.wiimm.de/file/src/lib-wdf.c
and: http://wit.wiimm.de/file/src/lib-wdf.h
*******************************************************************************
******* END *******
*******************************************************************************
|
