File and Directory IDs explained.
What are File and Directory ID's?
On a mac the file system stores all files and directory information on each
volume in a big file called the catalogue file. Inside the catalogue, all
files and directories are accessed using a key, central to which is a number,
called the ID. In the case of a file its a file ID (FID) or for a directory,
a directory ID (DID).
How many IDs can there be?
The ID in a catalogue key is stored using 4 bytes, which means it can only
be between 0 and 4,294,967,295 (FF FF FF FF in hex). However the first 16
IDs are reserved so you don't have quite that many. Each ID is unique for
ever, on any given volume. Once all 4 billion have been used, you cannot
create new files, so will need to reformat the volume to continue using it.
Why are IDs so important?
Most system calls relating to files inside a mac (either vi a network or a
hard disk) can refer to files or directories by ID. This makes the ID a
powerful piece of information.
So whats the problem?
The problem lies in file servers that don't use IDs. The protocol used by
macs to share files over a network is called the Apple Filing Protocol (AFP)
and it requires the use of IDs. So if you want to support AFP fully, any
AFP server, must adopt its own system for storing and maintaining a link
between each file or directory and its respective ID. This is most critical
when acessing directories.
So why does this matter on a non mac server like netatalk?
The three big stumbling blocks that crop up with AFP servers that don't
fully support IDs are 1) aliases, 2) the trash can and 3) linked documents.
An alias on a mac is quite special. Rather than just storing the path to
the original file (like a unix symlink), it stores the ID to that file and
a special identifier for the volume (and the server it's on). Ideally this
is great. If the file moves or is renamed, the alias still works. However
if either the file (or directory) ID changes, or the volume identifier
(or server identifer), then the alias will break. The file it claims to
point to will claim to have been removed.
Trash can (accidentally deleted file) problems.
The trash can has similar problems. Files that have been moved to the trash
are represented by their ID. When you empty the trash all ID's listed are
deleted. However if the ID of a file that was in the trash, is reallocated
to an ordinary file, then when the trash is emptied that file will be deleted.
Linked document problems.
Finally linked documents: Linked documents are documents that contain hidden
links to other documents. Print setting and layout application (such as
Quark) use this technique. Sometimes these documents contain IDs linking to
their embeded documents. These can break in the same way as aliases.
So how does netatalk approach the problem?
!!! The following is outdated, please refer to the Manual instead !!!
Netatalk has two different methods of allocating IDs: last and cnid.
DID = last.
This uses a running number to allocate IDs. When an ID is allocated the
server remembers this by adding it to a table. If an ID is referenced, then
the server looks up on the table. When the server is restarted, the table is
lost. This is the most simple method, but it is unreliable. If you stick to
the mac features which don't rely heavily on IDs it works fine. If you try
to use IDs much, things break.
DID = cnid.
The CNID scheme in Netatalk attempts to assign unique IDs to each file and
directory, then keep those IDs persistent across mounts of the volume. This
way, cross-volume aliases will work, and users are less likely to encounter
duplicate CNID errors. Prior to Netatalk 1.6.0, the CNID calculation
scheme was not persistent, and IDs were assigned based on the UNIX device and
inode number of a given file or directory (see DID = last above). This was
fine for the most part, but due to limitations, not all available CNIDs could
be used. As well, these IDs could change independently from Netatalk, and
thus were not persistent. As of Netatalk 1.6.0, the CNID scheme is now the
default. On top of that, Netatalk uses the Concurrent Datastore method to
avoid the need for database locking and transactions.
As stated above, CNID requires Berkeley DB. Currently, Netatalk supports
BDB 4.1.25 and 4.2.52 The recommended version is 4.2.52 as that is the version
on which most testing has been done.
CNID has seen many contributors over the years. It was conceived by
Adrian Sun <email@example.com>. His developer notes can be found
libatalk/cnid/README file. It was later picked up and modernized by Uwe Hees
<firstname.lastname@example.org>. Then, Joe Marcus Clarke <email@example.com>
started fixing bugs and adding additional features. The Concurrent
Datastore support was subsequently added by Dan Wilga <firstname.lastname@example.org>.
The CNID code is currently maintained by Joe Marcus Clarke.