.. _imap-developer-guidance-mailbox-format:

Cyrus IMAP Server: Mailbox File Formats
=======================================

Intro
-----

This documentation refers to the "version 12" cyrus index format and
associated mailbox files.

No external tools should make use of this information. The only
supported method of access to the mail store is through the standard
interfaces: IMAP, POP, NNTP, LMTP, etc.

A cyrus mailbox is a directory in the filesystem. It contains the
following files:

-  zero or more message files
-  the ``cyrus.header`` metadata file
-  the ``cyrus.index`` metadata file
-  the ``cyrus.cache`` metadata file
-  zero or one ``cyrus.squat`` search indexes
-  zero or more subdirectories

With "split metadata" configuration, the mailbox may actually be split
between multiple disks, with the files being in the same relative
directory on the meta disk. See the ``imapd.conf`` option
``metapartition_files`` for more information

Message Files
-------------

The message files are named by their UID, followed by a ".", so UID 423
would be named "``423.``". They are stored in wire-format: lines are
terminated by CRLF and binary data is not allowed.

``cyrus.header``
----------------

This file contains mailbox-wide information that does not change often.
Its format:

::

    <Mailbox Header Magic String>
    <Quota Root>\t<Mailbox Unique ID String>\n
    <Space-separated list of user flags>\n
    <Mailbox ACL>\n

The Mailbox Unique ID String is used for non-owner per-user \\Seen flags
so they remain with the mailbox during renames, and also by the
replication subsystem to detect mailbox renames.

The ACL is a copy of the value stored in mailboxes.db, and isn't
actually used.

Locking Considerations
~~~~~~~~~~~~~~~~~~~~~~

The ``cyrus.index`` file must be locked in exclusive mode while making
changes to the ``cyrus.header`` file to ensure consistency. All changes
are made by rewriting the entire file and renaming the new version into
place.

``cyrus.cache``
---------------

The ``cyrus.cache`` file is a pure cache of information that's also
present in the message files. It exists to make ENVELOPE and specific
header fetches more efficient, as well as to assist with searches and
sorts.

If a ``cyrus.cache`` file is missing or corrupted, it can be
re-generated by running a ``reconstruct`` on the mailbox.

The format is 10 individual records each prefixed with a 32 bit length
value in network byte order. The offset of each message's cache record
is stored in the ``cyrus.index`` file (documented below). The records in
a cyrus.cache file are of variable length, depending on the contents of
the associated message.

The first 4 bytes of the cyrus.cache file are a "generation number"
which must match the first 4 bytes of the associated cyrus.index file.
In the past this was used to track consistency between the files, but
the name locking scheme and per-record CRC check in cyrus 2.4 and above
means this is just a backup consistency check rather than an essential
format feature.

::

    +------------------------------------------------------------------------+
    |Gen # (32bits)|Size 1 (32bits)|Data 1                                   |
    +------------------------------------------------------------------------+
    |           |Size 2 (32bits)|Data 2            |Size 3 (32bits)| Data 3  |
    +------------------------------------------------------------------------+
    | .....                                                                  |
    +------------------------------------------------------------------------+

While there are occasional changes to the cache format, this information
is NOT stored in the cyrus.cache file. Instead, there is a
"cache\_version" field in the cyrus.index record, so multiple different
versions of cache data may exist in the same cache file.

The order of fields per record in the cache file is as follows: (keep in
mind that they are all preceded by a 4 byte network byte order size).

Envelope Response
    Raw IMAP response for a request for the envelope.
Bodystructure Response
    Raw IMAP response for a request for the bodystructure.
Body Response
    Raw IMAP response for an (old style) request for the body.
Binary Bodystructure
    Offsets into the message file to pull out various body parts.
    Because of the nature of MIME parts, this is somewhat recursive.

    This looks like the following (starting the octet following the
    cache field size). All of the fields are bit32s.

    ::

          [
           [Number of message parts+1 for the rfc822 header if present]
           [
            [Offset in the message file of the header of this part]
            [Size (octets) of the header of this part]
            [Offset in the message file of the content of this part]
            [Size (octets) of the content of this part]
            [Encoding Type of this part]
           ]
              (repeat for each part as well as once for the headers)
           [zero *or* number of sub-parts in the case of a multipart.
            if nonzero, this is a recursion into the top structure]
              (repeat for each part)
          ]

    Note if this is not a message/rfc822, than the values for the sizes
    of the part 0 are -1 (to indicate that it doesn't exist). Sub-parts
    are not possible for a part 0, so they aren't included when finding
    recursive entries.

    The offset and size info for both the mime header and content part
    are useful in order to do fast indexing on the appropriate parts of
    the message file when a client does a FETCH request for
    BODY[HEADER], or BODY[2.MIME].

    Note that the top level RFC822 headers are a treated as a separate
    part from their body text ("0" or "HEADER").

    In the case of a multipart/alternative, the content size & offset
    refers to the size of the entire mime part.

    A very simple message (with a single text/plain part) would
    therefore look like:

    ::

          [[2][rfc822 header][text/plain body part info][0]]

    A simple multipart/alternative message might look like:

    ::

          [[3][rfc822 header][text/plain message part info]
              [second message part info][0][0]]

    A message with an attachment that has two subparts:

    ::

          [[3][rfc822 header info][rfc822 first body part info][attachment info][0][
                [3][NIL header info][sub part 1 info][sub part 2 info][0][0]]]

    A message with an attached message/rfc822 message with the following
    total structure:

    ::

            message/rfc822
              0 headers; content-type: multipart/mixed
              1 text/plain
              2 message/rfc822
                0 headers; content-type: multipart/alternative
                1 text/plain
                2 text/html

    ::

          [[3][rfc822 header part 0][text/plain part 1][overall attachment info][0][
               [3][rfc822 header part 2.0][text/plain part 2.1][text/html part 2.2]
                  [0][0]]]

Cache Header
    Any cached header fields. The exact set of fields here depends on
    the cache record version - there is a function in ``imap/mailbox.c``
    to determine if a named header would be cached based on the version.
    These are in the same format they would appear in the message file:

    ::

          HeaderName: headerdata\r\n

    Examples include: References, In-Reply-To, etc.

From
    The from header.
To
    The to header.
Cc
    The CC header.
Bcc
    The BCC header.
Subject
    The Subject header.

Locking Considerations
~~~~~~~~~~~~~~~~~~~~~~

The ``cyrus.index`` file must be locked in exclusive mode while making
changes to the ``cyrus.cache`` file to ensure consistency. All new cache
records are created by reading the current end-of-file offset, appending
the new cache record, and storing that start offset into the associated
cyrus.index record.

``cyrus.index``
---------------

The cyrus.index file is NOT just a cache - it stores information not
present in the message file!

The cyrus.index file consists of a fixed width header, followed by fixed
width records. In the past, it would be rewritten on every expunge, but
since Cyrus 2.4 the expunged records remain in the cyrus.index file for
a configurable time to support QRESYNC and more efficient delayed
expunge.

The cyrus.index file is the "heart" of the mailbox format - containing
checksums (CRC32) of everything else, and the most frequently updated
fields. All fields are stored in network byte order and aligned on 4
byte boundaries. Due to some 64 bit values being stored, the header and
individual records are aligned on 8 byte boundaries.

The overall format looks sort of like this:

::

    cyrus.index:
    +----------------+
    | Mailbox Header |
    +----------------+
    | Msg: Num 1     |
    +----------------+
    | Msg: Num 2     |
    +----------------+
    |     ...        |
    +----------------+

The basic idea being that there is one header, and then all the message
records are evenly spaced throughout the file. All of the message
records are at well-known offsets, making any part of the file
accessable at roughly equal speed.

Locking Considerations
~~~~~~~~~~~~~~~~~~~~~~

``cyrus.index`` files can not be repacked (i.e. records can not change
UID for a particular offset, and the file can't be rewritten or deleted)
unless there's an exclusive namelock held for the mailbox name. This is
to avoid race conditions and simplify the use of mailboxes. Whenever a
mailbox is opened, the caller holds a shared namelock on the mailbox
name for the duration of the "mailbox object"'s existence.

All reads of a ``cyrus.index`` file must be done with a lock held, and
all writes must be done with an exclusive lock held. This ensures CRC32
checksums of individual headers and records are always consistent. There
are no direct "offset" reads done any more, instead the mailbox API
provides a way to read an entire cyrus.index header or cyrus.index
record into a struct, performing consistency checks. Writes are also
done with a complete record struct.

Detail of ``cyrus.index`` header
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The index header contains the following information, in order:

Generation Number (4 bytes)
    A number that is basically the "revision number" of the mailbox. It
    must match between the cache and index files. This is to ensure that
    if we fail to sync both the cache and index files and a crash
    happens (so that only one is synced), we do not provide bad data to
    the user. This is also backed by having individual cache checksums
    on each record.
Format (4 bytes)
    Basically obsolete (indicates netnews or regular).
Minor Version (4 bytes)
    Indicates the version number of the index file. This can be used for
    on-the-fly upgrades of the index and cache files.
Start Offset (4 bytes)
    Size of index header.
Record Size (4 bytes)
    Size of an index record.
Num Records (4 bytes)
    How many records are in this index (including records for expunged
    records. See below for "Exists" which has moved from pre-version 12
    files.
Last Appenddate (4 bytes)
    (time\_t) of the last time a message was appended
Last UID (4 bytes)
    Highest UID of all messages in the mailbox (UIDNEXT - 1).
Quota Mailbox Used (8 bytes)
    Total amount of storage used by all of the messages in the mailbox.
    Platforms that don't support 64-bit integers only use the last 4
    bytes.
POP3 Last Login (4 bytes)
    (time\_t) of the last pop3 login to this INBOX, used to enforce the
    "poptimeout" ``imapd.conf`` option.
UIDvalidity (4 bytes)
    The UID validitiy of this mailbox. Cyrus currently uses the
    ``time()`` when this mailbox was created.
Deleted, Answered, and Flagged (4 bytes each)
    Counts of how many messages have each flag.
Mailbox Options (4 bytes)
    Bitmask of mailbox options, consisting of any combination of the
    following:

    POP3\_NEW\_UIDL
        Flag signalling that we're using "*uidvalidity*.\ *uid*" instead
        of just "*uid*" for the output of the POP3 UIDL command.
    IMAP\_SHAREDSEEN
        Flag signalling that we're supporting a shared \\Seen flag on
        the mailbox.
    IMAP\_DUPDELIVER
        Flag signalling that we're allowing duplicate delivery of
        messages to the mailbox, overriding system-wide duplicate
        suppression.
    MAILBOX\_NEEDS\_REPACK
        Flag signalling that the mailbox is due to be repacked. During
        mailbox\_close() every process will attempt to take an exclusive
        namelock on the mailbox and repack.
    MAILBOX\_DELETED
        Flag signalling that the mailbox is deleted. This can be set
        with a shared namelock, and indicates to all other users of the
        mailbox that they need to close it and attempt cleanup. The last
        process to close the mailbox will perform the final cleanup
        under an exclusive namelock, giving the other processes a chance
        to finish their current operation first without files
        disappearing from under them!

Leaked Cache (4 bytes)
    Number of leaked records in the cache file.
Highest ModSeq (8 bytes)
    Highest Modification Sequence of all the messages in the mailbox
    (CONDSTORE).
Deleted ModSeq (8 bytes)
    Lowest Modification Sequence before which expunged message data may
    have been purged from the mailbox and forgotten (CONDSTORE/QRESYNC
    support).
Exists (4 bytes)
    See NumRecords above. This is the count of non-expunged records in
    the mailbox and corresponds to the IMAP status item "EXISTS".
First Expunged (4 bytes)
    lowest modified time of an expunged message in this mailbox (or zero
    if there are no expunged messages) - used to determine if the
    mailbox needs repacking.
Last Repack Time (4 bytes)
    a timestamp for the last repack, to ensure repacks aren't done too
    close together if expunges were closely spaced
Header File CRC (4 bytes)
    CRC32 value of the bytes in the ``cyrus.header`` file for this
    mailbox. Must be rewritten whenever the cyrus.header file is changed
    (see locking considerations above - this is why the cyrus.index must
    be exclusively locked!)
Sync CRC (4 bytes)
    An XOR of the CRC32 of a specially generated value for each of the
    non-expunged records in this mailbox. This is a cached value which
    allows the replication subsystem to quickly determine that all
    non-expunged records in a mailbox are in sync and detect possible
    "split brain" scenarios with low bandwidth use.
Recent UID (4 bytes)
    The highest UID last time an IMAP client logged in as the mailbox
    owner (or anybody if SHAREDSEEN is enabled) selected this mailbox.
    Used to generate the \\Recent flags in IMAP
Recent Time (4 bytes)
    Used for consistency with the seen\_db code, but probably not
    actually necessary. Oh well
Header CRC (4 bytes)
    Must always be the LAST record of the header. This is the CRC32 of
    the actual bytes on disk (network order format) for the rest of the
    cyrus.index. By keeping it last, it can be easily calculated with
    the following snippet of code:
    ``crc = crc32_map(buf, OFFSET_HEADER_CRC);`` - i.e. crc32 from the
    start of the buffer to just before this field.

There are also spare fields in the index header, to allow for future
expansion without forcing an upgrade of the file, and to round up to be
divisible by 8 bytes.

Detail of ``cyrus.index`` records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These records start immediately following the ``cyrus.index`` header,
and are all fixed size. They are in-order by uid of the message.

UID (4 bytes)
    UID of the message
INTERNALDATE (4 bytes)
    INTERNALDATE of the message (where possible, this matches the
    creation and modification times of the file on disk to help
    reconstruct in the event of data loss)
SENTDATE (4 bytes)
    Contents of the Date: header chomped to day resolution with timezone
    stripped.
SIZE (4 bytes)
    Size of the whole message (in octets)
HEADER SIZE (4 bytes)
    Size of the message header (in octets)
GMTIME (4 bytes)
    Contents of the Date: header at 1 second resolution and converted to
    GMT (for sort)
CACHE\_OFFSET (4 bytes)
    Offset into the ``cyrus.cache`` file for the beginning of this
    message's cache entry.
LAST UPDATED (4 bytes)
    (time\_t) of the last time this record was changed
SYSTEM FLAGS (4 bytes)
    Bitmask showing which system flags are set/unset
USER FLAGS (MAX\_USER\_FLAGS / 32 bytes)
    Bitmask showing which user flags are set/unset (bits correspond to
    positions in the cyrus.header flag list, i.e. (1<<0) == the flag
    name
CONTENT\_LINES (4 bytes)
    Number of text lines contained in the message content (body).
CACHE\_VERSION (4 bytes)
    Indicates the version number of the cache record for the message
    (determines which headers are cached, see list in mailbox.c).
GUID (MESSAGE\_GUID\_SIZE bytes)
    Globally Unique IDentifier of the message (used by replication
    engine). This is the sha1 value of the bytes as stored on disk.
MODSEQ (8 bytes)
    Modification Sequence of the message (CONDSTORE).
CACHE\_CRC (4 bytes)
    This is the CRC32 of all the bytes of the cache record (all 10
    fields) as stored on disk. Again, calculated over the exact bytes
    stored in the ``cyrus.cache`` file.
RECORD\_CRC (4 bytes)
    Like the header CRC - this is the CRC32 of all the bytes in on-disk
    order that exist in this record. Records are always rewritten as the
    entire record, including the updated CRC, so it's always consistent
    if you have a lock on the ``cyrus.index`` file, because writers will
    wait until they get an exclusive lock to make modifications.

Notes
-----

-  Expunge is super quick now - it's just a flag update!
-  Append is relatively fast (it only adds to the end of both the cache
   and index files and modifies the index header)
-  Message unlinks always happen during the "close" phase - which may be
   noticed when you select another mailbox, but otherwise are delayed
   from the actual action. With delayed expunge, the unlinks are pushed
   off to cyr\_expire which is a background task, and will never be
   noticed by the user.
-  Message delivery is something like this:

   #. write/sync message file
   #. write/sync new ``cyrus.cache`` record
   #. write/sync new ``cyrus.index`` record
   #. calculate, write, sync new ``cyrus.index`` header
   #. acknowledge message delivery

   The message isn't delivered until the new index header is written. In
   case of a crash before the new index header is written, any previous
   writes will be overwritten on the next delivery (and will not be
   noticed by the readers).

   Note that certain power failure situations (power failure in the
   middle of a disk sector write) could cause a mailbox to need
   reconstruction (possibly even losing some flag state). These failure
   modes are not possible in the "Hardware RAID disk model" (which we
   will describe somewhere else when we get around to it).

Future considerations
---------------------

-  Cache all header fields? (or all up to Xk?) This could greatly
   improve speeds of clients that just ask for everything, but also
   increases the expense of rewriting the cache file (as well as the
   size it takes on disk).
-  Reformat cache file to use a
   (size)(size)(size)(size)(data)(data)(data) format. This makes
   accesses anywhere in the cache file equally fast, as opposed to
   having to iterate through all the entires for a given message to get
   to the last one. Note that either way is still O(1) so maybe it
   doesn't matter much.
-  It would be useful to store a uniqueid -> mailbox name index, so that
   we could fix arbitron again.