1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
|
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Cyrus IMAP Server: Mailbox File Formats — Cyrus IMAP 3.12.1 documentation</title>
<link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/cyrus.css" type="text/css" />
<script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
<script src="../../../_static/jquery.js"></script>
<script src="../../../_static/underscore.js"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../../_static/doctools.js"></script>
<script src="../../../_static/sphinx_highlight.js"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="Cyrus IMAP Server: Namelocks" href="namelocks.html" />
<link rel="prev" title="Cyrus IMAP Server: Locking" href="locking.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
Cyrus IMAP
</a>
<div class="version">
3.12.1
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Cyrus IMAP</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../download.html">Download</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../quickstart.html">Quickstart Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../setup.html">Setup</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../operations.html">Operations</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../../developers.html">Developers</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../../contribute.html">We need your help</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documentation.html">Contribute docs</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../developer.html">Contribute code and tests</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cyrusworks.html">Cyrus.Works</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../../../developers.html#cyrus-internals">Cyrus Internals</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../API.html">Cyrus APIs</a></li>
<li class="toctree-l3"><a class="reference internal" href="../thoughts.html">Thoughts & Notes</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="../guidance.html">Guidance for Developers</a><ul class="current">
<li class="toctree-l4"><a class="reference internal" href="hacking.html">Cyrus IMAP Server: Hacking</a></li>
<li class="toctree-l4"><a class="reference internal" href="internationalization.html">Cyrus IMAP Server: Internationalization</a></li>
<li class="toctree-l4"><a class="reference internal" href="locking.html">Cyrus IMAP Server: Locking</a></li>
<li class="toctree-l4 current"><a class="current reference internal" href="#">Cyrus IMAP Server: Mailbox File Formats</a></li>
<li class="toctree-l4"><a class="reference internal" href="namelocks.html">Cyrus IMAP Server: Namelocks</a></li>
<li class="toctree-l4"><a class="reference internal" href="prot.html">Cyrus IMAP Server: prot layer</a></li>
<li class="toctree-l4"><a class="reference internal" href="replication_examples.html">Cyrus IMAP Server: Replication Examples</a></li>
<li class="toctree-l4"><a class="reference internal" href="replication_protocol.html">Cyrus IMAP Server: Replication Protocol v2.4+</a></li>
<li class="toctree-l4"><a class="reference internal" href="special_chars.html">Cyrus IMAP Server: Special Characters</a></li>
<li class="toctree-l4"><a class="reference internal" href="var_directory_structure.html">Cyrus IMAP Server: var directory structure</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../developers.html#unit-tests">Unit Tests</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../support.html">Support/Community</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Cyrus SASL</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="http://www.cyrusimap.org/sasl">Cyrus SASL</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">Cyrus IMAP</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../../developers.html">Developers</a></li>
<li class="breadcrumb-item"><a href="../guidance.html">Developer Guidance</a></li>
<li class="breadcrumb-item active">Cyrus IMAP Server: Mailbox File Formats</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/developer/guidance/mailbox-format.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="cyrus-imap-server-mailbox-file-formats">
<span id="imap-developer-guidance-mailbox-format"></span><h1>Cyrus IMAP Server: Mailbox File Formats<a class="headerlink" href="#cyrus-imap-server-mailbox-file-formats" title="Permalink to this heading"></a></h1>
<section id="intro">
<h2>Intro<a class="headerlink" href="#intro" title="Permalink to this heading"></a></h2>
<p>This documentation refers to the "version 12" cyrus index format and
associated mailbox files.</p>
<p>No external tools should make use of this information. The only
supported method of access to the mail store is through the standard
interfaces: IMAP, POP, NNTP, LMTP, etc.</p>
<p>A cyrus mailbox is a directory in the filesystem. It contains the
following files:</p>
<ul class="simple">
<li><p>zero or more message files</p></li>
<li><p>the <code class="docutils literal notranslate"><span class="pre">cyrus.header</span></code> metadata file</p></li>
<li><p>the <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> metadata file</p></li>
<li><p>the <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> metadata file</p></li>
<li><p>zero or one <code class="docutils literal notranslate"><span class="pre">cyrus.squat</span></code> search indexes</p></li>
<li><p>zero or more subdirectories</p></li>
</ul>
<p>With "split metadata" configuration, the mailbox may actually be split
between multiple disks, with the files being in the same relative
directory on the meta disk. See the <code class="docutils literal notranslate"><span class="pre">imapd.conf</span></code> option
<code class="docutils literal notranslate"><span class="pre">metapartition_files</span></code> for more information</p>
</section>
<section id="message-files">
<h2>Message Files<a class="headerlink" href="#message-files" title="Permalink to this heading"></a></h2>
<p>The message files are named by their UID, followed by a ".", so UID 423
would be named "<code class="docutils literal notranslate"><span class="pre">423.</span></code>". They are stored in wire-format: lines are
terminated by CRLF and binary data is not allowed.</p>
</section>
<section id="cyrus-header">
<h2><code class="docutils literal notranslate"><span class="pre">cyrus.header</span></code><a class="headerlink" href="#cyrus-header" title="Permalink to this heading"></a></h2>
<p>This file contains mailbox-wide information that does not change often.
Its format:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o"><</span><span class="n">Mailbox</span> <span class="n">Header</span> <span class="n">Magic</span> <span class="n">String</span><span class="o">></span>
<span class="o"><</span><span class="n">Quota</span> <span class="n">Root</span><span class="o">></span>\<span class="n">t</span><span class="o"><</span><span class="n">Mailbox</span> <span class="n">Unique</span> <span class="n">ID</span> <span class="n">String</span><span class="o">></span>\<span class="n">n</span>
<span class="o"><</span><span class="n">Space</span><span class="o">-</span><span class="n">separated</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">user</span> <span class="n">flags</span><span class="o">></span>\<span class="n">n</span>
<span class="o"><</span><span class="n">Mailbox</span> <span class="n">ACL</span><span class="o">></span>\<span class="n">n</span>
</pre></div>
</div>
<p>The Mailbox Unique ID String is used for non-owner per-user \Seen flags
so they remain with the mailbox during renames, and also by the
replication subsystem to detect mailbox renames.</p>
<p>The ACL is a copy of the value stored in mailboxes.db, and isn't
actually used.</p>
<section id="locking-considerations">
<h3>Locking Considerations<a class="headerlink" href="#locking-considerations" title="Permalink to this heading"></a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> file must be locked in exclusive mode while making
changes to the <code class="docutils literal notranslate"><span class="pre">cyrus.header</span></code> file to ensure consistency. All changes
are made by rewriting the entire file and renaming the new version into
place.</p>
</section>
</section>
<section id="cyrus-cache">
<h2><code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code><a class="headerlink" href="#cyrus-cache" title="Permalink to this heading"></a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> file is a pure cache of information that's also
present in the message files. It exists to make ENVELOPE and specific
header fetches more efficient, as well as to assist with searches and
sorts.</p>
<p>If a <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> file is missing or corrupted, it can be
re-generated by running a <code class="docutils literal notranslate"><span class="pre">reconstruct</span></code> on the mailbox.</p>
<p>The format is 10 individual records each prefixed with a 32 bit length
value in network byte order. The offset of each message's cache record
is stored in the <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> file (documented below). The records in
a cyrus.cache file are of variable length, depending on the contents of
the associated message.</p>
<p>The first 4 bytes of the cyrus.cache file are a "generation number"
which must match the first 4 bytes of the associated cyrus.index file.
In the past this was used to track consistency between the files, but
the name locking scheme and per-record CRC check in cyrus 2.4 and above
means this is just a backup consistency check rather than an essential
format feature.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">+------------------------------------------------------------------------+</span>
<span class="o">|</span><span class="n">Gen</span> <span class="c1"># (32bits)|Size 1 (32bits)|Data 1 |</span>
<span class="o">+------------------------------------------------------------------------+</span>
<span class="o">|</span> <span class="o">|</span><span class="n">Size</span> <span class="mi">2</span> <span class="p">(</span><span class="mi">32</span><span class="n">bits</span><span class="p">)</span><span class="o">|</span><span class="n">Data</span> <span class="mi">2</span> <span class="o">|</span><span class="n">Size</span> <span class="mi">3</span> <span class="p">(</span><span class="mi">32</span><span class="n">bits</span><span class="p">)</span><span class="o">|</span> <span class="n">Data</span> <span class="mi">3</span> <span class="o">|</span>
<span class="o">+------------------------------------------------------------------------+</span>
<span class="o">|</span> <span class="o">.....</span> <span class="o">|</span>
<span class="o">+------------------------------------------------------------------------+</span>
</pre></div>
</div>
<p>While there are occasional changes to the cache format, this information
is NOT stored in the cyrus.cache file. Instead, there is a
"cache_version" field in the cyrus.index record, so multiple different
versions of cache data may exist in the same cache file.</p>
<p>The order of fields per record in the cache file is as follows: (keep in
mind that they are all preceded by a 4 byte network byte order size).</p>
<dl>
<dt>Envelope Response</dt><dd><p>Raw IMAP response for a request for the envelope.</p>
</dd>
<dt>Bodystructure Response</dt><dd><p>Raw IMAP response for a request for the bodystructure.</p>
</dd>
<dt>Body Response</dt><dd><p>Raw IMAP response for an (old style) request for the body.</p>
</dd>
<dt>Binary Bodystructure</dt><dd><p>Offsets into the message file to pull out various body parts.
Because of the nature of MIME parts, this is somewhat recursive.</p>
<p>This looks like the following (starting the octet following the
cache field size). All of the fields are bit32s.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span>
<span class="p">[</span><span class="n">Number</span> <span class="n">of</span> <span class="n">message</span> <span class="n">parts</span><span class="o">+</span><span class="mi">1</span> <span class="k">for</span> <span class="n">the</span> <span class="n">rfc822</span> <span class="n">header</span> <span class="k">if</span> <span class="n">present</span><span class="p">]</span>
<span class="p">[</span>
<span class="p">[</span><span class="n">Offset</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">message</span> <span class="n">file</span> <span class="n">of</span> <span class="n">the</span> <span class="n">header</span> <span class="n">of</span> <span class="n">this</span> <span class="n">part</span><span class="p">]</span>
<span class="p">[</span><span class="n">Size</span> <span class="p">(</span><span class="n">octets</span><span class="p">)</span> <span class="n">of</span> <span class="n">the</span> <span class="n">header</span> <span class="n">of</span> <span class="n">this</span> <span class="n">part</span><span class="p">]</span>
<span class="p">[</span><span class="n">Offset</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">message</span> <span class="n">file</span> <span class="n">of</span> <span class="n">the</span> <span class="n">content</span> <span class="n">of</span> <span class="n">this</span> <span class="n">part</span><span class="p">]</span>
<span class="p">[</span><span class="n">Size</span> <span class="p">(</span><span class="n">octets</span><span class="p">)</span> <span class="n">of</span> <span class="n">the</span> <span class="n">content</span> <span class="n">of</span> <span class="n">this</span> <span class="n">part</span><span class="p">]</span>
<span class="p">[</span><span class="n">Encoding</span> <span class="n">Type</span> <span class="n">of</span> <span class="n">this</span> <span class="n">part</span><span class="p">]</span>
<span class="p">]</span>
<span class="p">(</span><span class="n">repeat</span> <span class="k">for</span> <span class="n">each</span> <span class="n">part</span> <span class="k">as</span> <span class="n">well</span> <span class="k">as</span> <span class="n">once</span> <span class="k">for</span> <span class="n">the</span> <span class="n">headers</span><span class="p">)</span>
<span class="p">[</span><span class="n">zero</span> <span class="o">*</span><span class="ow">or</span><span class="o">*</span> <span class="n">number</span> <span class="n">of</span> <span class="n">sub</span><span class="o">-</span><span class="n">parts</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">case</span> <span class="n">of</span> <span class="n">a</span> <span class="n">multipart</span><span class="o">.</span>
<span class="k">if</span> <span class="n">nonzero</span><span class="p">,</span> <span class="n">this</span> <span class="ow">is</span> <span class="n">a</span> <span class="n">recursion</span> <span class="n">into</span> <span class="n">the</span> <span class="n">top</span> <span class="n">structure</span><span class="p">]</span>
<span class="p">(</span><span class="n">repeat</span> <span class="k">for</span> <span class="n">each</span> <span class="n">part</span><span class="p">)</span>
<span class="p">]</span>
</pre></div>
</div>
<p>Note if this is not a message/rfc822, than the values for the sizes
of the part 0 are -1 (to indicate that it doesn't exist). Sub-parts
are not possible for a part 0, so they aren't included when finding
recursive entries.</p>
<p>The offset and size info for both the mime header and content part
are useful in order to do fast indexing on the appropriate parts of
the message file when a client does a FETCH request for
BODY[HEADER], or BODY[2.MIME].</p>
<p>Note that the top level RFC822 headers are a treated as a separate
part from their body text ("0" or "HEADER").</p>
<p>In the case of a multipart/alternative, the content size & offset
refers to the size of the entire mime part.</p>
<p>A very simple message (with a single text/plain part) would
therefore look like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">2</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">header</span><span class="p">][</span><span class="n">text</span><span class="o">/</span><span class="n">plain</span> <span class="n">body</span> <span class="n">part</span> <span class="n">info</span><span class="p">][</span><span class="mi">0</span><span class="p">]]</span>
</pre></div>
</div>
<p>A simple multipart/alternative message might look like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">3</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">header</span><span class="p">][</span><span class="n">text</span><span class="o">/</span><span class="n">plain</span> <span class="n">message</span> <span class="n">part</span> <span class="n">info</span><span class="p">]</span>
<span class="p">[</span><span class="n">second</span> <span class="n">message</span> <span class="n">part</span> <span class="n">info</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]]</span>
</pre></div>
</div>
<p>A message with an attachment that has two subparts:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">3</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">header</span> <span class="n">info</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">first</span> <span class="n">body</span> <span class="n">part</span> <span class="n">info</span><span class="p">][</span><span class="n">attachment</span> <span class="n">info</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span>
<span class="p">[</span><span class="mi">3</span><span class="p">][</span><span class="n">NIL</span> <span class="n">header</span> <span class="n">info</span><span class="p">][</span><span class="n">sub</span> <span class="n">part</span> <span class="mi">1</span> <span class="n">info</span><span class="p">][</span><span class="n">sub</span> <span class="n">part</span> <span class="mi">2</span> <span class="n">info</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]]]</span>
</pre></div>
</div>
<p>A message with an attached message/rfc822 message with the following
total structure:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">message</span><span class="o">/</span><span class="n">rfc822</span>
<span class="mi">0</span> <span class="n">headers</span><span class="p">;</span> <span class="n">content</span><span class="o">-</span><span class="nb">type</span><span class="p">:</span> <span class="n">multipart</span><span class="o">/</span><span class="n">mixed</span>
<span class="mi">1</span> <span class="n">text</span><span class="o">/</span><span class="n">plain</span>
<span class="mi">2</span> <span class="n">message</span><span class="o">/</span><span class="n">rfc822</span>
<span class="mi">0</span> <span class="n">headers</span><span class="p">;</span> <span class="n">content</span><span class="o">-</span><span class="nb">type</span><span class="p">:</span> <span class="n">multipart</span><span class="o">/</span><span class="n">alternative</span>
<span class="mi">1</span> <span class="n">text</span><span class="o">/</span><span class="n">plain</span>
<span class="mi">2</span> <span class="n">text</span><span class="o">/</span><span class="n">html</span>
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">3</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">header</span> <span class="n">part</span> <span class="mi">0</span><span class="p">][</span><span class="n">text</span><span class="o">/</span><span class="n">plain</span> <span class="n">part</span> <span class="mi">1</span><span class="p">][</span><span class="n">overall</span> <span class="n">attachment</span> <span class="n">info</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span>
<span class="p">[</span><span class="mi">3</span><span class="p">][</span><span class="n">rfc822</span> <span class="n">header</span> <span class="n">part</span> <span class="mf">2.0</span><span class="p">][</span><span class="n">text</span><span class="o">/</span><span class="n">plain</span> <span class="n">part</span> <span class="mf">2.1</span><span class="p">][</span><span class="n">text</span><span class="o">/</span><span class="n">html</span> <span class="n">part</span> <span class="mf">2.2</span><span class="p">]</span>
<span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]]]</span>
</pre></div>
</div>
</dd>
<dt>Cache Header</dt><dd><p>Any cached header fields. The exact set of fields here depends on
the cache record version - there is a function in <code class="docutils literal notranslate"><span class="pre">imap/mailbox.c</span></code>
to determine if a named header would be cached based on the version.
These are in the same format they would appear in the message file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HeaderName</span><span class="p">:</span> <span class="n">headerdata</span>\<span class="n">r</span>\<span class="n">n</span>
</pre></div>
</div>
<p>Examples include: References, In-Reply-To, etc.</p>
</dd>
<dt>From</dt><dd><p>The from header.</p>
</dd>
<dt>To</dt><dd><p>The to header.</p>
</dd>
<dt>Cc</dt><dd><p>The CC header.</p>
</dd>
<dt>Bcc</dt><dd><p>The BCC header.</p>
</dd>
<dt>Subject</dt><dd><p>The Subject header.</p>
</dd>
</dl>
<section id="id1">
<h3>Locking Considerations<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> file must be locked in exclusive mode while making
changes to the <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> file to ensure consistency. All new cache
records are created by reading the current end-of-file offset, appending
the new cache record, and storing that start offset into the associated
cyrus.index record.</p>
</section>
</section>
<section id="cyrus-index">
<h2><code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code><a class="headerlink" href="#cyrus-index" title="Permalink to this heading"></a></h2>
<p>The cyrus.index file is NOT just a cache - it stores information not
present in the message file!</p>
<p>The cyrus.index file consists of a fixed width header, followed by fixed
width records. In the past, it would be rewritten on every expunge, but
since Cyrus 2.4 the expunged records remain in the cyrus.index file for
a configurable time to support QRESYNC and more efficient delayed
expunge.</p>
<p>The cyrus.index file is the "heart" of the mailbox format - containing
checksums (CRC32) of everything else, and the most frequently updated
fields. All fields are stored in network byte order and aligned on 4
byte boundaries. Due to some 64 bit values being stored, the header and
individual records are aligned on 8 byte boundaries.</p>
<p>The overall format looks sort of like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cyrus</span><span class="o">.</span><span class="n">index</span><span class="p">:</span>
<span class="o">+----------------+</span>
<span class="o">|</span> <span class="n">Mailbox</span> <span class="n">Header</span> <span class="o">|</span>
<span class="o">+----------------+</span>
<span class="o">|</span> <span class="n">Msg</span><span class="p">:</span> <span class="n">Num</span> <span class="mi">1</span> <span class="o">|</span>
<span class="o">+----------------+</span>
<span class="o">|</span> <span class="n">Msg</span><span class="p">:</span> <span class="n">Num</span> <span class="mi">2</span> <span class="o">|</span>
<span class="o">+----------------+</span>
<span class="o">|</span> <span class="o">...</span> <span class="o">|</span>
<span class="o">+----------------+</span>
</pre></div>
</div>
<p>The basic idea being that there is one header, and then all the message
records are evenly spaced throughout the file. All of the message
records are at well-known offsets, making any part of the file
accessable at roughly equal speed.</p>
<section id="id2">
<h3>Locking Considerations<a class="headerlink" href="#id2" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> files can not be repacked (i.e. records can not change
UID for a particular offset, and the file can't be rewritten or deleted)
unless there's an exclusive namelock held for the mailbox name. This is
to avoid race conditions and simplify the use of mailboxes. Whenever a
mailbox is opened, the caller holds a shared namelock on the mailbox
name for the duration of the "mailbox object"'s existence.</p>
<p>All reads of a <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> file must be done with a lock held, and
all writes must be done with an exclusive lock held. This ensures CRC32
checksums of individual headers and records are always consistent. There
are no direct "offset" reads done any more, instead the mailbox API
provides a way to read an entire cyrus.index header or cyrus.index
record into a struct, performing consistency checks. Writes are also
done with a complete record struct.</p>
</section>
<section id="detail-of-cyrus-index-header">
<h3>Detail of <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> header<a class="headerlink" href="#detail-of-cyrus-index-header" title="Permalink to this heading"></a></h3>
<p>The index header contains the following information, in order:</p>
<dl>
<dt>Generation Number (4 bytes)</dt><dd><p>A number that is basically the "revision number" of the mailbox. It
must match between the cache and index files. This is to ensure that
if we fail to sync both the cache and index files and a crash
happens (so that only one is synced), we do not provide bad data to
the user. This is also backed by having individual cache checksums
on each record.</p>
</dd>
<dt>Format (4 bytes)</dt><dd><p>Basically obsolete (indicates netnews or regular).</p>
</dd>
<dt>Minor Version (4 bytes)</dt><dd><p>Indicates the version number of the index file. This can be used for
on-the-fly upgrades of the index and cache files.</p>
</dd>
<dt>Start Offset (4 bytes)</dt><dd><p>Size of index header.</p>
</dd>
<dt>Record Size (4 bytes)</dt><dd><p>Size of an index record.</p>
</dd>
<dt>Num Records (4 bytes)</dt><dd><p>How many records are in this index (including records for expunged
records. See below for "Exists" which has moved from pre-version 12
files.</p>
</dd>
<dt>Last Appenddate (4 bytes)</dt><dd><p>(time_t) of the last time a message was appended</p>
</dd>
<dt>Last UID (4 bytes)</dt><dd><p>Highest UID of all messages in the mailbox (UIDNEXT - 1).</p>
</dd>
<dt>Quota Mailbox Used (8 bytes)</dt><dd><p>Total amount of storage used by all of the messages in the mailbox.
Platforms that don't support 64-bit integers only use the last 4
bytes.</p>
</dd>
<dt>POP3 Last Login (4 bytes)</dt><dd><p>(time_t) of the last pop3 login to this INBOX, used to enforce the
"poptimeout" <code class="docutils literal notranslate"><span class="pre">imapd.conf</span></code> option.</p>
</dd>
<dt>UIDvalidity (4 bytes)</dt><dd><p>The UID validitiy of this mailbox. Cyrus currently uses the
<code class="docutils literal notranslate"><span class="pre">time()</span></code> when this mailbox was created.</p>
</dd>
<dt>Deleted, Answered, and Flagged (4 bytes each)</dt><dd><p>Counts of how many messages have each flag.</p>
</dd>
<dt>Mailbox Options (4 bytes)</dt><dd><p>Bitmask of mailbox options, consisting of any combination of the
following:</p>
<dl class="simple">
<dt>POP3_NEW_UIDL</dt><dd><p>Flag signalling that we're using "<em>uidvalidity</em>.<em>uid</em>" instead
of just "<em>uid</em>" for the output of the POP3 UIDL command.</p>
</dd>
<dt>IMAP_SHAREDSEEN</dt><dd><p>Flag signalling that we're supporting a shared \Seen flag on
the mailbox.</p>
</dd>
<dt>IMAP_DUPDELIVER</dt><dd><p>Flag signalling that we're allowing duplicate delivery of
messages to the mailbox, overriding system-wide duplicate
suppression.</p>
</dd>
<dt>MAILBOX_NEEDS_REPACK</dt><dd><p>Flag signalling that the mailbox is due to be repacked. During
mailbox_close() every process will attempt to take an exclusive
namelock on the mailbox and repack.</p>
</dd>
<dt>MAILBOX_DELETED</dt><dd><p>Flag signalling that the mailbox is deleted. This can be set
with a shared namelock, and indicates to all other users of the
mailbox that they need to close it and attempt cleanup. The last
process to close the mailbox will perform the final cleanup
under an exclusive namelock, giving the other processes a chance
to finish their current operation first without files
disappearing from under them!</p>
</dd>
</dl>
</dd>
<dt>Leaked Cache (4 bytes)</dt><dd><p>Number of leaked records in the cache file.</p>
</dd>
<dt>Highest ModSeq (8 bytes)</dt><dd><p>Highest Modification Sequence of all the messages in the mailbox
(CONDSTORE).</p>
</dd>
<dt>Deleted ModSeq (8 bytes)</dt><dd><p>Lowest Modification Sequence before which expunged message data may
have been purged from the mailbox and forgotten (CONDSTORE/QRESYNC
support).</p>
</dd>
<dt>Exists (4 bytes)</dt><dd><p>See NumRecords above. This is the count of non-expunged records in
the mailbox and corresponds to the IMAP status item "EXISTS".</p>
</dd>
<dt>First Expunged (4 bytes)</dt><dd><p>lowest modified time of an expunged message in this mailbox (or zero
if there are no expunged messages) - used to determine if the
mailbox needs repacking.</p>
</dd>
<dt>Last Repack Time (4 bytes)</dt><dd><p>a timestamp for the last repack, to ensure repacks aren't done too
close together if expunges were closely spaced</p>
</dd>
<dt>Header File CRC (4 bytes)</dt><dd><p>CRC32 value of the bytes in the <code class="docutils literal notranslate"><span class="pre">cyrus.header</span></code> file for this
mailbox. Must be rewritten whenever the cyrus.header file is changed
(see locking considerations above - this is why the cyrus.index must
be exclusively locked!)</p>
</dd>
<dt>Sync CRC (4 bytes)</dt><dd><p>An XOR of the CRC32 of a specially generated value for each of the
non-expunged records in this mailbox. This is a cached value which
allows the replication subsystem to quickly determine that all
non-expunged records in a mailbox are in sync and detect possible
"split brain" scenarios with low bandwidth use.</p>
</dd>
<dt>Recent UID (4 bytes)</dt><dd><p>The highest UID last time an IMAP client logged in as the mailbox
owner (or anybody if SHAREDSEEN is enabled) selected this mailbox.
Used to generate the \Recent flags in IMAP</p>
</dd>
<dt>Recent Time (4 bytes)</dt><dd><p>Used for consistency with the seen_db code, but probably not
actually necessary. Oh well</p>
</dd>
<dt>Header CRC (4 bytes)</dt><dd><p>Must always be the LAST record of the header. This is the CRC32 of
the actual bytes on disk (network order format) for the rest of the
cyrus.index. By keeping it last, it can be easily calculated with
the following snippet of code:
<code class="docutils literal notranslate"><span class="pre">crc</span> <span class="pre">=</span> <span class="pre">crc32_map(buf,</span> <span class="pre">OFFSET_HEADER_CRC);</span></code> - i.e. crc32 from the
start of the buffer to just before this field.</p>
</dd>
</dl>
<p>There are also spare fields in the index header, to allow for future
expansion without forcing an upgrade of the file, and to round up to be
divisible by 8 bytes.</p>
</section>
<section id="detail-of-cyrus-index-records">
<h3>Detail of <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> records<a class="headerlink" href="#detail-of-cyrus-index-records" title="Permalink to this heading"></a></h3>
<p>These records start immediately following the <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> header,
and are all fixed size. They are in-order by uid of the message.</p>
<dl class="simple">
<dt>UID (4 bytes)</dt><dd><p>UID of the message</p>
</dd>
<dt>INTERNALDATE (4 bytes)</dt><dd><p>INTERNALDATE of the message (where possible, this matches the
creation and modification times of the file on disk to help
reconstruct in the event of data loss)</p>
</dd>
<dt>SENTDATE (4 bytes)</dt><dd><p>Contents of the Date: header chomped to day resolution with timezone
stripped.</p>
</dd>
<dt>SIZE (4 bytes)</dt><dd><p>Size of the whole message (in octets)</p>
</dd>
<dt>HEADER SIZE (4 bytes)</dt><dd><p>Size of the message header (in octets)</p>
</dd>
<dt>GMTIME (4 bytes)</dt><dd><p>Contents of the Date: header at 1 second resolution and converted to
GMT (for sort)</p>
</dd>
<dt>CACHE_OFFSET (4 bytes)</dt><dd><p>Offset into the <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> file for the beginning of this
message's cache entry.</p>
</dd>
<dt>LAST UPDATED (4 bytes)</dt><dd><p>(time_t) of the last time this record was changed</p>
</dd>
<dt>SYSTEM FLAGS (4 bytes)</dt><dd><p>Bitmask showing which system flags are set/unset</p>
</dd>
<dt>USER FLAGS (MAX_USER_FLAGS / 32 bytes)</dt><dd><p>Bitmask showing which user flags are set/unset (bits correspond to
positions in the cyrus.header flag list, i.e. (1<<0) == the flag
name</p>
</dd>
<dt>CONTENT_LINES (4 bytes)</dt><dd><p>Number of text lines contained in the message content (body).</p>
</dd>
<dt>CACHE_VERSION (4 bytes)</dt><dd><p>Indicates the version number of the cache record for the message
(determines which headers are cached, see list in mailbox.c).</p>
</dd>
<dt>GUID (MESSAGE_GUID_SIZE bytes)</dt><dd><p>Globally Unique IDentifier of the message (used by replication
engine). This is the sha1 value of the bytes as stored on disk.</p>
</dd>
<dt>MODSEQ (8 bytes)</dt><dd><p>Modification Sequence of the message (CONDSTORE).</p>
</dd>
<dt>CACHE_CRC (4 bytes)</dt><dd><p>This is the CRC32 of all the bytes of the cache record (all 10
fields) as stored on disk. Again, calculated over the exact bytes
stored in the <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> file.</p>
</dd>
<dt>RECORD_CRC (4 bytes)</dt><dd><p>Like the header CRC - this is the CRC32 of all the bytes in on-disk
order that exist in this record. Records are always rewritten as the
entire record, including the updated CRC, so it's always consistent
if you have a lock on the <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> file, because writers will
wait until they get an exclusive lock to make modifications.</p>
</dd>
</dl>
</section>
</section>
<section id="notes">
<h2>Notes<a class="headerlink" href="#notes" title="Permalink to this heading"></a></h2>
<ul>
<li><p>Expunge is super quick now - it's just a flag update!</p></li>
<li><p>Append is relatively fast (it only adds to the end of both the cache
and index files and modifies the index header)</p></li>
<li><p>Message unlinks always happen during the "close" phase - which may be
noticed when you select another mailbox, but otherwise are delayed
from the actual action. With delayed expunge, the unlinks are pushed
off to cyr_expire which is a background task, and will never be
noticed by the user.</p></li>
<li><p>Message delivery is something like this:</p>
<ol class="arabic simple">
<li><p>write/sync message file</p></li>
<li><p>write/sync new <code class="docutils literal notranslate"><span class="pre">cyrus.cache</span></code> record</p></li>
<li><p>write/sync new <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> record</p></li>
<li><p>calculate, write, sync new <code class="docutils literal notranslate"><span class="pre">cyrus.index</span></code> header</p></li>
<li><p>acknowledge message delivery</p></li>
</ol>
<p>The message isn't delivered until the new index header is written. In
case of a crash before the new index header is written, any previous
writes will be overwritten on the next delivery (and will not be
noticed by the readers).</p>
<p>Note that certain power failure situations (power failure in the
middle of a disk sector write) could cause a mailbox to need
reconstruction (possibly even losing some flag state). These failure
modes are not possible in the "Hardware RAID disk model" (which we
will describe somewhere else when we get around to it).</p>
</li>
</ul>
</section>
<section id="future-considerations">
<h2>Future considerations<a class="headerlink" href="#future-considerations" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p>Cache all header fields? (or all up to Xk?) This could greatly
improve speeds of clients that just ask for everything, but also
increases the expense of rewriting the cache file (as well as the
size it takes on disk).</p></li>
<li><p>Reformat cache file to use a
(size)(size)(size)(size)(data)(data)(data) format. This makes
accesses anywhere in the cache file equally fast, as opposed to
having to iterate through all the entires for a given message to get
to the last one. Note that either way is still O(1) so maybe it
doesn't matter much.</p></li>
<li><p>It would be useful to store a uniqueid -> mailbox name index, so that
we could fix arbitron again.</p></li>
</ul>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="locking.html" class="btn btn-neutral float-left" title="Cyrus IMAP Server: Locking" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="namelocks.html" class="btn btn-neutral float-right" title="Cyrus IMAP Server: Namelocks" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 1993–2025, The Cyrus Team.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
