1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
|
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Cyrus Murder: Concepts — Cyrus IMAP 3.12.1 documentation</title>
<link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/graphviz.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/cyrus.css" type="text/css" />
<script data-url_root="../../../../" id="documentation_options" src="../../../../_static/documentation_options.js"></script>
<script src="../../../../_static/jquery.js"></script>
<script src="../../../../_static/underscore.js"></script>
<script src="../../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../../../_static/doctools.js"></script>
<script src="../../../../_static/sphinx_highlight.js"></script>
<script src="../../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../../genindex.html" />
<link rel="search" title="Search" href="../../../../search.html" />
<link rel="next" title="Cyrus Murder: Installation and Administration" href="murder-installation.html" />
<link rel="prev" title="Cyrus Murder" href="murder.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../../index.html" class="icon icon-home">
Cyrus IMAP
</a>
<div class="version">
3.12.1
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Cyrus IMAP</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../../download.html">Download</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../quickstart.html">Quickstart Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../setup.html">Setup</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../../../operations.html">Operations</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../manpages/index.html">Man pages</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../../admin.html">Administrator Guide</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../../admin.html#architecture">Architecture</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="../../admin.html#management">Management</a><ul class="current">
<li class="toctree-l4"><a class="reference internal" href="../locations.html">File & Directory Locations</a></li>
<li class="toctree-l4"><a class="reference internal" href="../ports-sockets.html">Ports and Sockets</a></li>
<li class="toctree-l4"><a class="reference internal" href="../access-control.html">Access Control</a></li>
<li class="toctree-l4"><a class="reference internal" href="../quotas.html">Quotas</a></li>
<li class="toctree-l4"><a class="reference internal" href="../sieve.html">Cyrus Sieve</a></li>
<li class="toctree-l4"><a class="reference internal" href="../nntp.html">Cyrus NNTP</a></li>
<li class="toctree-l4"><a class="reference internal" href="../protlayer.html">Cyrus Prot Layer</a></li>
<li class="toctree-l4"><a class="reference internal" href="../sop.html">Standard Operating Procedures</a></li>
<li class="toctree-l4"><a class="reference internal" href="../eventsource.html">Cyrus Event Source</a></li>
<li class="toctree-l4"><a class="reference internal" href="../monitoring.html">Monitoring</a></li>
<li class="toctree-l4"><a class="reference internal" href="../config-mailboxdistribution.html">Mailbox Distribution</a></li>
<li class="toctree-l4 current"><a class="reference internal" href="murder.html">Cyrus Murder</a></li>
<li class="toctree-l4"><a class="reference internal" href="../tweaking.html">Tweaking Cyrus IMAP</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../faq.html">Frequently Asked Questions</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../../developers.html">Developers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../support.html">Support/Community</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Cyrus SASL</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="http://www.cyrusimap.org/sasl">Cyrus SASL</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../../index.html">Cyrus IMAP</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../../../operations.html">Operations</a></li>
<li class="breadcrumb-item"><a href="../../admin.html">Administrator Guide</a></li>
<li class="breadcrumb-item"><a href="murder.html">Cyrus Murder</a></li>
<li class="breadcrumb-item active">Cyrus Murder: Concepts</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/reference/admin/murder/murder-concepts.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="cyrus-murder-concepts">
<span id="murder-concepts"></span><h1>Cyrus Murder: Concepts<a class="headerlink" href="#cyrus-murder-concepts" title="Permalink to this heading"></a></h1>
<p>The Cyrus IMAP Aggregator transparently distributes IMAP and POP
mailboxes across multiple servers. Unlike other systems for load
balancing IMAP mailboxes, the aggregator allows users to access
mailboxes on any of the IMAP servers in the system.</p>
<p>Note that although this document focuses on IMAP and POP, the same
concepts also apply to NNTP and HTTP (CalDAV, CardDAV, RSS).</p>
<section id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Permalink to this heading"></a></h2>
<p>Scaling a service usually takes one of two paths:</p>
<ol class="arabic simple">
<li><dl class="simple">
<dt>Buy bigger and faster machines</dt><dd><p>This approach is obvious and (hopefully) easy, though at some point
software tuning becomes necessary to take advantage of the bigger
machines. However, if one of these large machines go down, then your
entire system is unavailable.</p>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt>Distribute the load across multiple machines.</dt><dd><p>The second approach has the benefit that there is no longer a single
point of failure and the aggregate cost of multiple machines may be
significantly lower than the cost of a single large machine. However,
the system may be harder to implement as well as harder to manage.</p>
</dd>
</dl>
</li>
</ol>
<p>In the IMAP space, the approach of buying a larger machine is pretty
obvious. Distributing the load is a bit trickier since there is no
concept of mailbox location in IMAP (excluding <span class="target" id="index-0"></span><a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2193.html"><strong>RFC 2193</strong></a> mailbox
referrals, which are not widely implemented by clients). Clients
traditionally assume that the server they are talking to is the server
with the mailbox they are looking for.</p>
<p>The approaches to distributing the load among IMAP servers generally
sacrifice the unified system image. For pure email, this is an
acceptable compromise; however, trying to share mailboxes becomes
difficult or even impossible. Specific examples can be found in <a class="reference internal" href="#appendix-a-dns-name-load-balancing">Appendix
A: DNS Name Load Balancing</a> and <a class="reference internal" href="#id2">Appendix B: IMAP Multiplexing</a>).</p>
<p>We propose a new approach to overcome these problems. We call it the Cyrus
IMAP Aggregator. The Cyrus aggregator takes a <a class="reference internal" href="#def-murder"><span class="std std-ref">murder</span></a> of IMAP
servers and presents a server independent view to the clients. That is,
<strong>all the mailboxes across all the IMAP servers are aggregated to a single
image</strong>, thereby appearing to be only one IMAP server to the clients.</p>
</section>
<section id="architecture">
<h2>Architecture<a class="headerlink" href="#architecture" title="Permalink to this heading"></a></h2>
<dl class="simple">
<dt>The Cyrus IMAP Aggregator has three classes of servers:</dt><dd><ol class="arabic simple">
<li><p>IMAP frontend,</p></li>
<li><p>IMAP backend, and</p></li>
<li><p>MUPDATE.</p></li>
</ol>
</dd>
</dl>
<p>The frontend servers act as the primary communication point between the
end user clients and the backend servers. The frontends use the MUPDATE
server as an authoritative source for mailbox names, locations, and
permissions. The backend servers store the actual IMAP data (and keep
the MUPDATE server appraised as to changes in the Mailbox list).</p>
<section id="backend-servers">
<h3>Backend Servers<a class="headerlink" href="#backend-servers" title="Permalink to this heading"></a></h3>
<p>The backend servers serve the actual data and are fully functional
standalone IMAP servers that serve a set of mailboxes. Each backend
server maintains a local mailboxes database that lists what mailboxes
are available on that server.</p>
<p>The imapd processes on a backend server can stand by themselves, so that
each backend IMAP server can be used in isolation, without a MUPDATE
server or any frontend servers. However, they are configured so that
they won't process any mailbox operations (CREATE, DELETE, RENAME,
SETACL, etc) unless the master MUPDATE server can be contacted and
authorizes the transaction.</p>
<p>In this mode, the imapd processes update the local mailboxes database
themselves. Additionally, on a CREATE they need to reserve a place with
the MUPDATE server to insure that other backend servers aren't creating
the same mailbox before proceeding. Once the local aspects of mailbox
creation are complete, the mailbox is activated on the MUPDATE server,
and is considered available to any client through the frontends.</p>
</section>
<section id="frontend-servers">
<h3>Frontend Servers<a class="headerlink" href="#frontend-servers" title="Permalink to this heading"></a></h3>
<p>The frontend servers, unlike the backend servers, are fully
interchangeable with each other and the frontend servers can be
considered 'dataless'; any loss of a proxy results in no loss of data.
The only persistent data that is needed (the mailbox list) is kept on
the MUPDATE master server. This list is synchronized between the
frontend and the MUPDATE master when the frontend comes up.</p>
<p>The list of mailboxes in the murder is maintained by the MUPDATE server.
The MUPDATE protocol is described at <span class="target" id="index-1"></span><a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc3656.html"><strong>RFC 3656</strong></a>.</p>
<p>For IMAP service on a frontend, there are two main types of processes,
the proxyd and the mupdate (slave mode) synchronization process. The
proxyd handles the IMAP session with clients. It relies on a consistent
and complete mailboxes database that reflects the state of the world. It
never writes to the mailboxes database. Instead, the mailboxes database
is kept in sync with the master by a slave mupdate process.</p>
</section>
<section id="mail-delivery">
<h3>Mail Delivery<a class="headerlink" href="#mail-delivery" title="Permalink to this heading"></a></h3>
<p>The incoming mail messages go to an lmtp proxy (either running on a
frontend, a mail exchanger, or any other server). The lmtp proxy running
on the frontend server uses the master MUPDATE server to determine the
location of the destination folder and then transfers the message to the
appropriate backend server via LMTP. If the backend is not up (or
otherwise fails to accept the message), then the LMTP proxy returns a
failure to the connected MTA.</p>
<p>If a sieve script is present, the lmtp proxy server must do the
processing as the end result of the processing may result in the mail
message going to a different backend server than where the user's INBOX
is.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The current implementation runs SIEVE on the backend servers, and
holds the requirement that all of a user's mailboxes live on the
same backend.</p>
</div>
</section>
<section id="clients">
<h3>Clients<a class="headerlink" href="#clients" title="Permalink to this heading"></a></h3>
<p>Clients that support <span class="target" id="index-2"></span><a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2193.html"><strong>RFC 2193</strong></a> IMAP referrals can bypass the
aggregator frontend. See <a class="reference internal" href="#id1">IMAP Referrals</a> for more details.</p>
<p>Clients are encouraged to bypass the frontends via approved mechanisms.
This should result in better performance for the client and less load
for the servers.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Sites choosing to locate front ends in a DMZ, or other isolated
network segment, should disable IMAP Referrals which might lead
clients to attempt impossible actions, such as direct access to
back ends which are firewalled. Please consult
<a class="reference internal" href="../../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a> for details of the
<code class="docutils literal notranslate"><span class="pre">proxyd_disable_mailbox_referrals</span></code> setting.</p>
</div>
</section>
</section>
<section id="implementation">
<h2>Implementation<a class="headerlink" href="#implementation" title="Permalink to this heading"></a></h2>
<section id="assumptions">
<h3>Assumptions<a class="headerlink" href="#assumptions" title="Permalink to this heading"></a></h3>
<ul>
<li><p>Operations that change the mailbox list are (comparatively) rare.
The vast majority of IMAP sessions do not manipulate the state of the
mailbox list.</p></li>
<li><p>Read operations on the mailbox list are very frequent.</p></li>
<li><p>A mailbox name must be unique among all the back end servers.</p></li>
<li><p>The MUPDATE master server will be able to handle the load from the
frontend, backend, and LMTP proxy servers. Currently, the MUPDATE
master can be a bottleneck in the throughput of mailbox operations,
but as the MUPDATE protocol allows for slave server to act as
replicas, it is theoretically possible to reduce the load of read
operations against the master to a very low level.</p></li>
<li><p>IMAP clients are not sensitive to somewhat loose mailbox tree
consistency, and some amount of consistency can be sacrificed for
speed. As is, IMAP gives no guarantees about the state of the mailbox
tree from one command to the next. However, it's important to note
that different IMAP sessions do communicate out of band: two sessions
for the same client should see sensible results. In the Murder case,
this means that the same client talking to two different frontends
should see sensible results.</p></li>
<li><p>A single IMAP connection should see consistent results: once an
operation is done, it is done, and needs to be reflected in the
current session. The straightforward case that must work correctly is
(provided there is no interleaved DELETE in another session):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">A001</span> <span class="n">CREATE</span> <span class="n">INBOX</span><span class="o">.</span><span class="n">new</span>
<span class="n">A002</span> <span class="n">SELECT</span> <span class="n">INBOX</span><span class="o">.</span><span class="n">new</span>
</pre></div>
</div>
</li>
<li><p>Accesses to non-existent mailboxes are rare.</p></li>
</ul>
</section>
<section id="authentication">
<h3>Authentication<a class="headerlink" href="#authentication" title="Permalink to this heading"></a></h3>
<p>The user authenticates to the frontend server via any supported SASL
mechanism or via plaintext. If authentication is successful, the front
end server will authenticate to the backend server using a SASL
mechanism (in our case GSSAPI) as a privileged user. This
user is able to switch to the authorization of the actual user being
proxied for and any authorization checks happen as if the user actually
authenticated directly to the backend server. Note this is a native
feature of many SASL mechanisms and nothing special with the aggregator.</p>
<p>To help protect the backends from a compromised frontends, all
administrative actions (creating users, top level mailboxes, quota
changes, etc) must be done directly from the client to the backend, as
administrative permissions are not granted to any of the proxy servers.
IMAP Referrals provide a way to accomplish this with minimal client UI
changes.</p>
</section>
<section id="subscriptions">
<h3>Subscriptions<a class="headerlink" href="#subscriptions" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">[LSUB,</span> <span class="pre">SUBSCRIBE,</span> <span class="pre">UNSUBSCRIBE]</span></code></p>
<p>The frontend server directs the LSUB to the backend server that has
the user's INBOX. As such, the backend server may have entries in the
subscription database that do not exist on that server. The frontend
server needs to process the list returned by the backend server and
either remove or tag with \NoSelect the entries which are not currently
active within the murder.</p>
<p>If the user's INBOX server is down and the LSUB fails, then the
aggregator replies with NO with an appropriate error message. Clients
should not assume that the user has no subscriptions (though apparently
some clients do this).</p>
</section>
<section id="finding-a-mailbox">
<h3>Finding a Mailbox<a class="headerlink" href="#finding-a-mailbox" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">[SETQUOTA,</span> <span class="pre">GETQUOTA,</span> <span class="pre">EXAMINE,</span> <span class="pre">STATUS]</span></code></p>
<p>The frontend machine looks up the location of the mailbox, connects
via IMAP to the backend server, and issues the equivalent command there.
A quota root is not allowed to span across multiple servers.
At least, not with the semantics that it will be inclusive across the murder.</p>
<p><code class="docutils literal notranslate"><span class="pre">[SELECT]</span></code></p>
<blockquote>
<div><p>To SELECT a mailbox:</p>
<ol class="arabic simple">
<li><p>proxyd: lookup foo.bar in local mailboxes database</p></li>
<li><p>if yes, proxyd -> back end: send SELECT</p></li>
<li><p>if no, proxyd -> mupdate slave -> mupdate master: send a ping
along the UPDATE channel in order to ensure that we have received
the latest data from the MUPDATE master.</p></li>
<li><p>if mailbox still doesn't exist, fail operation</p></li>
<li><p>if mailbox does exist, and the client supports referrals, refer
the client. Otherwise continue as a proxy with a selected mailbox.</p></li>
</ol>
<p>SELECT on mailboxes that do not exist are much more expensive but
the assumption is that this does not frequently occur (or if it
does, it is just after the mailbox has been created and the
frontend hasn't seen the update yet).</p>
</div></blockquote>
</section>
<section id="operations-within-a-mailbox">
<h3>Operations within a Mailbox<a class="headerlink" href="#operations-within-a-mailbox" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">[APPEND,</span> <span class="pre">CHECK,</span> <span class="pre">CLOSE,</span> <span class="pre">EXPUNGE,</span> <span class="pre">SEARCH,</span> <span class="pre">FETCH,</span> <span class="pre">STORE,</span> <span class="pre">UID]</span></code></p>
<p>These commands are sent to the appropriate backend server.
The aggregator does not need to modify any of these commands
before sending them to the backend server.</p>
</section>
<section id="copy">
<h3>COPY<a class="headerlink" href="#copy" title="Permalink to this heading"></a></h3>
<p>COPY is somewhat special as it acts upon messages in the currently
SELECT'd mailbox but then interacts with another mailbox.</p>
<p>In the case where the destination mailbox is on the same backend server
as the source folder, the COPY command is issued to the backend
server and the backend server takes care of the command.</p>
<p>If the destination folder is on a different backend server, the
frontend intervenes and does the COPY by FETCHing the messages from the
source backend server and then APPENDs the messages to the destination server.</p>
</section>
<section id="operations-on-the-mailbox-list">
<h3>Operations on the Mailbox List<a class="headerlink" href="#operations-on-the-mailbox-list" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">[CREATE,</span> <span class="pre">DELETE,</span> <span class="pre">RENAME,</span> <span class="pre">SETACL]</span></code></p>
<blockquote>
<div><p>These commands are all done by the backend server using the MUPDATE
server as a lock manager. Changes are then propagated to the frontend
via the MUPDATE protocol.</p>
</div></blockquote>
<p><code class="docutils literal notranslate"><span class="pre">[LIST]</span></code></p>
<blockquote>
<div><p>LIST is handled by the frontend servers; no interaction is
required with the backend server as the front ends have a local
database that is never more than a few seconds out of date.</p>
</div></blockquote>
<p><code class="docutils literal notranslate"><span class="pre">[CREATE]</span></code></p>
<blockquote>
<div><p>CREATE creates the mailbox on the same backend server as the
parent mailbox. If the parent exists but exists on multiple backend
servers, if there is no parent folder, a tagged NO response is
returned.</p>
<p>When this happens, the administrator has two choices. He may
connect directly to a backend server and issue the CREATE on that
server. Alternatively, a second argument can be given to CREATE
after the mailbox name. This argument specifies the specific host
name on which the mailbox is to be created.</p>
<p>The following operations occur for CREATE on the frontend:</p>
<ul class="simple">
<li><p>proxyd: verify that mailbox doesn't exist in MUPDATE mailbox list.</p></li>
<li><p>proxyd: decide where to send CREATE (the server of the parent
mailbox, as top level mailboxes cannot be created by the proxies).</p></li>
<li><p>proxyd -> back end: duplicate CREATE command and verifies that
the CREATE does not create an inconsistency in the mailbox list
(i.e. the folder name is still unique).</p></li>
</ul>
<p>The following operations occur for CREATE on the backend:</p>
<ul class="simple">
<li><p>imapd: verify ACLs to best of ability (CRASH: aborted)</p></li>
<li><p>imapd: start mailboxes transaction (CRASH: aborted)</p></li>
<li><p>imapd may have to open an MUPDATE connection here if one doesn't
already exist</p></li>
<li><p>imapd -> MUPDATE: set foo.bar reserved (CRASH: MUPDATE externally
inconsistent)</p></li>
<li><p>imapd: create foo.bar in spool disk (CRASH: MUPDATE externally
inconsistent, back end externally inconsistent, this can be
resolved when the backend comes back up by clearing the state from
both MUPDATE and the backend)</p></li>
<li><p>imapd: add foo.bar to mailboxes dataset (CRASH: ditto)</p></li>
<li><p>imapd: commit transaction (CRASH: ditto, but the recovery can
activate the mailbox in mupdate instead)</p></li>
<li><p>imapd -> MUPDATE: set foo.bar active (CRASH: committed)</p></li>
</ul>
<p>Failure modes: Above, all backend inconsistencies result in the
next CREATE attempt failing. The earlier MUPDATE inconsistency
results in any attempts to CREATE the mailbox on another backend
failing. The latter one makes the mailbox unreachable and
un-createable. Though, this is safer than potentially having the
mailbox appear in two places when the failed backend comes back
up.</p>
</div></blockquote>
<p><code class="docutils literal notranslate"><span class="pre">[RENAME]</span></code></p>
<blockquote>
<div><p>RENAME is only interesting in the cross-server case. In this case
it issues a (non-standard) XFER command to the backend that
currently hosts the mailbox, which performs a binary transfer of
the mailbox (and in the case of a user's inbox, their associated
seen state and subscription list) to the new backend. During this
time the mailbox is marked as RESERVED in mupdate, and when it is
complete it is activated on the new server in MUPDATE. The
deactivation prevents clients from accessing the mailbox, and
causes mail delivery to temporarily fail.</p>
</div></blockquote>
</section>
<section id="imap-referrals">
<span id="id1"></span><h3>IMAP Referrals<a class="headerlink" href="#imap-referrals" title="Permalink to this heading"></a></h3>
<p>If clients support IMAP Mailbox Referrals (<span class="target" id="index-3"></span><a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2193.html"><strong>RFC 2193</strong></a>), the client can
improve performance and reduce the load on the aggregator by using the
IMAP referrals that are sent to it and going to the appropriate
backend servers.</p>
<p>The frontend servers will advertise the <code class="docutils literal notranslate"><span class="pre">MAILBOX-REFERRALS</span></code>
capability. The backend servers will also advertise this capability
(but only because they need to refer clients while a mailbox is moving
between servers).</p>
<p>Since there is no way for the server to know if a client supports
referrals, the Cyrus IMAP Aggregator will assume the clients do not
support referrals unless the client issues a RLSUB or a RLIST command.</p>
<p>Once a client issues one of those commands, then the aggregator will
issue referrals for any command for which the client may safely contact
the IMAP server directly. Most commands performing operations within
a mailbox (cf Section 3.3) fall into this category. Some commands will
not be possible without a referrals-capable client (such as most
commands done as administrator).</p>
<p><span class="target" id="index-4"></span><a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2193.html"><strong>RFC 2193</strong></a> indicates that the client does not stick the referred
server. As such the SELECT will get issued to the frontend server and
not the referred server. Additionally, CREATE, RENAME, and DELETE get
sent to the frontend which will proxy the command to the correct
backend server.</p>
</section>
<section id="pop">
<h3>POP<a class="headerlink" href="#pop" title="Permalink to this heading"></a></h3>
<p>POP is easy given that POP only allows access to the user's INBOX. When
it comes to POP, the IMAP Aggregator acts just like a <a class="reference internal" href="#appendix-b-imap-multiplexing"><span class="std std-ref">multiplexor</span></a>. The user authenticates to frontend
server. The frontend determines where the user's INBOX is located and
does a direct pass through of the POP commands from the client to the
appropriate backend server.</p>
</section>
<section id="mupdate">
<h3>MUPDATE<a class="headerlink" href="#mupdate" title="Permalink to this heading"></a></h3>
<p>The mupdate (slave) process (one per frontend) holds open a MUPDATE
connection and listens for updates from the MUPDATE master server (as
backends inform it of updates). The slave makes these modifications on
the local copy of the mailboxes database.</p>
</section>
</section>
<section id="analysis">
<h2>Analysis<a class="headerlink" href="#analysis" title="Permalink to this heading"></a></h2>
<section id="mailboxes-database">
<h3>Mailboxes Database<a class="headerlink" href="#mailboxes-database" title="Permalink to this heading"></a></h3>
<p>A benefit of having the mailbox information on the frontend is that
LIST is very cheap. The frontend servers can process this request
without having to contact each backend server.</p>
<p>We're also assuming that LIST is a much more frequent operation than any
of the mailbox operations and thus should be the case to optimize. (In
addition to the fact that any operation that needs to be forwarded to a
backend needs to know to which backend it is being forwarded, so lookups
in the mailbox list are also quite frequent).</p>
</section>
<section id="failure-mode-analysis">
<h3>Failure Mode Analysis<a class="headerlink" href="#failure-mode-analysis" title="Permalink to this heading"></a></h3>
<section id="what-happens-when-a-backend-server-comes-up">
<h4>What happens when a backend server comes up?<a class="headerlink" href="#what-happens-when-a-backend-server-comes-up" title="Permalink to this heading"></a></h4>
<blockquote>
<div><p>Resynchronization with the MUPDATE server. Any mailboxes that exist
locally but are not in MUPDATE are pushed to MUPDATE. Any mailboxes
that exist locally but are in MUPDATE as living on a different
server are deleted. Any mailboxes that do not exist locally but
exist in MUPDATE as living on this server are removed from MUPDATE.</p>
</div></blockquote>
</section>
<section id="what-happens-when-a-frontend-server-comes-up">
<h4>What happens when a frontend server comes up?<a class="headerlink" href="#what-happens-when-a-frontend-server-comes-up" title="Permalink to this heading"></a></h4>
<blockquote>
<div><p>The only thing that needs to happen is for the frontend to connect
to the MUPDATE server, issue an UPDATE command, and resynchronize
its local database copy with the copy on the master server.</p>
</div></blockquote>
</section>
<section id="where-s-the-true-mailboxes-file">
<h4>Where's the true mailboxes file?<a class="headerlink" href="#where-s-the-true-mailboxes-file" title="Permalink to this heading"></a></h4>
<blockquote>
<div><p>The MUPDATE master contains authoritative information as to the
location of any mailbox (in the case of a conflict), but the
backends are authoritative as to which mailboxes actually exist.</p>
</div></blockquote>
</section>
</section>
<section id="summary-of-benefits">
<h3>Summary of Benefits<a class="headerlink" href="#summary-of-benefits" title="Permalink to this heading"></a></h3>
<ul>
<li><p><strong>Availability</strong> - By allowing multiple frontends, failures of the
frontend only result in a reduction of capacity. Users currently
connected still lose their session but can just reconnect to get back
online.</p>
<blockquote>
<div><ul class="simple">
<li><p>The failure of the backends will result in the loss of
availability. However, given that the data is distributed among
multiple servers, the failure of a single server does not result
the entire system being down. Our experience with AFS was that
this type of partitioned failure was acceptable (if not ideal).</p></li>
<li><p>The failure of the mupdate master will cause write operations to
the mailbox list to fail, but accesses to mailboxes themselves (as
well as read operations to the mailbox list) will continue
uninterrupted.</p></li>
<li><p>At this point, there may be some ideas but no plans for providing
a high availability solution which would allow for backend
servers or the MUPDATE server to fail with no availability impact.</p></li>
</ul>
</div></blockquote>
</li>
<li><p><strong>Load scalability</strong> - No specific benchmarks have been run to
show that this system actually performs better. However, it is clear
that it scales to a larger number of users than a single server
architecture would. Though, based on the fact that there are no further
performance problems beyond when running a single machine,
but handling about 20% more concurrent users, this is a success.</p></li>
<li><p><strong>Management benefits</strong> - As with AFS, administrators have the
flexibility of placement of data on the servers, "live" move of data
between servers,</p></li>
<li><p><strong>User benefits</strong> - The user only needs to know a single server name
for configuration. The same name can be handed out to all users.</p>
<blockquote>
<div><ul class="simple">
<li><p>Users don't lose the ability to share their folders and those
folders are visible to other users. A user's INBOX folder
hierarchy can also exist across multiple machines.</p></li>
</ul>
</div></blockquote>
</li>
</ul>
</section>
</section>
<section id="futures">
<h2>Futures<a class="headerlink" href="#futures" title="Permalink to this heading"></a></h2>
<p>It would be nice to be able to replicate the messages in a mailbox
among multiple servers and not just do partitioning for availability.</p>
<p>We are also evaluating using the aggregator to be able to provide
mailboxes to the user with a different backup policy or even different
"quality of service." For example, we are looking to give users a
larger quota than default but not back up the servers where these
mailboxes exist.</p>
<p>There is possibility that LDAP could be used instead of MUPDATE.
However at this time the replication capabilities of LDAP are
insufficient for the needs of the Aggregator.</p>
<p>It would be nice if quotaroots had some better semantics with respect
to the murder (either make them first-class entities, or have them
apply across servers).</p>
</section>
<section id="appendices">
<h2>Appendices<a class="headerlink" href="#appendices" title="Permalink to this heading"></a></h2>
<section id="appendix-a-dns-name-load-balancing">
<h3>Appendix A: DNS Name Load Balancing<a class="headerlink" href="#appendix-a-dns-name-load-balancing" title="Permalink to this heading"></a></h3>
<p>One method of load balancing is to use DNS to spread your users to
multiple machines.</p>
<p>One method is to create a DNS CNAME for each letter of the alphabet.
Then, each user sets their IMAP server to be the first letter of their
userid. For example, the userid 'tom' would set his IMAP server to be
<code class="docutils literal notranslate"><span class="pre">T.IMAP.ANDREW.CMU.EDU</span></code> and <code class="docutils literal notranslate"><span class="pre">T.IMAP.ANDREW.CMU.EDU</span></code> would resolve to
an actual mail server.</p>
<p>Given that this does not provide a good distribution, another option is
to create a DNS CNAME for each user. Using the previous example, the
user 'tom' would set his IMAP server to be <code class="docutils literal notranslate"><span class="pre">TOM.IMAP.ANDREW.CMU.EDU</span></code>
which then points to an actual mail server.</p>
<p>The good part is that you don't have all your users on one machine and
growth can be accommodated without any user reconfiguration.</p>
<p>The drawback is with shared folders. The mail client now must support
multiple servers and users must potentially configure a server for each
user with a shared folder he wishes to view. Also, the user's INBOX
hierarchy must also reside on a single machine.</p>
</section>
<section id="appendix-b-imap-multiplexing">
<span id="id2"></span><h3>Appendix B: IMAP Multiplexing<a class="headerlink" href="#appendix-b-imap-multiplexing" title="Permalink to this heading"></a></h3>
<p>Another method of spreading out the load is to use IMAP multiplexing.
This is very similar to the IMAP Aggregator in that there are frontend
and backend servers. The frontend servers do the lookup and then
forward the request to the appropriate backend server.</p>
<p>The multiplexor looks at the user who has authenticated. Once the user
has authenticated, the frontend does a lookup for the backend server
and then connects the session to a single backend server. This provides
the flexibility of balancing the users among any arbitrary server but
it creates a problem where a user can not share a folder with a user on
a different backend server.</p>
<p>Multiplexors references:</p>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference external" href="http://docs.oracle.com/cd/E19079-01/nscp.mes.svr40/816-6037-10/">Netscape Messaging Multiplexor</a></p></li>
<li><p><a class="reference external" href="http://www.siumed.edu/~pfleming/development/email/">Paul Fleming's IMAP Proxy</a></p></li>
<li><p><a class="reference external" href="http://horms.net/projects/perdition/">Perdition IMAP Proxy</a></p></li>
<li><p><a class="reference external" href="http://owmessaging.com/Mirapoint_Message_Server">Mirapoint Message Director</a> - This is a hardware solution that
also does content filtering.</p></li>
</ul>
</div></blockquote>
</section>
<section id="appendix-c-definitions">
<h3>Appendix C: Definitions<a class="headerlink" href="#appendix-c-definitions" title="Permalink to this heading"></a></h3>
<dl class="simple">
<dt>IMAP connection</dt><dd><p>A single IMAP TCP/IP session with a single IMAP server is a
"connection".</p>
</dd>
<dt>client</dt><dd><p>A client is a process on a remote computer that communicates with
the set of servers distributing mail data, be they ACAP, IMAP,
or LDAP servers. A client opens one or more connections to
various servers.</p>
</dd>
<dt>mailbox tree</dt><dd><p>The collection of all mailboxes at a given site in a namespace is
called the mailbox tree. Generally, the user Bovik's personal data
is found in <code class="docutils literal notranslate"><span class="pre">user.bovik</span></code>.</p>
</dd>
<dt>mailboxes database</dt><dd><p>A local database containing a list of mailboxes known to a
particular server.</p>
</dd>
<dt>mailbox dataset</dt><dd><p>The store of mailbox information on the ACAP server is the "mailbox
dataset".</p>
</dd>
<dt>mailbox operation</dt><dd><p>The following IMAP commands are "mailbox operations": CREATE,
RENAME, DELETE, and SETACL.</p>
</dd>
<dt>MTA</dt><dd><p>The mail transport agent (e.g. sendmail, postfix).</p>
</dd>
</dl>
<dl class="simple" id="def-murder">
<dt>Murder of IMAP servers</dt><dd><p>A grouping of IMAP servers. It sounded cool for crows so we decided
to use it for IMAP servers as well.</p>
</dd>
<dt>quota operations</dt><dd><p>The quota IMAP commands (GETQUOTA, GETQUOTAROOT, and SETQUOTA)
operate on mailbox trees. In future versions of Cyrus, it is
expected that a quotaroot will be a subset of a mailbox tree that
resides on one partition on one server. For rationale, see section
xxx.</p>
</dd>
</dl>
</section>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="murder.html" class="btn btn-neutral float-left" title="Cyrus Murder" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="murder-installation.html" class="btn btn-neutral float-right" title="Cyrus Murder: Installation and Administration" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 1993–2025, The Cyrus Team.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
