1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
|
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Cyrus Backups — Cyrus IMAP 3.10.2 documentation</title>
<link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/cyrus.css" type="text/css" />
<script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
<script src="../../../_static/jquery.js"></script>
<script src="../../../_static/underscore.js"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../../_static/doctools.js"></script>
<script src="../../../_static/sphinx_highlight.js"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="Cyrus NNTP" href="nntp.html" />
<link rel="prev" title="Cyrus Sieve" href="sieve.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
Cyrus IMAP
</a>
<div class="version">
3.10.2
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Cyrus IMAP</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../download.html">Download</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../quickstart.html">Quickstart Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../setup.html">Setup</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../../operations.html">Operations</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../manpages/index.html">Man pages</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../admin.html">Administrator Guide</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../admin.html#architecture">Architecture</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="../admin.html#management">Management</a><ul class="current">
<li class="toctree-l4"><a class="reference internal" href="locations.html">File & Directory Locations</a></li>
<li class="toctree-l4"><a class="reference internal" href="ports-sockets.html">Ports and Sockets</a></li>
<li class="toctree-l4"><a class="reference internal" href="access-control.html">Access Control</a></li>
<li class="toctree-l4"><a class="reference internal" href="quotas.html">Quotas</a></li>
<li class="toctree-l4"><a class="reference internal" href="sieve.html">Cyrus Sieve</a></li>
<li class="toctree-l4 current"><a class="current reference internal" href="#">Cyrus Backups</a></li>
<li class="toctree-l4"><a class="reference internal" href="nntp.html">Cyrus NNTP</a></li>
<li class="toctree-l4"><a class="reference internal" href="protlayer.html">Cyrus Prot Layer</a></li>
<li class="toctree-l4"><a class="reference internal" href="sop.html">Standard Operating Procedures</a></li>
<li class="toctree-l4"><a class="reference internal" href="eventsource.html">Cyrus Event Source</a></li>
<li class="toctree-l4"><a class="reference internal" href="monitoring.html">Monitoring</a></li>
<li class="toctree-l4"><a class="reference internal" href="config-mailboxdistribution.html">Mailbox Distribution</a></li>
<li class="toctree-l4"><a class="reference internal" href="murder/murder.html">Cyrus Murder</a></li>
<li class="toctree-l4"><a class="reference internal" href="nginx-proxy.html">HOWTO: Using an NGINX IMAP Proxy</a></li>
<li class="toctree-l4"><a class="reference internal" href="tweaking.html">Tweaking Cyrus IMAP</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../faq.html">Frequently Asked Questions</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../developers.html">Developers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../support.html">Support/Community</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Cyrus SASL</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="http://www.cyrusimap.org/sasl">Cyrus SASL</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">Cyrus IMAP</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../../operations.html">Operations</a></li>
<li class="breadcrumb-item"><a href="../admin.html">Administrator Guide</a></li>
<li class="breadcrumb-item active">Cyrus Backups</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/reference/admin/backups.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="cyrus-backups">
<span id="id1"></span><h1><a class="toc-backref" href="#id2" role="doc-backlink">Cyrus Backups</a><a class="headerlink" href="#cyrus-backups" title="Permalink to this heading"></a></h1>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>This experimental feature is no longer under active development. It
is considered deprecated as of 3.10, and will be removed entirely in
a future version.</p>
</div>
<nav class="contents" id="contents">
<p class="topic-title">Contents</p>
<ul class="simple">
<li><p><a class="reference internal" href="#cyrus-backups" id="id2">Cyrus Backups</a></p>
<ul>
<li><p><a class="reference internal" href="#introduction" id="id3">Introduction</a></p></li>
<li><p><a class="reference internal" href="#limitations" id="id4">Limitations</a></p></li>
<li><p><a class="reference internal" href="#architecture" id="id5">Architecture</a></p></li>
<li><p><a class="reference internal" href="#installation" id="id6">Installation</a></p>
<ul>
<li><p><a class="reference internal" href="#requirements" id="id7">Requirements</a></p></li>
<li><p><a class="reference internal" href="#cyrus-backups-server" id="id8">Cyrus Backups server</a></p></li>
<li><p><a class="reference internal" href="#cyrus-imap-servers" id="id9">Cyrus IMAP servers</a></p>
<ul>
<li><p><a class="reference internal" href="#using-rolling-replication" id="id10">Using rolling replication</a></p></li>
<li><p><a class="reference internal" href="#using-scheduled-replication-push" id="id11">Using scheduled replication (push)</a></p></li>
<li><p><a class="reference internal" href="#using-scheduled-replication-pull" id="id12">Using scheduled replication (pull)</a></p></li>
</ul>
</li>
</ul>
</li>
<li><p><a class="reference internal" href="#administration" id="id13">Administration</a></p>
<ul>
<li><p><a class="reference internal" href="#storage-requirements" id="id14">Storage requirements</a></p></li>
<li><p><a class="reference internal" href="#initial-backups" id="id15">Initial backups</a></p></li>
<li><p><a class="reference internal" href="#restoring-from-backups" id="id16">Restoring from backups</a></p></li>
<li><p><a class="reference internal" href="#file-locking" id="id17">File locking</a></p></li>
<li><p><a class="reference internal" href="#moving-backup-files-to-different-backup-partitions" id="id18">Moving backup files to different backup partitions</a></p></li>
<li><p><a class="reference internal" href="#provoking-a-backup-for-a-particular-user-user-group-everyone-etc-right-now" id="id19">Provoking a backup for a particular user/user group/everyone/etc right now</a></p></li>
<li><p><a class="reference internal" href="#what-about-tape-backups" id="id20">What about tape backups?</a></p></li>
<li><p><a class="reference internal" href="#multiple-imap-servers-one-backup-server" id="id21">Multiple IMAP servers, one backup server</a></p></li>
<li><p><a class="reference internal" href="#multiple-imap-servers-multiple-backup-servers" id="id22">Multiple IMAP servers, multiple backup servers</a></p></li>
<li><p><a class="reference internal" href="#one-imap-server-multiple-backup-servers" id="id23">One IMAP server, multiple backup servers</a></p></li>
<li><p><a class="reference internal" href="#reducing-load" id="id24">Reducing load</a></p></li>
<li><p><a class="reference internal" href="#other-setups" id="id25">Other setups</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#tools" id="id26">Tools</a></p>
<ul>
<li><p><a class="reference internal" href="#ctl-backups" id="id27">ctl_backups</a></p></li>
<li><p><a class="reference internal" href="#cyr-backup" id="id28">cyr_backup</a></p></li>
<li><p><a class="reference internal" href="#restore" id="id29">restore</a></p></li>
</ul>
</li>
</ul>
</li>
</ul>
</nav>
<section id="introduction">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Introduction</a><a class="headerlink" href="#introduction" title="Permalink to this heading"></a></h2>
<p>Cyrus Backups are a replication-based backup service for Cyrus IMAP servers.
This is a deprecated experimental feature.</p>
<p>This document is intended to be a guide to the configuration and
administration of Cyrus Backups.</p>
<p>This document is a work in progress and at this point is incomplete.</p>
<p>This document assumes that you are familiar with compiling, installing,
configuring and maintaining Cyrus IMAP servers generally, and will only discuss
backup-related portions in detail.</p>
<p>This document assumes a passing familiarity with
<a class="reference internal" href="sop/replication.html#replication"><span class="std std-ref">Cyrus Replication</span></a>.</p>
</section>
<section id="limitations">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Limitations</a><a class="headerlink" href="#limitations" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Cyrus Backups are experimental, incomplete, and deprecated as of 3.10.</p>
</div>
<p>The following components exist and appear to work:</p>
<ul class="simple">
<li><p>backupd, and therefore inbound replication</p></li>
<li><p>autovivification of backup storage for new users, with automatic partition
selection</p></li>
<li><p>rebuilding of backup indexes from backup data files</p></li>
<li><p>compaction of backup files to remove stale data and combine chunks for
better compression</p></li>
<li><p>deep verification of backup file/index state</p></li>
<li><p>examination of backup data</p></li>
<li><p>locking tool, for safe non-cyrus operations on backup files</p></li>
<li><p>recovery of data back into a Cyrus IMAP server</p></li>
</ul>
<p>The following components don't yet exist in a workable state -- these tasks
must be massaged through manually (with care):</p>
<ul class="simple">
<li><p>reconstruct of backups.db from backup files</p></li>
</ul>
<p>The following types of information are currently backed up and recoverable</p>
<ul class="simple">
<li><p>mailbox state and annotations</p></li>
<li><p>messages</p></li>
<li><p>mailbox message records, flags, and annotations</p></li>
</ul>
<p>The following types of information are currently backed up, but tools to
recover them don't yet exist:</p>
<ul class="simple">
<li><p>sieve scripts (but not active script status)</p></li>
<li><p>subscriptions</p></li>
<li><p>seen data</p></li>
</ul>
<p>The following types of information are not currently backed up</p>
<ul class="simple">
<li><p>quota information</p></li>
</ul>
</section>
<section id="architecture">
<h2><a class="toc-backref" href="#id5" role="doc-backlink">Architecture</a><a class="headerlink" href="#architecture" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Cyrus Backups are experimental, incomplete, and deprecated as of 3.10.</p>
</div>
<p>Cyrus Backups are designed to run on one or more standalone, dedicated backup
servers, with suitably-sized storage partitions. These servers generally do
not run an IMAP daemon, nor do they have conventional mailbox storage.</p>
<p>Your Cyrus IMAP servers synchronise mailbox state to the Cyrus Backup server(s)
using the Cyrus replication (aka sync, aka csync) protocol.</p>
<p>Backup data is stored in two files per user: a data file, containing gzipped
chunks of replication commands; and an SQLite database, which indexes the
current state of the backed up data. User backup files are stored in a hashed
subdirectory of their containing partition.</p>
<p>A twoskip database, backups.db, stores mappings of users to their backup file
locations</p>
</section>
<section id="installation">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Installation</a><a class="headerlink" href="#installation" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Cyrus Backups are experimental, incomplete, and deprecated as of 3.10.</p>
</div>
<section id="requirements">
<h3><a class="toc-backref" href="#id7" role="doc-backlink">Requirements</a><a class="headerlink" href="#requirements" title="Permalink to this heading"></a></h3>
<ul class="simple">
<li><p>At least one Cyrus IMAP server, serving and storing user data.</p></li>
<li><p>At least one machine which will become the first backup server.</p></li>
</ul>
</section>
<section id="cyrus-backups-server">
<h3><a class="toc-backref" href="#id8" role="doc-backlink">Cyrus Backups server</a><a class="headerlink" href="#cyrus-backups-server" title="Permalink to this heading"></a></h3>
<ol class="arabic">
<li><p>Compile cyrus with the <code class="docutils literal notranslate"><span class="pre">--enable-backup</span></code> configure option and install it.</p></li>
<li><p>Set up an <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a> file for it with the following options
(default values shown):</p>
<blockquote>
<div><dl class="simple">
<dt>backup_db: twoskip</dt><dd><p>The twoskip database format is recommended for backups.db</p>
</dd>
<dt>backup_db_path: {configdirectory}/backups.db</dt><dd><p>The backups db contains a mapping of user ids to their backup locations</p>
</dd>
<dt>backup_staging_path: {temp_path}/backup</dt><dd><p>Directory to use for staging message files during backup operations.
The replication protocol will transfer as many as 1024 messages in a
single sync operation, so, conservatively, this directory needs to
contain enough storage for 1024 * your maximum message size * number
of running backupd's, plus some wiggle room.</p>
</dd>
<dt>backup_retention_days: 7</dt><dd><p>Number of days for which backup data (messages etc) should be kept
within the backup storage after the corresponding item has been
deleted/expunged from the Cyrus IMAP server.</p>
</dd>
<dt>backuppartition-<em>name</em>: /path/to/this/partition</dt><dd><p>You need at least one backuppartition-<em>name</em> to store backup data.
These work similarly to regular/archive IMAP partitions, but note that
there is no relationship between backup partition names and
regular/archive partition names. New users will be have their backup
storage provisioned according to the usual partition selection rules.</p>
</dd>
<dt>backup_compact_minsize: 0</dt><dd><p>The ideal minimum data chunk size within backup files, in kB. The
compact tool will try to combine chunks that are smaller than this
into neighbouring chunks. Larger values tend to yield better
compression ratios, but if the data is corrupted on disk, the entire
chunk will become unreadable. Zero turns this behaviour off.</p>
</dd>
<dt>backup_compact_maxsize: 0</dt><dd><p>The ideal maximum data chunk size within backup files, in kB. The
compact tool will try to split chunks that are larger than this into
multiple smaller chunks. Zero turns this behaviour off.</p>
</dd>
<dt>backup_compact_work_threshold: 1</dt><dd><p>The number of chunks within a backup file that must obviously need
compaction before the compact tool will attempt to compact the file.
Larger values are expected to reduce compaction I/O load at the expense
of delayed recovery of storage space.</p>
</dd>
</dl>
</div></blockquote>
</li>
<li><p>Create a user for authenticating to the backup system, and add it to the
<code class="docutils literal notranslate"><span class="pre">admins</span></code> setting in <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a></p></li>
<li><p>Add appropriate <code class="docutils literal notranslate"><span class="pre">sasl_*</span></code> settings for your authentication method to
<a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a></p></li>
<li><p>Set up a <a class="reference internal" href="../manpages/configs/cyrus.conf.html#std-cyrusman-cyrus.conf-5">cyrus.conf(5)</a> file for it:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">START</span> <span class="p">{</span>
<span class="c1"># this is required</span>
<span class="n">recover</span> <span class="n">cmd</span><span class="o">=</span><span class="s2">"ctl_cyrusdb -r"</span>
<span class="p">}</span>
<span class="n">SERVICES</span> <span class="p">{</span>
<span class="c1"># backupd is probably the only service entry your backup server needs</span>
<span class="n">backupd</span> <span class="n">cmd</span><span class="o">=</span><span class="s2">"backupd"</span> <span class="n">listen</span><span class="o">=</span><span class="s2">"csync"</span> <span class="n">prefork</span><span class="o">=</span><span class="mi">0</span>
<span class="p">}</span>
<span class="n">EVENTS</span> <span class="p">{</span>
<span class="c1"># this is required</span>
<span class="n">checkpoint</span> <span class="n">cmd</span><span class="o">=</span><span class="s2">"ctl_cyrusdb -c"</span> <span class="n">period</span><span class="o">=</span><span class="mi">30</span>
<span class="c1"># arrange for compact to run at some interval</span>
<span class="n">compact</span> <span class="n">cmd</span><span class="o">=</span><span class="s2">"ctl_backups compact -A"</span> <span class="n">at</span><span class="o">=</span><span class="mi">0400</span>
<span class="p">}</span>
</pre></div>
</div>
</li>
<li><p>Start up the server, and use <a class="reference internal" href="../manpages/usercommands/synctest.html#std-cyrusman-synctest-1">synctest(1)</a> to verify that you can
authenticate to backupd</p></li>
</ol>
</section>
<section id="cyrus-imap-servers">
<h3><a class="toc-backref" href="#id9" role="doc-backlink">Cyrus IMAP servers</a><a class="headerlink" href="#cyrus-imap-servers" title="Permalink to this heading"></a></h3>
<p>Your Cyrus IMAP servers must be running version 3 or later of Cyrus, and must
have been compiled with the <code class="docutils literal notranslate"><span class="pre">--enable-replication</span></code> configure option. It does
<em>not</em> need to be recompiled with the <code class="docutils literal notranslate"><span class="pre">--enable-backup</span></code> option.</p>
<p>It's recommended to set up a dedicated replication channel for backups, so that
your backup replication can coexist independently of your other replication
configurations</p>
<p>Add settings to <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a> like (default values shown):</p>
<dl class="simple">
<dt><em>channel</em>_sync_host: backup-server.example.com</dt><dd><p>The host name of your Cyrus Backup server</p>
</dd>
<dt><em>channel</em>_sync_port: csync</dt><dd><p>The port on which your Cyrus Backup server's backupd process listens</p>
</dd>
<dt><em>channel</em>_sync_authname: ...</dt><dd><p>Credentials for authenticating to the Cyrus Backup server</p>
</dd>
<dt><em>channel</em>_sync_password: ...</dt><dd><p>Credentials for authenticating to the Cyrus Backup server</p>
</dd>
</dl>
<section id="using-rolling-replication">
<h4><a class="toc-backref" href="#id10" role="doc-backlink">Using rolling replication</a><a class="headerlink" href="#using-rolling-replication" title="Permalink to this heading"></a></h4>
<p>You can configure backups to use rolling replication. Depending on the sync
repeat interval you configure, this can be used to keep your backups very
current -- potentially as current as your other replicas.</p>
<p>To configure rolling replication, add additional settings to
<a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a> like:</p>
<dl class="simple">
<dt>sync_log: 1</dt><dd><p>Enable sync log if it wasn't already.</p>
</dd>
<dt>sync_log_channels: <em>channel</em></dt><dd><p>Add a new channel "<em>channel</em>" to whatever was already here. Suggest calling
this "backup"</p>
</dd>
<dt><em>channel</em>_sync_repeat_interval: 1</dt><dd><p>Minimum time in seconds between rolling replication runs. Smaller value
means livelier backups but more network I/O. Larger value reduces I/O.</p>
</dd>
</dl>
<p>Update <a class="reference internal" href="../manpages/configs/cyrus.conf.html#std-cyrusman-cyrus.conf-5">cyrus.conf(5)</a> to add a <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> invocation
to the DAEMON section specifying (at least) the <code class="docutils literal notranslate"><span class="pre">-r</span></code> and <code class="docutils literal notranslate"><span class="pre">-n</span> <span class="pre">channel</span></code>
options.</p>
<p>See <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a> for additional <em>sync_</em> settings that can
be used to affect the replication behaviour. Many can be prefixed with
a channel to limit their affect to only backups, if necessary.</p>
</section>
<section id="using-scheduled-replication-push">
<h4><a class="toc-backref" href="#id11" role="doc-backlink">Using scheduled replication (push)</a><a class="headerlink" href="#using-scheduled-replication-push" title="Permalink to this heading"></a></h4>
<p>You can configure backups to occur on a schedule determined by the IMAP
server.</p>
<p>To do this, add <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> invocations to the EVENTS section
of <a class="reference internal" href="../manpages/configs/cyrus.conf.html#std-cyrusman-cyrus.conf-5">cyrus.conf(5)</a> (or cron, etc), specifying at least the
<code class="docutils literal notranslate"><span class="pre">-n</span> <span class="pre">channel</span></code> option (to use the channel-specific configuration), plus
whatever other options you need for selecting users to back up. See the
<a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> manpage for details.</p>
<p>You could also invoke <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> in a similar way from a
custom script running on the IMAP server.</p>
</section>
<section id="using-scheduled-replication-pull">
<h4><a class="toc-backref" href="#id12" role="doc-backlink">Using scheduled replication (pull)</a><a class="headerlink" href="#using-scheduled-replication-pull" title="Permalink to this heading"></a></h4>
<p>You can configure backups to occur on a schedule determined by the
backup server. For example, you may have a custom script that examines
the existing backups, and provokes fresh backups to occur if they are
determined to be out of date.</p>
<p>To to this, enable XBACKUP on your IMAP server by adding the following
setting to <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a>:</p>
<dl class="simple">
<dt>xbackup_enabled: yes</dt><dd><p>Enables the XBACKUP command in imapd.</p>
</dd>
</dl>
<p>Your custom script can then authenticate to the IMAP server as an admin
user, and invoke the command <code class="docutils literal notranslate"><span class="pre">XBACKUP</span> <span class="pre">pattern</span> <span class="pre">[channel]</span></code>. A replication
of the users or shared mailboxes matching the specified pattern will occur
to the backup server defined by the named channel. If no channel is
specified, default sync configuration will be used.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">C</span><span class="p">:</span> <span class="mi">1</span> <span class="n">XBACKUP</span> <span class="n">user</span><span class="o">.*</span> <span class="n">backup</span>
<span class="n">S</span><span class="p">:</span> <span class="o">*</span> <span class="n">OK</span> <span class="n">USER</span> <span class="n">anne</span>
<span class="n">S</span><span class="p">:</span> <span class="o">*</span> <span class="n">OK</span> <span class="n">USER</span> <span class="n">bethany</span>
<span class="n">S</span><span class="p">:</span> <span class="o">*</span> <span class="n">NO</span> <span class="n">USER</span> <span class="n">cassandane</span> <span class="p">(</span><span class="n">Operation</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">supported</span> <span class="n">on</span> <span class="n">mailbox</span><span class="p">)</span>
<span class="n">S</span><span class="p">:</span> <span class="o">*</span> <span class="n">OK</span> <span class="n">USER</span> <span class="n">demi</span>
<span class="n">S</span><span class="p">:</span> <span class="o">*</span> <span class="n">OK</span> <span class="n">USER</span> <span class="n">ellie</span>
<span class="n">S</span><span class="p">:</span> <span class="mi">1</span> <span class="n">OK</span> <span class="n">Completed</span>
</pre></div>
</div>
<p>This replicates all users to the channel <em>backup</em>.</p>
</section>
</section>
</section>
<section id="administration">
<h2><a class="toc-backref" href="#id13" role="doc-backlink">Administration</a><a class="headerlink" href="#administration" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Cyrus Backups are experimental, incomplete, and deprecated as of 3.10.</p>
</div>
<section id="storage-requirements">
<h3><a class="toc-backref" href="#id14" role="doc-backlink">Storage requirements</a><a class="headerlink" href="#storage-requirements" title="Permalink to this heading"></a></h3>
<p>It's not really known yet how to predict the storage requirements for a backup
server. Experimentation in dev environment suggests around 20-40% compressed
backup file size relative to the backed up data, depending on compact settings,
but this is with relatively tiny mailboxes and non-pathological data.</p>
<p>The backup staging spool conservatively needs to be large enough to hold an
entire sync's worth of message files at once. Which is your maximum message
size * 1024 messages * the number of backupd processes you're running, plus
some wiggle room probably. In practice it'll probably not hit this limit
unless someone is trying to. (Most users, I suspect, don't have 1024
maximum-sized messages in their account, or don't receive them all at once
anyway.)</p>
<p>Certain invocations of ctl_backups and cyr_backup also require staging spool
space, due to the way replication protocol (and thus backup data) parsing
handles messages. So keep this in mind I suppose.</p>
</section>
<section id="initial-backups">
<h3><a class="toc-backref" href="#id15" role="doc-backlink">Initial backups</a><a class="headerlink" href="#initial-backups" title="Permalink to this heading"></a></h3>
<p>Once a Cyrus Backup system is configured and running, new users that are
created on the IMAP servers will be backed up seamlessly without administrator
intervention.</p>
<p>The very first backup taken of a pre-existing mailbox will be big -- the entire
mailbox in one hit. It's suggested that, when initially provisioning a Cyrus
Backup server for an existing Cyrus IMAP environment, that the
<a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> commands be run carefully, for a small group of
mailboxes at a time, until all/most of your mailboxes have been backed up at
least once. Also run the <a class="reference internal" href="../manpages/systemcommands/ctl_backups.html#std-cyrusman-ctl_backups-8">ctl_backups(8)</a> <code class="docutils literal notranslate"><span class="pre">compact</span></code> command on the
backups, to break up big chunks, if you wish. Only then should you enable
rolling/scheduled replication.</p>
</section>
<section id="restoring-from-backups">
<h3><a class="toc-backref" href="#id16" role="doc-backlink">Restoring from backups</a><a class="headerlink" href="#restoring-from-backups" title="Permalink to this heading"></a></h3>
<p>The <a class="reference internal" href="../manpages/systemcommands/restore.html#std-cyrusman-restore-8">restore(8)</a> tool will restore mailboxes and messages from a
specified backup to a specified destination server. The destination server must
be running a replication-capable <a class="reference internal" href="../manpages/systemcommands/imapd.html#std-cyrusman-imapd-8">imapd(8)</a> or
<a class="reference internal" href="../manpages/systemcommands/sync_server.html#std-cyrusman-sync_server-8">sync_server(8)</a>. The restore tool should be run from the backup
server containing the specified backup.</p>
</section>
<section id="file-locking">
<h3><a class="toc-backref" href="#id17" role="doc-backlink">File locking</a><a class="headerlink" href="#file-locking" title="Permalink to this heading"></a></h3>
<p>All <a class="reference internal" href="../manpages/systemcommands/backupd.html#std-cyrusman-backupd-8">backupd(8)</a>/<a class="reference internal" href="../manpages/systemcommands/ctl_backups.html#std-cyrusman-ctl_backups-8">ctl_backups(8)</a>/<a class="reference internal" href="../manpages/systemcommands/cyr_backup.html#std-cyrusman-cyr_backup-8">cyr_backup(8)</a>
operations first obtain a lock on the relevant backup file. ctl_backups and
cyr_backup will try to do this without blocking (unless told otherwise),
whereas backupd will never block.</p>
</section>
<section id="moving-backup-files-to-different-backup-partitions">
<h3><a class="toc-backref" href="#id18" role="doc-backlink">Moving backup files to different backup partitions</a><a class="headerlink" href="#moving-backup-files-to-different-backup-partitions" title="Permalink to this heading"></a></h3>
<p>There's no tool for this (yet). To do it manually, stop backupd, copy the files
to the new partition, then use <a class="reference internal" href="../manpages/systemcommands/cyr_dbtool.html#std-cyrusman-cyr_dbtool-8">cyr_dbtool(8)</a> to update the user's
backups.db entry to point to the new location. Run the
<a class="reference internal" href="../manpages/systemcommands/ctl_backups.html#std-cyrusman-ctl_backups-8">ctl_backups(8)</a> <code class="docutils literal notranslate"><span class="pre">verify</span></code> command on both the new filename (<code class="docutils literal notranslate"><span class="pre">-f</span></code>
mode) and the user's userid (<code class="docutils literal notranslate"><span class="pre">-u</span></code> mode) to ensure everything is okay, then
restart backupd.</p>
</section>
<section id="provoking-a-backup-for-a-particular-user-user-group-everyone-etc-right-now">
<h3><a class="toc-backref" href="#id19" role="doc-backlink">Provoking a backup for a particular user/user group/everyone/etc right now</a><a class="headerlink" href="#provoking-a-backup-for-a-particular-user-user-group-everyone-etc-right-now" title="Permalink to this heading"></a></h3>
<p>Just run <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> by hand with appropriate options (as cyrus
user, of course). See its man page for ways of specifying items to replicate.</p>
<p>If the IMAP server with the user's mail has been configured with the
<code class="docutils literal notranslate"><span class="pre">xbackup_enabled:</span> <span class="pre">yes</span></code> option in <a class="reference internal" href="../manpages/configs/imapd.conf.html#std-cyrusman-imapd.conf-5">imapd.conf(5)</a>, then an admin
user can cause a backup to occur by sending the IMAP server an <code class="docutils literal notranslate"><span class="pre">XBACKUP</span></code>
command.</p>
</section>
<section id="what-about-tape-backups">
<h3><a class="toc-backref" href="#id20" role="doc-backlink">What about tape backups?</a><a class="headerlink" href="#what-about-tape-backups" title="Permalink to this heading"></a></h3>
<p>As long as backupd, ctl_backups and cyr_backup are not currently running (and
assuming no-one's poking around in things otherwise), it's safe to take/restore
a filesystem snapshot of backup partitions. So to schedule, say, a nightly tape
dump of your Cyrus Backup server, make your cron job shut down Cyrus, make the
copy, then restart Cyrus.</p>
<p>Meanwhile, your Cyrus IMAP servers are still online and available. Regular
backups will resume once your backupd is running again.</p>
<p>If you can work at a finer granularity than file system, you don't need to shut
down backupd. Just use the <a class="reference internal" href="../manpages/systemcommands/ctl_backups.html#std-cyrusman-ctl_backups-8">ctl_backups(8)</a> <code class="docutils literal notranslate"><span class="pre">lock</span></code> command to hold
a lock on each backup while you work with its files, and the rest of the backup
system will work around that.</p>
<p>Restoring is more complicated, depending on what you actually need to do:
when you restart the backupd after restoring a filesystem snapshot, the next
time your Cyrus IMAP server replicates to it, the restored backups will be
brought up to date. Probably not what you wanted -- so don't restart backupd
until you've done whatever you were doing.</p>
</section>
<section id="multiple-imap-servers-one-backup-server">
<h3><a class="toc-backref" href="#id21" role="doc-backlink">Multiple IMAP servers, one backup server</a><a class="headerlink" href="#multiple-imap-servers-one-backup-server" title="Permalink to this heading"></a></h3>
<p>This is fine, as long as each user being backed up is only being backed up by
one server (or they are otherwise synchronised). If IMAP servers have different
ideas about the state of a user's mailboxes, one of those will be in sync with
the backup server and the other will get a lot of replication failures.</p>
</section>
<section id="multiple-imap-servers-multiple-backup-servers">
<h3><a class="toc-backref" href="#id22" role="doc-backlink">Multiple IMAP servers, multiple backup servers</a><a class="headerlink" href="#multiple-imap-servers-multiple-backup-servers" title="Permalink to this heading"></a></h3>
<p>Make sure your <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> configuration(s) on each IMAP server
knows which users are being backed up to which backup servers, and selects
them appropriately. See the <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> man page for options for
specifying users, and run it as an event (rather than rolling).</p>
<p>Or just distribute it at server granularity, such that backup server A serves
IMAP servers A, B and C, and backup server B serves IMAP servers D, E, F, etc.</p>
</section>
<section id="one-imap-server-multiple-backup-servers">
<h3><a class="toc-backref" href="#id23" role="doc-backlink">One IMAP server, multiple backup servers</a><a class="headerlink" href="#one-imap-server-multiple-backup-servers" title="Permalink to this heading"></a></h3>
<p>Configure one channel plus one rolling <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> per backup
server, and your IMAP server can be more or less simultaneously backed up to
multiple backup destinations.</p>
</section>
<section id="reducing-load">
<h3><a class="toc-backref" href="#id24" role="doc-backlink">Reducing load</a><a class="headerlink" href="#reducing-load" title="Permalink to this heading"></a></h3>
<p>To reduce load on your client-facing IMAP servers, configure sync log chaining
on their replicas and let those take the load of replicating to the backup
servers.</p>
<p>To reduce network traffic, do the same thing, specifically using replicas that
are already co-located with the backup server.</p>
</section>
<section id="other-setups">
<h3><a class="toc-backref" href="#id25" role="doc-backlink">Other setups</a><a class="headerlink" href="#other-setups" title="Permalink to this heading"></a></h3>
<p>The use of the replication protocol and <a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> allows a lot
of interesting configuration possibilities to shake out. Have a rummage in the
<a class="reference internal" href="../manpages/systemcommands/sync_client.html#std-cyrusman-sync_client-8">sync_client(8)</a> man page for inspiration.</p>
</section>
</section>
<section id="tools">
<h2><a class="toc-backref" href="#id26" role="doc-backlink">Tools</a><a class="headerlink" href="#tools" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Cyrus Backups are experimental, incomplete, and deprecated as of 3.10.</p>
</div>
<section id="ctl-backups">
<h3><a class="toc-backref" href="#id27" role="doc-backlink">ctl_backups</a><a class="headerlink" href="#ctl-backups" title="Permalink to this heading"></a></h3>
<p>This tool is generally for mass operations that require few/fixed arguments
across multiple/all backups</p>
<p>Supported operations:</p>
<dl class="simple">
<dt>compact</dt><dd><p>Reduce backups' disk usage by:</p>
<ul class="simple">
<li><p>combining small chunks for better gzip compression -- especially
important for hot backups, which produce many tiny chunks</p></li>
<li><p>removing deleted content that has passed its retention period</p></li>
</ul>
</dd>
<dt>list</dt><dd><p>List known backups.</p>
</dd>
<dt>lock</dt><dd><p>Lock a single backup, so you can safely work on it with non-cyrus tools.</p>
</dd>
<dt>reindex</dt><dd><p>Regenerate indexes for backups from their data files. Useful if index
becomes corrupted by some bug, or invalidated by working on data with
non-cyrus tools.</p>
</dd>
<dt>stat</dt><dd><p>Show statistics about backups -- disk usage, compression ratio, etc.</p>
</dd>
<dt>verify</dt><dd><p>Deep verification of backups. Verifies that:</p>
<ul class="simple">
<li><p>Checksums for each chunk in index match data</p></li>
<li><p>Mailbox states are in the chunk that the index says they're in</p></li>
<li><p>Mailbox states match indexed states</p></li>
<li><p>Messages are in the chunk the index says they're in</p></li>
<li><p>Message data checksum matches indexed checksums</p></li>
</ul>
</dd>
</dl>
<p>See the <a class="reference internal" href="../manpages/systemcommands/ctl_backups.html#std-cyrusman-ctl_backups-8">ctl_backups(8)</a> man page for more information.</p>
</section>
<section id="cyr-backup">
<h3><a class="toc-backref" href="#id28" role="doc-backlink">cyr_backup</a><a class="headerlink" href="#cyr-backup" title="Permalink to this heading"></a></h3>
<p>This tool is generally for operations on a single mailbox that require multiple
additional arguments</p>
<p>Supported operations</p>
<dl class="simple">
<dt>list [ chunks | mailboxes | messages | all ]</dt><dd><p>Line-per-item listing of information stored in a backup.</p>
</dd>
<dt>show [ chunks | mailboxes | messages ] items...</dt><dd><p>Paragraph-per-item listing of information for specified items. Chunk items
are specified by id, mailboxes by mboxname or uniqueid, messages by guid.</p>
</dd>
<dt>dump [ chunk | message ] item</dt><dd><p>Full dump of one item. chunk dumps the uncompressed content of a chunk
(i.e. a bunch of sync protocol commands). message dumps a raw rfc822
message (useful for manually restoring)</p>
</dd>
</dl>
<p>See the <a class="reference internal" href="../manpages/systemcommands/cyr_backup.html#std-cyrusman-cyr_backup-8">cyr_backup(8)</a> man page for more information.</p>
</section>
<section id="restore">
<h3><a class="toc-backref" href="#id29" role="doc-backlink">restore</a><a class="headerlink" href="#restore" title="Permalink to this heading"></a></h3>
<p>This tool is for restoring mail from backup files.</p>
<p>Required arguments are a destination server (in ip:port or host:port format),
a backup file, and mboxnames, uniqueids or guids specifying the mailboxes or
messages to be restored.</p>
<p>If the target mailbox does not already exist on the destination server, options
are available to preserve the mailbox and message properties as they existed
in the backup. This is useful for rebuilding a lost server from backups, such
that client state remains consistent.</p>
<p>If the target mailbox already exists on the destination server, restored
messages will be assigned new, unused uids and will appear to the client as new
messages.</p>
<p>See the <a class="reference internal" href="../manpages/systemcommands/restore.html#std-cyrusman-restore-8">restore(8)</a> man page for more information.</p>
</section>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="sieve.html" class="btn btn-neutral float-left" title="Cyrus Sieve" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="nntp.html" class="btn btn-neutral float-right" title="Cyrus NNTP" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 1993–2025, The Cyrus Team.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
