1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
|
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Shared File Systems GPFS for high availability — Cyrus IMAP 3.12.1 documentation</title>
<link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/cyrus.css" type="text/css" />
<script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
<script src="../../../_static/jquery.js"></script>
<script src="../../../_static/underscore.js"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../../_static/doctools.js"></script>
<script src="../../../_static/sphinx_highlight.js"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="How to enable telemetry" href="o-telemetry.html" />
<link rel="prev" title="Why does reconstruct -m not work?" href="o-reconstruct.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
Cyrus IMAP
</a>
<div class="version">
3.12.1
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Cyrus IMAP</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../download.html">Download</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../quickstart.html">Quickstart Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../setup.html">Setup</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../../operations.html">Operations</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../manpages/index.html">Man pages</a></li>
<li class="toctree-l2"><a class="reference internal" href="../admin.html">Administrator Guide</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../faq.html">Frequently Asked Questions</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../faq.html#features">Features</a></li>
<li class="toctree-l3"><a class="reference internal" href="../faq.html#installation-problems">Installation Problems</a></li>
<li class="toctree-l3"><a class="reference internal" href="../faq.html#common-feature-requests">Common Feature Requests</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="../faq.html#common-operational-questions">Common Operational Questions</a><ul class="current">
<li class="toctree-l4"><a class="reference internal" href="o-acls.html">How do I view ACLs on a mailbox?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-annotations.html">What annotations are available?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-configdir-tempfs.html">Is it safe to put <configdirectory>/proc and <configdirectory>/lock on a tmpfs filesystem?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-coredump.html">How to enable core dumps</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-delete-mailbox.html">Why can I not delete a mailbox as an admin user?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-deleted-expired-expunged-purged.html">When is What ... Deleted, Expired, Expunged or Purged?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-deliverdb-size.html">Why is deliver.db so large?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-freezes.html">I have multiple imapd-SERVICES configured and experience occasional freezes when I try to log in!</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-gdb.html">How to run gdb on Cyrus components</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-mailbox-doesnotexist.html">Cyrus delivers claims that the mailbox does not exist</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-mixedcase.html">Why is mail being rejected with No Mailbox found due to MiXed CaSe incoming e-mail?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-otherdatabases.html">Can I use MySQL (or another SQL database) as the primary mail store?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-pop3slow.html">Why do POP3 connections take so long, but once the connection is established all is well?</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-reconstruct.html">Why does reconstruct -m not work?</a></li>
<li class="toctree-l4 current"><a class="current reference internal" href="#">Shared File Systems GPFS for high availability</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-telemetry.html">How to enable telemetry</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-toomanyprocesses.html">The process count keeps growing!</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-unable-join-environment.html">"unable to join environment" error</a></li>
<li class="toctree-l4"><a class="reference internal" href="o-vacation-mailfrom.html">Why does Cyrus set the MAIL FROM address of the sender of vacation responses to '<>'?</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../faq.html#common-interoperability-problems">Common Interoperability Problems</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../developers.html">Developers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../support.html">Support/Community</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Cyrus SASL</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="http://www.cyrusimap.org/sasl">Cyrus SASL</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">Cyrus IMAP</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../../operations.html">Operations</a></li>
<li class="breadcrumb-item"><a href="../faq.html">Frequently Asked Questions</a></li>
<li class="breadcrumb-item active">Shared File Systems GPFS for high availability</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/reference/faqs/o-sharedfilesystem-gpfs.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="shared-file-systems-gpfs-for-high-availability">
<h1>Shared File Systems GPFS for high availability<a class="headerlink" href="#shared-file-systems-gpfs-for-high-availability" title="Permalink to this heading"></a></h1>
<p>Stephen L. Ulmer (<a class="reference external" href="mailto:ulmer%40ufl.edu">ulmer<span>@</span>ufl<span>.</span>edu</a>) actually has used Cyrus with GPFS, and
writes:</p>
<p>FOR THE LOVE OF GOD, RUN AWAY!</p>
<p>We had our Cyrus message store on GPFS for just about a year. I've been
a Unix systems administrator for almost 15 years; It was the worst
single judgement of my professional career. Period.</p>
<p>During the 18 months when we had GPFS deployed, my unit had TWO crit
sits and uncovered over 30 bugs in the GPFS software alone (not counting
stuff we found in VSD, AIX, et cetera). The situation ended with the
GPFS architect suggesting that we do something else. He's a great guy,
and he helped us many times, but the product just doesn't do what we
wanted.</p>
<p>GPFS is the successor to the Multi-Media Filesystem, which was used in
IBM's Videocharger product. It's excellent at streaming small numbers of
large files (like, say, movies). It's horrible when you get above a few
hundred-thousand files, as the systems can't possibly have enough memory
to keep track of the filesystem meta-data.</p>
<p>Our Cyrus installation has about 80K users, close to 1TB of disk, and
many millions of files. Just the number of files alone would be enough
to kill the idea of using GPFS.</p>
<p>Cyrus makes pretty extensive use of mmap().
While GPFS implements mmap(), the GPFS architect had some words about
the way certain operations are accomplished in Cyrus. I think there are
(or used to be) places where an mmap'd file is opened for write with
another file handle (or from another process). GPFS doesn't handle this
well. This technique works accidentally on non-clustered filesystems
because AIX (also) mmap's things for you behind your back (in addition
to whatever you do) and then serializes all of the access to regions of
those files. That's really the only reason why Cyrus works on JFS.</p>
<p>Also note that the other groups/developers within IBM (especially the
group that does the automounter) have their collective heads up their
ass with respect to supporting "after market" filesystems on AIX. After
two freakin' years of PMRs they still couldn't figure out how to make
autofs interact predictably with locally-mounted GPFSs. I constantly had
to employ work-arounds in my automounter maps.</p>
<p>If you just want failover, then use the HACMP product to deal with the
failover scenario. If you need to scale beyond one system image, try out
a Cyrus Murder. That's what we're using now, it works great.</p>
<p>Note that in the Murder scenario, you can still use HACMP to have one of
your back-ends take over for another if it fails. You just have to
carefully craft your cyrus.conf files to only bind to a single IP
address, so that you can run two separate instances of Cyrus on the same
machine during the failover.</p>
<p>I will be happy to discuss our particular situation, implementation
choices and details with you if you'd like to contact me out-of-band.</p>
<p>We're currently running our Murder on:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">2</span> <span class="n">x</span> <span class="n">p630</span> <span class="p">[</span><span class="n">backends</span><span class="p">]</span>
<span class="mi">4</span> <span class="n">x</span> <span class="mf">1.4</span><span class="n">GHz</span> <span class="n">Power4</span><span class="o">+</span> <span class="n">CPU</span>
<span class="mi">8</span><span class="n">GB</span> <span class="n">Real</span> <span class="n">Memory</span>
<span class="mi">4</span> <span class="n">x</span> <span class="n">p615</span> <span class="p">[</span><span class="n">frontends</span><span class="p">]</span>
<span class="mi">2</span> <span class="n">x</span> <span class="mf">1.2</span><span class="n">GHz</span> <span class="n">Pwer4</span><span class="o">+</span> <span class="n">CPU</span>
<span class="mi">4</span><span class="n">GB</span> <span class="n">Real</span> <span class="n">Memory</span>
</pre></div>
</div>
<p>The frontends are also the web servers for our Virtual Hosting cluster. We're running version 2.1.x of Cyrus. Now that 2.2.x is stable we'll upgrade, but you can imagine that it'll take some planning.</p>
<p>In short, we don't recommend it. If you want to do it, it may possibly work but you may also lose your email or have corrupted cyrus.* files. You can look at the mailing list archives for more information.</p>
<p>There are several critical things that must be supported for this to work:</p>
<blockquote>
<div><ul class="simple">
<li><p>file locking</p></li>
<li><p>mmap()</p></li>
<li><p>writing to an mmap()'d file with the write() syscall</p></li>
</ul>
</div></blockquote>
<p>In general, this is bad news. Don't do it. Really. About the closest you can currently get is having a "warm spare" that takes over the IP of the affected backend, and have a SAN between the two systems.</p>
<p><a class="reference internal" href="../admin/murder/murder.html#murder"><span class="std std-ref">Cyrus Murder</span></a> may be a "good enough" solution in some environments, as a way to partition failures and spread load across many machines. Combined with a warm-spare approach this can be a good way to achieve highly available systems (though not ones which are fully redundant)</p>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="o-reconstruct.html" class="btn btn-neutral float-left" title="Why does reconstruct -m not work?" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="o-telemetry.html" class="btn btn-neutral float-right" title="How to enable telemetry" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 1993–2025, The Cyrus Team.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
