1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
|
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>7.24. Tuning — Groonga v10.1.1-31-g1e46ba6 documentation</title>
<link rel="stylesheet" href="../_static/groonga.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/language_data.js"></script>
<link rel="shortcut icon" href="../_static/favicon.ico"/>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="7.25. API" href="api.html" />
<link rel="prev" title="7.23. Log" href="log.html" />
</head><body>
<div class="header">
<h1 class="title">
<a id="top-link" href="../index.html">
<span class="project">groonga</span>
<span class="separator">-</span>
<span class="description">An open-source fulltext search engine and column store.</span>
</a>
</h1>
<div class="other-language-links">
<ul>
<li><a href="../../../ja/html/reference/tuning.html">日本語</a></li>
</ul>
</div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="api.html" title="7.25. API"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="log.html" title="7.23. Log"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">Groonga v10.1.1-31-g1e46ba6 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="../reference.html" accesskey="U"><span class="section-number">7. </span>Reference manual</a> »</li>
<li class="nav-item nav-item-this"><a href=""><span class="section-number">7.24. </span>Tuning</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="tuning">
<h1><span class="section-number">7.24. </span>Tuning<a class="headerlink" href="#tuning" title="Permalink to this headline">¶</a></h1>
<div class="section" id="summary">
<h2><span class="section-number">7.24.1. </span>Summary<a class="headerlink" href="#summary" title="Permalink to this headline">¶</a></h2>
<p>There are some tuning parameters for handling a large database.</p>
</div>
<div class="section" id="parameters">
<h2><span class="section-number">7.24.2. </span>Parameters<a class="headerlink" href="#parameters" title="Permalink to this headline">¶</a></h2>
<p>This section describes tuning parameters.</p>
<div class="section" id="the-max-number-of-open-files-per-process">
<span id="tuning-max-n-open-files"></span><h3><span class="section-number">7.24.2.1. </span>The max number of open files per process<a class="headerlink" href="#the-max-number-of-open-files-per-process" title="Permalink to this headline">¶</a></h3>
<p>This parameter is for handling a large database.</p>
<p>Groonga creates one or more files per table and column. If your
database has many tables and columns, Groonga process needs to open
many files.</p>
<p>System limits the max number of open files per process. So you need to
relax the limitation.</p>
<p>Here is an expression that compute how many files are opened by
Groonga:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>3 (for DB) +
N tables +
N columns (except index clumns) +
(N index columns * 2) +
X (the number of plugins etc.)
</pre></div>
</div>
<p>Here is an example schema:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>table_create Entries TABLE_HASH_KEY ShortText
column_create Entries content COLUMN_SCALAR Text
column_create Entries n_likes COLUMN_SCALAR UInt32
table_create Terms TABLE_PAT_KEY ShortText --default_tokenizer TokenBigram --normalizer NormalizerAuto
column_create Terms entries_key_index COLUMN_INDEX|WITH_POSITION Entries _key
column_create Terms entries_content_index COLUMN_INDEX|WITH_POSITION Entries content
</pre></div>
</div>
<p>This example opens at least 11 files:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>3 +
2 (Entries and Terms) +
2 (Entries.content and Entries.n_likes) +
4 (Terms.entries_key_index and Terms.entries_content_index) +
X = 11 + X
</pre></div>
</div>
</div>
<div class="section" id="memory-usage">
<span id="tuning-memory-usage"></span><h3><span class="section-number">7.24.2.2. </span>Memory usage<a class="headerlink" href="#memory-usage" title="Permalink to this headline">¶</a></h3>
<p>This parameter is for handling a large database.</p>
<p>Groonga maps database files onto memory and accesses to them. Groonga
doesn’t maps unnecessary files onto memory. Groonga maps files when
they are needed.</p>
<p>If you access to all data in database, all database files are mapped
onto memory. If total size of your database files is 6GiB, your
Groonga process uses 6GiB memory.</p>
<p>Normally, your all database files aren’t mapped onto memory. But it may
be occurred. It is an example case that you dump your database.</p>
<p>Normally, you must have memory and swap that is larger than
database. Linux has tuning parameter to work with less memory and swap
than database size.</p>
</div>
</div>
<div class="section" id="linux">
<span id="tuning-linux"></span><h2><span class="section-number">7.24.3. </span>Linux<a class="headerlink" href="#linux" title="Permalink to this headline">¶</a></h2>
<p>This section describes how to configure parameters on Linux.</p>
<div class="section" id="nofile">
<span id="tuning-linux-nofile"></span><h3><span class="section-number">7.24.3.1. </span><code class="docutils literal notranslate"><span class="pre">nofile</span></code><a class="headerlink" href="#nofile" title="Permalink to this headline">¶</a></h3>
<p>You can relax the <a class="reference internal" href="#tuning-max-n-open-files"><span class="std std-ref">The max number of open files per process</span></a> parameter by creating
a configuration file <code class="docutils literal notranslate"><span class="pre">/etc/security/limits.d/groonga.conf</span></code> that has
the following content:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>${USER} soft nofile ${MAX_VALUE}
${USER} hard nofile ${MAX_VALUE}
</pre></div>
</div>
<p>If you run Groonga process by <code class="docutils literal notranslate"><span class="pre">groonga</span></code> user and your Groonga
process needs to open less than 10000 files, use the following
configuration:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>groonga soft nofile 10000
groonga hard nofile 10000
</pre></div>
</div>
<p>The configuration is applied after your Groonga service is restarted
or re-login as your <code class="docutils literal notranslate"><span class="pre">groonga</span></code> user.</p>
</div>
<div class="section" id="vm-overcommit-memory">
<span id="tuning-linux-overcommit-memory"></span><h3><span class="section-number">7.24.3.2. </span><code class="docutils literal notranslate"><span class="pre">vm.overcommit_memory</span></code><a class="headerlink" href="#vm-overcommit-memory" title="Permalink to this headline">¶</a></h3>
<p>This is <a class="reference internal" href="#tuning-memory-usage"><span class="std std-ref">Memory usage</span></a> related parameter. You can handle a
database that is larger than your memory and swap by setting
<code class="docutils literal notranslate"><span class="pre">vm.overcommit_memory</span></code> kernel parameter to <code class="docutils literal notranslate"><span class="pre">1</span></code>. <code class="docutils literal notranslate"><span class="pre">1</span></code> means that
Groonga can always map database files onto memory. Groonga recommends
the configuration.</p>
<p>See <a class="reference external" href="https://www.kernel.org/doc/Documentation/vm/overcommit-accounting">Linux kernel documentation about overcommit</a>
about <code class="docutils literal notranslate"><span class="pre">vm.overcommit_memory</span></code> parameter details.</p>
<p>You can set the configuration by putting a configuration file
<code class="docutils literal notranslate"><span class="pre">/etc/sysctl.d/groonga.conf</span></code> that has the following content:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>vm.overcommit_memory = 1
</pre></div>
</div>
<p>The configuration can be applied by restarting your system or run the
following command:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>% sudo sysctl --system
</pre></div>
</div>
</div>
<div class="section" id="vm-max-map-count">
<span id="tuning-linux-max-map-count"></span><h3><span class="section-number">7.24.3.3. </span><code class="docutils literal notranslate"><span class="pre">vm.max_map_count</span></code><a class="headerlink" href="#vm-max-map-count" title="Permalink to this headline">¶</a></h3>
<p>This is <a class="reference internal" href="#tuning-memory-usage"><span class="std std-ref">Memory usage</span></a> related parameter. You can handle a
16GiB or more larger size database by increasing <code class="docutils literal notranslate"><span class="pre">vm.max_map_count</span></code>
kernel parameter. The parameter limits the max number of memory maps.</p>
<p>The default value of the kernel parameter may be <code class="docutils literal notranslate"><span class="pre">65530</span></code> or
<code class="docutils literal notranslate"><span class="pre">65536</span></code>. Groonga maps 256KiB memory chunk at one time. If a
database is larger than 16GiB, Groonga reaches the
limitation. (<code class="docutils literal notranslate"><span class="pre">256KiB</span> <span class="pre">*</span> <span class="pre">65536</span> <span class="pre">=</span> <span class="pre">16GiB</span></code>)</p>
<p>You needs to increase the value of the kernel parameter to handle
16GiB or more larger size database. For example, you can handle almost
32GiB size database by <code class="docutils literal notranslate"><span class="pre">65536</span> <span class="pre">*</span> <span class="pre">2</span> <span class="pre">=</span> <span class="pre">131072</span></code>. You can set the
configuration by putting a configuration file
<code class="docutils literal notranslate"><span class="pre">/etc/sysctl.d/groonga.conf</span></code> that has the following content:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>vm.max_map_count = 131072
</pre></div>
</div>
<p>Note that your real configuration file will be the following because
you already have <code class="docutils literal notranslate"><span class="pre">vm.overcommit_memory</span></code> configuration:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>vm.overcommit_memory = 1
vm.max_map_count = 131072
</pre></div>
</div>
<p>The configuration can be applied by restarting your system or run the
following command:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>% sudo sysctl -p
</pre></div>
</div>
</div>
</div>
<div class="section" id="freebsd">
<h2><span class="section-number">7.24.4. </span>FreeBSD<a class="headerlink" href="#freebsd" title="Permalink to this headline">¶</a></h2>
<p>This section describes how to configure parameters on FreeBSD.</p>
<div class="section" id="kern-maxfileperproc">
<span id="tuning-freebsd-maxfilesperproc"></span><h3><span class="section-number">7.24.4.1. </span><code class="docutils literal notranslate"><span class="pre">kern.maxfileperproc</span></code><a class="headerlink" href="#kern-maxfileperproc" title="Permalink to this headline">¶</a></h3>
<p>TODO</p>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="../index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">7.24. Tuning</a><ul>
<li><a class="reference internal" href="#summary">7.24.1. Summary</a></li>
<li><a class="reference internal" href="#parameters">7.24.2. Parameters</a><ul>
<li><a class="reference internal" href="#the-max-number-of-open-files-per-process">7.24.2.1. The max number of open files per process</a></li>
<li><a class="reference internal" href="#memory-usage">7.24.2.2. Memory usage</a></li>
</ul>
</li>
<li><a class="reference internal" href="#linux">7.24.3. Linux</a><ul>
<li><a class="reference internal" href="#nofile">7.24.3.1. <code class="docutils literal notranslate"><span class="pre">nofile</span></code></a></li>
<li><a class="reference internal" href="#vm-overcommit-memory">7.24.3.2. <code class="docutils literal notranslate"><span class="pre">vm.overcommit_memory</span></code></a></li>
<li><a class="reference internal" href="#vm-max-map-count">7.24.3.3. <code class="docutils literal notranslate"><span class="pre">vm.max_map_count</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#freebsd">7.24.4. FreeBSD</a><ul>
<li><a class="reference internal" href="#kern-maxfileperproc">7.24.4.1. <code class="docutils literal notranslate"><span class="pre">kern.maxfileperproc</span></code></a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="log.html"
title="previous chapter"><span class="section-number">7.23. </span>Log</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="api.html"
title="next chapter"><span class="section-number">7.25. </span>API</a></p>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="api.html" title="7.25. API"
>next</a> |</li>
<li class="right" >
<a href="log.html" title="7.23. Log"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">Groonga v10.1.1-31-g1e46ba6 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="../reference.html" ><span class="section-number">7. </span>Reference manual</a> »</li>
<li class="nav-item nav-item-this"><a href=""><span class="section-number">7.24. </span>Tuning</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2009-2021, Brazil, Inc.
</div>
</body>
</html>
|
