1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
|
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Warming the memory pool</title>
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
<link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
<link rel="up" href="mp.html" title="Chapter 18. The Memory Pool Subsystem" />
<link rel="prev" href="mp_config.html" title="Configuring the memory pool" />
<link rel="next" href="txn.html" title="Chapter 19. The Transaction Subsystem" />
</head>
<body>
<div xmlns="" class="navheader">
<div class="libver">
<p>Library Version 11.2.5.3</p>
</div>
<table width="100%" summary="Navigation header">
<tr>
<th colspan="3" align="center">Warming the memory pool</th>
</tr>
<tr>
<td width="20%" align="left"><a accesskey="p" href="mp_config.html">Prev</a> </td>
<th width="60%" align="center">Chapter 18.
The Memory Pool Subsystem
</th>
<td width="20%" align="right"> <a accesskey="n" href="txn.html">Next</a></td>
</tr>
</table>
<hr />
</div>
<div class="sect1" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h2 class="title" style="clear: both"><a id="mp_warm"></a>Warming the memory pool</h2>
</div>
</div>
</div>
<div class="toc">
<dl>
<dt>
<span class="sect2">
<a href="mp_warm.html#warm_cache">The warm_cache() function</a>
</span>
</dt>
</dl>
</div>
<p>
Some applications find it is useful to pre-load the memory pool
upon application startup. This is a strictly optional activity that
provides faster initial access to your data at the expense of
longer application startup times.
</p>
<p>
To warm the cache, you simply have to read the records that your
application will operate on most frequently. You can do this with
normal database reads, and you can also use cursors. But the most
efficient way to warm the cache is to use memory pool APIs to get
the pages that contain your most frequently accessed records.
</p>
<p>
You read pages into the memory pool using the
<code class="methodname">DB_MPOOLFILE->get()</code> method. This method
acquires locks on the page, so immediately upon getting the page
you need to put it so as to release the locks.
</p>
<p>
Also, you obtain a memory pool file handle using a database handle.
This means that if your data is contained in more than one Berkeley
DB database, you must operate on each database handle in turn.
</p>
<p>
The following example code illustrates this. It does the following:
</p>
<div class="itemizedlist">
<ul type="disc">
<li>
<p>
Opens an environment and two database handles.
</p>
</li>
<li>
<p>
Determines how many database pages can fit into the memory
pool.
</p>
</li>
<li>
<p>
Uses <code class="methodname">DB_MPOOLFILE->get()</code> and
<code class="methodname">DB_MPOOLFILE->put()</code>
to load that number of pages into the memory pool.
</p>
</li>
</ul>
</div>
<p>
First, we include the libraries that we need, forward declare some
functions, and intialize some variables.
</p>
<a id="prog_mp01-1"></a>
<pre class="programlisting">#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <db.h>
/* Forward declarations */
int warm_cache(DB *, int *, int);
int open_db(DB_ENV *, DB **, const char *);
int
main(void)
{
DB *dbp1 = 0, *dbp2 = 0;
DB_ENV *envp = 0;
u_int32_t env_flags, pagesize, gbytes, bytes;
int ret = 0, ret_t = 0, numcachepages, pagecount; </pre>
<p>
Then we open the environment and our databases. The
<code class="methodname">open_db()</code> function that we use here simply
opens a database. We will provide that code at the end of this
example, but it should hold no surprises for you.
We only use the function so as to reuse the code.
</p>
<a id="prog_mp01-2"></a>
<pre class="programlisting"> /*
* Open the environment and the databases
*/
ret = db_env_create(&envp, 0);
if (ret != 0) {
fprintf(stderr, "Error creating environment handle: %s\n",
db_strerror(ret));
goto err;
}
env_flags =
DB_CREATE | /* Create the environment if it does
not exist */
DB_RECOVER | /* Run normal recovery. */
DB_INIT_LOCK | /* Initialize the locking subsystem */
DB_INIT_LOG | /* Initialize the logging subsystem */
DB_INIT_TXN | /* Initialize the transactional subsystem. This
* also turns on logging. */
DB_INIT_MPOOL; /* Initialize the memory pool */
/* Now actually open the environment */
ret = envp->open(envp, "./env", env_flags, 0);
if (ret != 0) {
fprintf(stderr, "Error opening environment: %s\n",
db_strerror(ret));
goto err;
}
ret = open_db(envp, &dbp1, "mydb1.db");
if (ret != 0)
goto err;
ret = open_db(envp, &dbp2, "mydb2.db");
if (ret != 0)
goto err; </pre>
<p>
Next we determine how many database pages we can fit into the
cache. We do this by finding out how large our pages are, and then
finding out how large our cache can be.
</p>
<a id="prog_mp01-3"></a>
<pre class="programlisting"> /* Find out how many pages can fit at most in the cache */
ret = envp->get_mp_pagesize(envp, &pagesize);
if (ret != 0) {
fprintf(stderr, "Error retrieving the cache pagesize: %s\n",
db_strerror(ret));
goto err;
}
ret = envp->get_cache_max(envp, &gbytes, &bytes);
if (ret != 0) {
fprintf(stderr, "Error retrieving maximum cache size: %s\n",
db_strerror(ret));
goto err;
}
/* Avoid an overflow by first calculating pages per gigabyte. */
numcachepages = gbytes * ((1024 * 1024 * 1024) / pagesize);
numcachepages += bytes / pagesize; </pre>
<p>
Now we call our <code class="methodname">warm_cache()</code>
function. We will describe this function in a little while, but
note that we call <code class="methodname">warm_cache()</code>
twice. This is because our example uses two databases, and the
memory pool methods operate on a per-handle basis.
</p>
<a id="prog_mp01-4"></a>
<pre class="programlisting"> /*
* Warm the cache by loading pages from each of the databases
* in turn.
*/
pagecount = 0;
ret = warm_cache(dbp1, &pagecount, numcachepages);
if (ret != 0) {
fprintf(stderr, "Error warming the cache: %s\n",
db_strerror(ret));
goto err;
}
ret = warm_cache(dbp2, &pagecount, numcachepages);
if (ret != 0) {
fprintf(stderr, "Error warming the cache: %s\n",
db_strerror(ret));
goto err;
} </pre>
<p>
Now we close all our handles and finish our
<code class="methodname">main()</code> function. Again, this is
straight-forward boilerplate code that we provide simply to be
complete.
</p>
<a id="prog_mp01-5"></a>
<pre class="programlisting">err:
/* Close our database handles, if they were opened. */
if (dbp1 != NULL) {
ret_t = dbp1->close(dbp1, 0);
if (ret_t != 0) {
fprintf(stderr, "dbp1 close failed: %s\n",
db_strerror(ret_t));
ret = ret_t;
}
}
if (dbp2 != NULL) {
ret_t = dbp2->close(dbp2, 0);
if (ret_t != 0) {
fprintf(stderr, "dbp2 close failed: %s\n",
db_strerror(ret_t));
ret = ret_t;
}
}
/* Close our environment, if it was opened. */
if (envp != NULL) {
ret_t = envp->close(envp, 0);
if (ret_t != 0) {
fprintf(stderr, "environment close failed: %s\n",
db_strerror(ret_t));
ret = ret_t;
}
}
/* Final status message and return. */
printf("I'm all done.\n");
return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE);
} </pre>
<p>
As noted above, this example uses an <code class="methodname">open_db()</code>
function, which opens a database handle inside the provided
environment. To be complete, this is the implementation of that
function:
</p>
<a id="prog_mp01-6"></a>
<pre class="programlisting">/* Open a database handle */
int
open_db(DB_ENV *envp, DB **dbpp, const char *file_name)
{
int ret = 0;
u_int32_t db_flags = 0;
DB *dbp;
/* Open the database */
ret = db_create(&dbp, envp, 0);
if (ret != 0) {
fprintf(stderr, "Error opening database: %s : %s\n",
file_name, db_strerror(ret));
return ret;
}
/* Point to the memory malloc'd by db_create() */
*dbpp = dbp;
db_flags = DB_CREATE | /* Create the database if it does
not exist */
DB_AUTO_COMMIT; /* Allow autocommit */
ret = dbp->open(dbp, /* Pointer to the database */
0, /* Txn pointer */
file_name, /* File name */
0, /* Logical db name */
DB_BTREE, /* Database type (using btree) */
db_flags, /* Open flags */
0); /* File mode. Using defaults */
if (ret != 0) {
dbp->err(dbp, ret, "Database open failed: %s : %s\n",
file_name, db_strerror(ret));
return ret;
}
return 0;
} </pre>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="warm_cache"></a>The warm_cache() function</h3>
</div>
</div>
</div>
<p>
In this section we provide the implementation of the
<code class="methodname">warm_cache()</code> function. This example
function simply loads all the database pages that will fit into
the memory pool. It starts from the first database page and
continues until it either runs out of database pages or it runs
out of room in the memory pool.
</p>
<a id="prog_mp01-7"></a>
<pre class="programlisting">/* Warm the cache */
int
warm_cache(DB *dbp, int *pagecountp, int numcachepages)
{
DB_MPOOLFILE *mpf = 0;
void *page_addrp = 0;
db_pgno_t page_number = 0;
int ret = 0;
int pagecount = *pagecountp;
/*
* Get the mpool handle
*/
mpf = dbp->get_mpf(dbp);
/* Load pages until there are no more pages in the database,
* or until we've put as many pages into the cache as will fit.
*/
while (ret != DB_PAGE_NOTFOUND && pagecount < numcachepages) {
/*
* Get the page from the cache. This causes DB to retrieve
* the page from disk if it isn't already in the cache.
*/
ret = mpf->get(mpf, &page_number, 0, 0, &page_addrp);
if (ret && ret != DB_PAGE_NOTFOUND) {
fprintf(stderr, "Error retrieving db page: %i : %s\n",
page_number, db_strerror(ret));
return ret;
}
/*
* If a page was retrieved, put it back into the cache. This
* releases the page latch so that the page can be evicted
* if DB needs more room in the cache at some later time.
*/
if (ret != DB_PAGE_NOTFOUND) {
ret = mpf->put(mpf, page_addrp, DB_PRIORITY_UNCHANGED, 0);
if (ret) {
fprintf(stderr, "Error putting db page: %i : %s\n",
page_number, db_strerror(ret));
return ret;
}
}
++page_number;
++pagecount;
*pagecountp = pagecount;
}
return 0;
} </pre>
</div>
</div>
<div class="navfooter">
<hr />
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left"><a accesskey="p" href="mp_config.html">Prev</a> </td>
<td width="20%" align="center">
<a accesskey="u" href="mp.html">Up</a>
</td>
<td width="40%" align="right"> <a accesskey="n" href="txn.html">Next</a></td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Configuring the memory pool </td>
<td width="20%" align="center">
<a accesskey="h" href="index.html">Home</a>
</td>
<td width="40%" align="right" valign="top"> Chapter 19.
The Transaction Subsystem
</td>
</tr>
</table>
</div>
</body>
</html>
|
