1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
<title>Reducing host memory consumption</title>
</head>
<body alink="#FF0000" vlink="#55188A" link="#0000EF" bgcolor="#FFFFFF" text="#000099">
<table border="0">
<tr align="left">
<td valign="top">
<table border="0">
<tr align="left"><td valign="top" >
<img src="http://user-mode-linux.sourceforge.net/uml-small.png" height="171" width="120"/>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/index.html">Site Home Page</a></font>
<br>
<font size="-1"><a href="http://uml.harlowhill.com">The UML Wiki</a></font>
<br>
<font size="-1"><a href="http://usermodelinux.org">UML Community Site</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/roadmap.html">The UML roadmap</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/uses.html">What it's good for</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/case-studies.html">Case Studies</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/kernel.html">Kernel Capabilities</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/dl-sf.html">Downloading it</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/run.html">Running it</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/compile.html">Compiling</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/install.html">Installation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/skas.html">Skas Mode</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/patches.html">Incremental Patches</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/tests.html">Test Suite</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/devanon.html">Host memory use</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/fs_making.html">Building filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/faq.html">Troubles</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/contrib.html">User Contributions</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/links.html">Related Links</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/todo.html">The ToDo list</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/projects.html">Projects</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/diary.html">Diary</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/thanks.html">Thanks</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/contacts.html">Contacts</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Tutorials<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/UserModeLinux-HOWTO.html">The HOWTO (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/UserModeLinux-HOWTO.txt">The HOWTO (text)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/hostfs.html">Host file access</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/input.html">Device inputs</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/shared_fs.html">Sharing filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/fs.html">Creating filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/resize.html">Resizing filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/networking.html">Virtual Networking</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/mconsole.html">Management Console</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/debugging.html">Kernel Debugging</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/honeypots.html">UML Honeypots</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/gprof.html">gprof and gcov</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/xtut.html">Running X</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/trouble.html">Diagnosing problems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/config.html">Configuration</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slack_readme.html">Installing Slackware</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/arch-port.html">Porting UML</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/iomem.html">IO memory emulation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/2G-2G.html">UML on 2G/2G hosts</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/lksct/index.html">Adding a UML system call</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/nesting.html">Running nested UMLs</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">How you can help<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-gen.html">Overview</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-doc.html">Documentation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-userspace.html">Utilities</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-kernel-v1.html">Kernel bugs</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/projects.html">Kernel projects</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Screenshots<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/net.html">A virtual network</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/x.html">An X session</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Transcripts<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/login.html">A login session</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/debug-session.html">A debugging session</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slackinst.html">Slackware installation</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Reference<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/switches.html">Kernel switches</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slack_readme.html">Slackware README</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Papers<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2000/index.html">ALS 2000 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2000.tex">ALS 2000 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/als2000/slides.html">ALS 2000 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lca2001/lca.html">LCA 2001 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ols2001/index.html">OLS 2001 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/ols2001.tex">OLS 2001 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2001/index.html">ALS 2001 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2001.tex">ALS 2001 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ists2002/umlsec.htm">UML security (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lca2002/lca2002.htm">LCA 2002 (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/wvu2002/wvu2002.htm">WVU 2002 (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ists_rt/ists_rt.htm">Security Roundtable (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ols2002/ols2002.html">OLS 2002 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lwe2005/LWE2005.html">LWE 2005 slides</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Fun and Games<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/cgi-bin/hangman">Kernel Hangman</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/sdotm.html">Disaster of the Month</a></font>
</td></tr>
</table>
</td>
<td valign="top" align="left">
<center>
<h3>Reducing host memory consumption</h3>
</center>
<a name="The problem"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">The problem</font>
</b>
</td>
</tr>
</table>
<blockquote head="The problem">
When you start UML, it will create a file whose size is the same as
the UML physical memory size, which is whatever you specified with the
'mem=' switch or its default. Initially, none of that is backed with
real host memory, but that changes as it touches its physical memory.
Its host memory consumption will asymptotically approach its physical
memory limit as it reads data into cache. That consumption will not
tend to decrease for two reasons
<ul>
<li>
UML, like any other Linux kernel, will keep data in cache as long as
there is no pressing need for memory. This makes sense for a native
kernel running on hardware since there is no other possible use for
the memory. However, for UML, the host quite possibly could make
better use of any under-used memory.
</li>
<li>
Even if UML did release a bunch of cache, it has no way of
communicating that to the host, which will still see that memory as
being dirty, and needing preservation, by swapping it if necessary.
</li>
</ul>
<p>
Another problem is the unnecessary duplication of data in host
memory. If you have a number, say 'n', of UMLs booted from similar
disk images, the data that's the same is present 2 * n times in the
host memory, n times in the host page cache, and n times in the UML
page caches.
<p>
Booting those UMLs from the same image with COW files will reduce the
number of copies in the host page cache to 1, but there will still be
n copies in the UML page caches, one in each UML. These copies can be
eliminated with the use of ubd=mmap, which causes the ubd driver to
mmap pages from disk images rather than using read() and write().
This causes UML to share page cache with the host.
<p>
In order to see any reduction in host memory use, it is necessary to
free the UML phsyical memory pages that were over-mapped by the ubd
driver. If these are mapped in from tmpfs or any other filesystem,
they won't be freed since filesystems preserve data even when the data
isn't mapped in anywhere. So, this is where the host /dev/anon driver
comes in. It has the property that when a page is mapped in, and
subsequently unmapped, the page will be freed back to the host, and
any data that was stored there will be lost. Obviously, this is wrong
for a filesystem, but this is exactly what's needed for UML.
</blockquote>
<a name="/dev/anon"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">/dev/anon</font>
</b>
</td>
</tr>
</table>
<blockquote head="/dev/anon">
To have UML use /dev/anon, you need to do the following:
<ul>
<li>
apply the devanon patch (available from the
<a href="dl-sf.html#Host devanon patches">download
page</a>) to your host kernel
</li>
<li>
make /dev/anon a character device with major number 1 and minor 10
<blockquote>
<tt>
<font color="#000000"> mknod /dev/anon c 1 10</font>
</tt>
</blockquote>
with read-write permissions for everyone (or at least everyone who
will be running UML)
<blockquote>
<tt>
<font color="#000000"> chmod 666 /dev/anon</font>
</tt>
</blockquote>
</li>
<li>
get a UML that supports /dev/anon (2.4.23-2 or later)
</li>
<li>
run it with the 'ubd=mmap' switch
</li>
<li>
make sure that the UML filesystems have 4K (one page) blocksize
</li>
</ul>
<b>NOTE:</b> At this point, 'ubd=mmap' is known to eat filesystems, so
don't try this yet with data that you care about. If you use COW
files, the backing files are safe since they're opened read-only, but
any data that UML has write access to is at risk from 'ubd=mmap'.
<p>
Use of the 'ubd=mmap' switch is needed in order to get any memory use
benefits from /dev/anon. However, UML will use it regardless of
whether the ubd driver is doing mmap. This is almost the same as
having it use files in /tmp for its physical memory, except that the
UMLs won't be limited to the capacity of /tmp. This makes the host
management somewhat easier. Without /dev/anon, you need tmpfs mounted
on /tmp for optimal UML performance. With it, you get the same
performance without needing to make sure that your tmpfs mount is big
enough for the UMLs running on the system.
</blockquote>
<a name="Memory savings"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Memory savings</font>
</b>
</td>
</tr>
</table>
<blockquote head="Memory savings">
In my testing, which involves booting a number of instances on a
Debian image to a login prompt, the combination of /dev/anon and
'ubd=mmap' results in about a 25% decrease in host memory
consumption. This was measured by looking at the decrease in free
memory per UML when the host was not under memory pressure. Without
'ubd=mmap', this is ~28M. With 'ubd=mmap', this went down to ~21M. I
checked this by counting the number of instances I could boot before
the host went into swap. This increased from 16 to 20, again a 25%
increase.
<p>
This is obviously workload-dependent. Workloads that involve lots of
readonly data, such as code, will benefit more than those that involve
read-write data.
</blockquote>
<a name="Performance of 'ubd=mmap'"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Performance of 'ubd=mmap'</font>
</b>
</td>
</tr>
</table>
<blockquote head="Performance of 'ubd=mmap'">
People commonly ask if switching the ubd driver from using read() and
write() (and the data copying they do) to using mmap (and the tlb
flushing that it does) will help the performance of a single UML. The
answer is that the performance is probably about the same either way.
The reason is that mmap is an expensive operation for the CPU. People
have measured how large data copies have to be in order to be more
expensive than playing games with mmap.
<p>
The size that I've seen is about 8K, or two pages on x86. Since UML
is doing maps with 1 page granularity, which is in the ballpark, you'd
expect the performance to be about the same. My testing, which has
mostly been kernel builds, bears this out. The speed is pretty close,
to within the error, either way.
</blockquote>
<a name="Future work"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Future work</font>
</b>
</td>
</tr>
</table>
<blockquote head="Future work">
This is leading to more flexibility in managing the memory consumption
of UMLs and the host. The next step is to allow the host to add and
remove memory from UMLs at run-time, i.e. hot-plug memory.
<p>
This will allow a utility on the host to keep track of the memory
usage of the UMLs and the host, and shuffle memory between the UMLs
and to and from the host in order to implement whatever policy is
desired.
<p>
I am planning a shipping a sample utility which implements the
host-should-not-swap policy. This would keep track of the free memory
on the host, and, when it gets low, it would remove memory from an
idle UML and release it to the host. When the host has ample memory,
it would look for a UML that's under memory pressure and give it some
memory.
<p>
It is also possible to use this facility to reverse the monotonic
increase in UML memory consumption described above. By pulling memory
out of a UML until it squeals, you will force it to drop its page
cache, resetting its host memory consumption to what it was much
earlier in its life. Doing this periodically to all the UMLs on the
host will result in their cached data being much more up-to-date, at
the expense of some performance loss from old cache data not being
available and needing to be reread.
<p>
These are probably a decent start for any UML site, and might be
sufficient for many, but some may need something different. For
example, a UML ISP might have different classes of UMLs on the same
host, and would want the expensive UMLs to have preference over the
cheap ones for whatever memory is available. UML will provide
mechanism, plus an example policy or two, and it will be up to the
user community to implement whatever policies are appropriate for host
memory management.
</blockquote>
</td>
</tr>
</table>
<center>
<font size="-1">Hosted at </font>
<a href="http://sourceforge.net">
<img src="http://sourceforge.net/sflogo.php?group_id=429" width="88" height="31" border="0" alt="SourceForge Logo">
</a>
</center>
</body>
</html>
|
