1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.21">
<TITLE>SQUID Frequently Asked Questions: DISKD</TITLE>
<LINK HREF="FAQ-23.html" REL=next>
<LINK HREF="FAQ-21.html" REL=previous>
<LINK HREF="FAQ.html#toc22" REL=contents>
</HEAD>
<BODY>
<A HREF="FAQ-23.html">Next</A>
<A HREF="FAQ-21.html">Previous</A>
<A HREF="FAQ.html#toc22">Contents</A>
<HR>
<H2><A NAME="s22">22.</A> <A HREF="FAQ.html#toc22">DISKD</A></H2>
<H2><A NAME="ss22.1">22.1</A> <A HREF="FAQ.html#toc22.1">What is DISKD?</A>
</H2>
<P>DISKD refers to some features in Squid-2.4 and later to improve Disk I/O
performance. The basic idea is that each <EM>cache_dir</EM> has its
own <EM>diskd</EM> child process. The diskd process performs all disk
I/O operations (open, close, read, write, unlink) for the cache_dir.
Message queues are used to send requests and responses between the
Squid and diskd processes. Shared memory is used for chunks of
data to be read and written.</P>
<H2><A NAME="ss22.2">22.2</A> <A HREF="FAQ.html#toc22.2">Does it perform better?</A>
</H2>
<P>Yes. We benchmarked Squid-2.4 with DISKD at the
<A HREF="http://polygraph.ircache.net/Results/bakeoff-2/">Second IRCache Bake-Off</A>. The results are also described
<A HREF="/Benchmarking/bakeoff-02/">here</A>. At the bakeoff, we
got 160 req/sec with diskd. Without diskd, we'd have gotten about
40 req/sec.</P>
<H2><A NAME="ss22.3">22.3</A> <A HREF="FAQ.html#toc22.3">How do I use it?</A>
</H2>
<P>You need to run Squid version
<A HREF="/Versions/v2/2.4">2.4</A> or later.
Your operating system must support message queues, and shared memory.</P>
<P>To configure Squid for DISKD, use the <EM>--enable-storeio</EM> option:
<PRE>
% ./configure --enable-storeio=diskd,ufs
</PRE>
</P>
<H2><A NAME="ss22.4">22.4</A> <A HREF="FAQ.html#toc22.4">FATAL: Unknown cache_dir type 'diskd'</A>
</H2>
<P>You didn't put <EM>diskd</EM> in the list of storeio modules as described
above. You need to run <EM>configure</EM> and and recompile Squid.</P>
<H2><A NAME="ss22.5">22.5</A> <A HREF="FAQ.html#toc22.5">If I use DISKD, do I have to wipe out my current cache?</A>
</H2>
<P>No. Diskd uses the same storage scheme as the standard "UFS"
type. It only changes how I/O is performed.</P>
<H2><A NAME="ss22.6">22.6</A> <A HREF="FAQ.html#toc22.6">How do I configure message queues?</A>
</H2>
<P>Most Unix operating systems have message queue support
by default. One way to check is to see if you have
an <EM>ipcs</EM> command.</P>
<P>However, you will likely need to increase the message
queue parameters for Squid. Message queue implementations
normally have the following parameters:
<DL>
<DT><B>MSGMNB</B><DD><P>Maximum number of bytes per message queue.</P>
<DT><B>MSGMNI</B><DD><P>Maximum number of message queue identifiers (system wide).</P>
<DT><B>MSGSEG</B><DD><P>Maximum number of message segments per queue.</P>
<DT><B>MSGSSZ</B><DD><P>Size of a message segment.</P>
<DT><B>MSGTQL</B><DD><P>Maximum number of messages (system wide).</P>
<DT><B>MSGMAX</B><DD><P>Maximum size of a whole message. On some systems you may need to
increase this limit. On other systems, you may not be able
to change it.</P>
</DL>
</P>
<P>The messages between Squid and diskd are 32 bytes for 32-bit CPUs
and 40 bytes for 64-bit CPUs. Thus, MSGSSZ should be 32 or greater.
You may want to set it to a larger value, just to be safe.</P>
<P>We'll have two queues for each <EM>cache_dir</EM> -- one in each direction.
So, MSGMNI needs to be at least two times the number of <EM>cache_dir</EM>'s.</P>
<P>I've found that 75 messages per queue is about the limit of decent performance.
If each diskd message consists of just one segment (depending on your
value of MSGSSZ), then MSGSEG should be greater than 75.</P>
<P>MSGMNB and MSGTQL affect how many messages can be in the queues
at one time. Diskd messages shouldn't be
more than 40 bytes, but let's use 64 bytes to be safe. MSGMNB
should be at least 64*75. I recommend rounding up to the nearest
power of two, or 8192.</P>
<P>MSGTQL should be at least 75 times the number of <EM>cache_dir</EM>'s
that you'll have.</P>
<H3>FreeBSD</H3>
<P>Your kernel must have
<PRE>
options SYSVMSG
</PRE>
</P>
<P>You can set the parameters in the kernel as follows. This is just
an example. Make sure the values are appropriate for your system:
<PRE>
options MSGMNB=8192 # max # of bytes in a queue
options MSGMNI=40 # number of message queue identifiers
options MSGSEG=512 # number of message segments per queue
options MSGSSZ=64 # size of a message segment
options MSGTQL=2048 # max messages in system
</PRE>
</P>
<H3>OpenBSD</H3>
<P>You can set the parameters in the kernel as follows. This is just
an example. Make sure the values are appropriate for your system:
<PRE>
option MSGMNB=16384 # max characters per message queue
option MSGMNI=40 # max number of message queue identifiers
option MSGSEG=2048 # max number of message segments in the system
option MSGSSZ=64 # size of a message segment (Must be 2^N)
option MSGTQL=1024 # max amount of messages in the system
</PRE>
</P>
<H3>Digital Unix</H3>
<P>Message queue support seems to be in the kernel
by default. Setting the options is as follows:
<PRE>
options MSGMNB="8192" # max # bytes on queue
options MSGMNI="40" # # of message queue identifiers
options MSGMAX="2048" # max message size
options MSGTQL="2048" # # of system message headers
</PRE>
</P>
<P>by
<A HREF="mailto:B.C.Phillips at massey dot ac dot nz">Brenden Phillips</A></P>
<P>If you have a newer version (DU64), then you can probably use
<EM>sysconfig</EM> instead. To see what the current IPC settings are run
<PRE>
# sysconfig -q ipc
</PRE>
To change them make a file like this called ipc.stanza:
<PRE>
ipc:
msg-max = 2048
msg-mni = 40
msg-tql = 2048
msg-mnb = 8192
</PRE>
then run
<PRE>
# sysconfigdb -a -f ipc.stanza
</PRE>
You have to reboot for the change to take effect.</P>
<H3>Linux</H3>
<P>Stefan Köpsell reports that if you compile sysctl support
into your kernel, then you can change the following values:
<UL>
<LI>kernel.msgmnb</LI>
<LI>kernel.msgmni</LI>
<LI>kernel.msgmax</LI>
</UL>
</P>
<P>Winfried Truemper reports: The default values should be large enough for most common cases.
You can modify the message queue configuration by writing to these files:
<UL>
<LI>/proc/sys/kernel/msgmax</LI>
<LI>/proc/sys/kernel/msgmnb</LI>
<LI>/proc/sys/kernel/msgmni</LI>
</UL>
</P>
<H3>Solaris</H3>
<P>Refer to
<A HREF="http://www.sunworld.com/sunworldonline/swol-11-1997/swol-11-insidesolaris.html">Demangling Message Queues</A> in Sunworld Magazine.</P>
<P>I don't think the above article really tells you how to set the parameters.
You do it in <EM>/etc/system</EM> with lines like this:
<PRE>
set msgsys:msginfo_msgmax=2048
set msgsys:msginfo_msgmnb=8192
set msgsys:msginfo_msgmni=40
set msgsys:msginfo_msgssz=64
set msgsys:msginfo_msgtql=2048
</PRE>
</P>
<P>Of course, you must reboot whenever you modify <EM>/etc/system</EM>
before changes take effect.</P>
<H2><A NAME="ss22.7">22.7</A> <A HREF="FAQ.html#toc22.7">How do I configure shared memory?</A>
</H2>
<P>Shared memory uses a set of parameters similar to the ones for message
queues. The Squid DISKD implementation uses one shared memory area
for each cache_dir. Each shared memory area is about
800 kilobytes in size. You may need to modify your system's
shared memory parameters:</P>
<P>
<DL>
<DT><B>SHMSEG</B><DD><P>Maximum number of shared memory segments per process.</P>
<DT><B>SHMMNI</B><DD><P>Maximum number of shared memory segments for the whole system.</P>
<DT><B>SHMMAX</B><DD><P>Largest shared memory segment size allowed.</P>
<DT><B>SHMALL</B><DD><P>Total amount of shared memory that can be used.</P>
</DL>
</P>
<P>For Squid and DISKD, <EM>SHMMNI</EM> and <EM>SHMMNI</EM> must be greater than
or equal to the number of <EM>cache_dir</EM>'s that you have. <EM>SHMMAX</EM>
must be at least 800 kilobytes. <EM>SHMALL</EM> must be at least
<EM>SHMMAX</EM> 800 kilobytes multiplied by the number of <EM>cache_dir</EM>'s.</P>
<H3>FreeBSD</H3>
<P>Your kernel must have
<PRE>
options SYSVSHM
</PRE>
</P>
<P>You can set the parameters in the kernel as follows. This is just
an example. Make sure the values are appropriate for your system:
<PRE>
options SHMSEG=16 # max shared mem id's per process
options SHMMNI=32 # max shared mem id's per system
options SHMMAX=2097152 # max shared memory segment size (bytes)
options SHMALL=4096 # max amount of shared memory (pages)
</PRE>
</P>
<H3>OpenBSD</H3>
<P>OpenBSD is similar to FreeBSD, except you must use <EM>option</EM> instead
of <EM>options</EM>, and SHMMAX is in pages instead of bytes:
<PRE>
option SHMSEG=16 # max shared mem id's per process
option SHMMNI=32 # max shared mem id's per system
option SHMMAX=2048 # max shared memory segment size (pages)
option SHMALL=4096 # max amount of shared memory (pages)
</PRE>
</P>
<H3>Digital Unix</H3>
<P>Message queue support seems to be in the kernel
by default. Setting the options is as follows:
<PRE>
options SHMSEG="16" # max shared mem id's per process
options SHMMNI="32" # max shared mem id's per system
options SHMMAX="2097152" # max shared memory segment size (bytes)
options SHMALL=4096 # max amount of shared memory (pages)
</PRE>
</P>
<P>by
<A HREF="mailto:B.C.Phillips at massey dot ac dot nz">Brenden Phillips</A></P>
<P>If you have a newer version (DU64), then you can probably use
<EM>sysconfig</EM> instead. To see what the current IPC settings are run
<PRE>
# sysconfig -q ipc
</PRE>
To change them make a file like this called ipc.stanza:
<PRE>
ipc:
shm-seg = 16
shm-mni = 32
shm-max = 2097152
shm-all = 4096
</PRE>
then run
<PRE>
# sysconfigdb -a -f ipc.stanza
</PRE>
You have to reboot for the change to take effect.</P>
<H3>Linux</H3>
<P>Winfried Truemper reports: The default values should be large enough
for most common cases. You can modify the shared memory configuration
by writing to these files:
<UL>
<LI>/proc/sys/kernel/shmall</LI>
<LI>/proc/sys/kernel/shmmax</LI>
<LI>/proc/sys/kernel/shmmni</LI>
<LI>/proc/sys/kernel/shm-use-bigpages</LI>
</UL>
</P>
<P>Stefan Köpsell reports that if you compile sysctl support
into your kernel, then you can change the following values:
<UL>
<LI>kernel.shmall</LI>
<LI>kernel.shmmni</LI>
<LI>kernel.shmmax</LI>
</UL>
</P>
<H3>Solaris</H3>
<P>Refer to
<A HREF="http://www.sunworld.com/swol-09-1997/swol-09-insidesolaris.html">Shared memory uncovered</A>
in Sunworld Magazine.</P>
<P>To set the values, you can put these lines in <EM>/etc/system</EM>:
<PRE>
set shmsys:shminfo_shmmax=2097152
set shmsys:shminfo_shmmni=32
set shmsys:shminfo_shmseg=16
</PRE>
</P>
<H2><A NAME="ss22.8">22.8</A> <A HREF="FAQ.html#toc22.8">Sometimes shared memory and message queues aren't released when Squid exits.</A>
</H2>
<P>Yes, this is a little problem sometimes. Seems like the operating system
gets confused and doesn't always release shared memory and message
queue resources when processes exit, especially if they exit abnormally.
To fix it you can ``manually'' clear the resources with the <EM>ipcs</EM> command.
Add this command into your <EM>RunCache</EM> or <EM>squid_start</EM>
script:
<PRE>
ipcs | awk '/squid/ {printf "ipcrm -%s %s\n", $1, $2}' | /bin/sh
</PRE>
</P>
<H2><A NAME="ss22.9">22.9</A> <A HREF="FAQ.html#toc22.9">What are the Q1 and Q2 parameters?</A>
</H2>
<P>In the source code, these are called <EM>magic1</EM> and <EM>magic2</EM>.
These numbers refer to the number of oustanding requests on a message
queue. They are specified on the <EM>cache_dir</EM> option line, after
the L1 and L2 directories:
<PRE>
cache_dir diskd /cache1 1024 16 256 Q1=72 Q2=64
</PRE>
</P>
<P>If there are more than Q1 messages outstanding, then Squid will
intentionally fail to open disk files for reading and writing.
This is a load-shedding mechanism. If your cache gets really really
busy and the disks can not keep up, Squid bypasses the disks until
the load goes down again.</P>
<P>If there are more than Q2 messages outstanding, then the main Squid
process ``blocks'' for a little bit until the diskd process services
some of the messages and sends back some replies. </P>
<P>Reasonable Q1 and Q2 values are 64 and 72.
If you would rather have good hit ratio and bad response time,
set Q1 > Q2. Otherwise, if you would rather have good response
time and bad hit ratio, set Q1 < Q2.</P>
<HR>
<A HREF="FAQ-23.html">Next</A>
<A HREF="FAQ-21.html">Previous</A>
<A HREF="FAQ.html#toc22">Contents</A>
</BODY>
</HTML>
|
