1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<META NAME="GENERATOR" CONTENT="LinuxDoc-Tools 0.9.21">
<TITLE>SQUID Frequently Asked Questions: Squid Log Files</TITLE>
<LINK HREF="FAQ-7.html" REL=next>
<LINK HREF="FAQ-5.html" REL=previous>
<LINK HREF="FAQ.html#toc6" REL=contents>
</HEAD>
<BODY>
<A HREF="FAQ-7.html">Next</A>
<A HREF="FAQ-5.html">Previous</A>
<A HREF="FAQ.html#toc6">Contents</A>
<HR>
<H2><A NAME="s6">6.</A> <A HREF="FAQ.html#toc6">Squid Log Files</A></H2>
<P>The logs are a valuable source of information about Squid workloads and
performance. The logs record not only access information, but also system
configuration errors and resource consumption (eg, memory, disk
space). There are several log file maintained by Squid. Some have to be
explicitely activated during compile time, others can safely be deactivated
during run-time.</P>
<P>There are a few basic points common to all log files. The time stamps
logged into the log files are usually UTC seconds unless stated otherwise.
The initial time stamp usually contains a millisecond extension.</P>
<H2><A NAME="ss6.1">6.1</A> <A HREF="FAQ.html#toc6.1"><EM>squid.out</EM></A>
</H2>
<P>If you run your Squid from the <EM>RunCache</EM> script, a file
<EM>squid.out</EM> contains the Squid startup times, and also all fatal
errors, e.g. as produced by an <EM>assert()</EM> failure. If you are not
using <EM>RunCache</EM>, you will not see such a file.</P>
<H2><A NAME="ss6.2">6.2</A> <A HREF="FAQ.html#toc6.2"><EM>cache.log</EM></A>
</H2>
<P>The <EM>cache.log</EM> file contains the debug and error messages that Squid
generates. If you start your Squid using the default <EM>RunCache</EM> script,
or start it with the <EM>-s</EM> command line option, a copy of certain
messages will go into your syslog facilities. It is a matter of personal
preferences to use a separate file for the squid log data.</P>
<P>From the area of automatic log file analysis, the <EM>cache.log</EM> file does
not have much to offer. You will usually look into this file for automated
error reports, when programming Squid, testing new features, or searching
for reasons of a perceived misbehaviour, etc.</P>
<H2><A NAME="ss6.3">6.3</A> <A HREF="FAQ.html#toc6.3"><EM>useragent.log</EM></A>
</H2>
<P>The user agent log file is only maintained, if </P>
<P>
<OL>
<LI>you configured the compile time <EM>--enable-useragent-log</EM>
option, and</LI>
<LI>you pointed the <EM>useragent_log</EM> configuration option to a
file.</LI>
</OL>
</P>
<P>From the user agent log file you are able to find out about distributation
of browsers of your clients. Using this option in conjunction with a loaded
production squid might not be the best of all ideas.</P>
<H2><A NAME="ss6.4">6.4</A> <A HREF="FAQ.html#toc6.4"><EM>store.log</EM></A>
</H2>
<P>The <EM>store.log</EM> file covers the objects currently kept on disk or
removed ones. As a kind of transaction log it is ususally used for
debugging purposes. A definitive statement, whether an object resides on
your disks is only possible after analysing the <EM>complete</EM> log file.
The release (deletion) of an object may be logged at a later time than the
swap out (save to disk).</P>
<P>The <EM>store.log</EM> file may be of interest to log file analysis which
looks into the objects on your disks and the time they spend there, or how
many times a hot object was accessed. The latter may be covered by another
log file, too. With knowledge of the <EM>cache_dir</EM> configuration option,
this log file allows for a URL to filename mapping without recursing your
cache disks. However, the Squid developers recommend to treat
<EM>store.log</EM> primarily as a debug file, and so should you, unless you
know what you are doing.</P>
<P>The print format for a store log entry (one line) consists of eleven
space-separated columns, compare with the <EM>storeLog()</EM> function in file
<EM>src/store_log.c</EM>:</P>
<P>
<PRE>
"%9d.%03d %-7s %02d %08X %4d %9d %9d %9d %s %d/%d %s %s\n"
</PRE>
</P>
<P>
<DL>
<DT><B>time</B><DD><P>The timestamp when the line was logged in UTC with a millisecond fraction.</P>
<DT><B>action</B><DD><P>The action the object was sumitted to, compare with <EM>src/store_log.c</EM>:</P>
<P>
<UL>
<LI><B>CREATE</B> Seems to be unused.</LI>
<LI><B>RELEASE</B> The object was removed from the cache (see also
<A HREF="#log-fileno">file number</A>). </LI>
<LI><B>SWAPOUT</B> The object was saved to disk.</LI>
<LI><B>SWAPIN</B> The object existed on disk and was read into memory.</LI>
</UL>
</P>
<DT><B>dir numer</B><DD><P>
<A NAME="log-dirno"></A> </P>
<P>The cache_dir number this object was stored into, starting at 0 for your first
cache_dir line.</P>
<DT><B>file number</B><DD><P>
<A NAME="log-fileno"></A> </P>
<P>The file number for the object storage file. Please note that the path to
this file is calculated according to your <EM>cache_dir</EM> configuration.</P>
<P>A file number of <EM>FFFFFFFF</EM> denominates "memory only" objects. Any
action code for such a file number refers to an object which existed only
in memory, not on disk. For instance, if a <EM>RELEASE</EM> code was logged
with file number <EM>FFFFFFFF</EM>, the object existed only in memory, and was
released from memory.</P>
<DT><B>status</B><DD><P>The HTTP reply status code.</P>
<DT><B>datehdr</B><DD><P>
<A NAME="log-Date"></A> </P>
<P>The value of the HTTP "Date: " reply header.</P>
<DT><B>lastmod
<A NAME="log-LM"></A> </B><DD><P>The value of the HTTP "Last-Modified: " reply header.</P>
<DT><B>expires</B><DD><P>
<A NAME="log-Expires"></A> </P>
<P>The value of the HTTP "Expires: " reply header.</P>
<DT><B>type</B><DD><P>The HTTP "Content-Type" major value, or "unknown" if it cannot be
determined.</P>
<DT><B>sizes</B><DD><P>This column consists of two slash separated fields:</P>
<P>
<OL>
<LI>The advertised content length from the HTTP "Content-Length: " reply
header.</LI>
<LI>The size actually read.</LI>
</OL>
</P>
<P>If the advertised (or expected) length is missing, it will be set to
zero. If the advertised length is not zero, but not equal to the real
length, the object will be realeased from the cache.</P>
<DT><B>method</B><DD><P>The request method for the object, e.g. <EM>GET</EM>.</P>
<DT><B>key</B><DD><P>
<A NAME="log-key"></A> </P>
<P>The key to the object, usually the URL.</P>
</DL>
</P>
<P>The timestamp format for the columns
<A HREF="#log-Date">Date</A> to
<A HREF="#log-Expires">Expires</A> are all expressed in UTC seconds. The
actual values are parsed from the HTTP reply headers. An unparsable header
is represented by a value of -1, and a missing header is represented by a
value of -2.</P>
<P>The column
<A HREF="#log-key">key</A> usually contains just the URL of
the object. Some objects though will never become public. Thus the key is
said to include a unique integer number and the request method in addition
to the URL.</P>
<H2><A NAME="ss6.5">6.5</A> <A HREF="FAQ.html#toc6.5"><EM>hierarchy.log</EM></A>
</H2>
<P>This logfile exists for Squid-1.0 only. The format is
<PRE>
[date] URL peerstatus peerhost
</PRE>
</P>
<H2><A NAME="ss6.6">6.6</A> <A HREF="FAQ.html#toc6.6"><EM>access.log</EM></A>
</H2>
<P>Most log file analysis program are based on the entries in
<EM>access.log</EM>. Currently, there are two file formats possible for the log
file, depending on your configuration for the <EM>emulate_httpd_log</EM>
option. By default, Squid will log in its native log file format. If the
above option is enabled, Squid will log in the common log file format as
defined by the CERN web daemon.</P>
<P>The common log file format contains other information than the native log
file, and less. The native format contains more information for the admin
interested in cache evaluation.</P>
<H3><EM>The common log file format</EM></H3>
<P>The
<A HREF="http://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format">Common Logfile Format</A>
is used by numerous HTTP servers. This format consists of the following
seven fields:
<PRE>
remotehost rfc931 authuser [date] "method URL" status bytes
</PRE>
</P>
<P>It is parsable by a variety of tools. The common format contains different
information than the native log file format. The HTTP version is logged,
which is not logged in native log file format.</P>
<H3><EM>The native log file format</EM></H3>
<P>The native format is different for different major versions of Squid. For
Squid-1.0 it is:
<PRE>
time elapsed remotehost code/status/peerstatus bytes method URL
</PRE>
</P>
<P>For Squid-1.1, the information from the <EM>hierarchy.log</EM> was moved
into <EM>access.log</EM>. The format is:
<PRE>
time elapsed remotehost code/status bytes method URL rfc931 peerstatus/peerhost type
</PRE>
</P>
<P>For Squid-2 the columns stay the same, though the content within may change
a little.</P>
<P>The native log file format logs more and different information than the
common log file format: the request duration, some timeout information,
the next upstream server address, and the content type.</P>
<P>There exist tools, which convert one file format into the other. Please
mind that even though the log formats share most information, both formats
contain information which is not part of the other format, and thus this
part of the information is lost when converting. Especially converting back
and forth is not possible without loss.</P>
<P><EM>squid2common.pl</EM> is a conversion utility, which converts any of the
squid log file formats into the old CERN proxy style output. There exist
tools to analyse, evaluate and graph results from that format.</P>
<H3><EM>access.log native format in detail</EM></H3>
<P>It is recommended though to use Squid's native log format due to its
greater amount of information made available for later analysis. The print
format line for native <EM>access.log</EM> entries looks like this:</P>
<P>
<PRE>
"%9d.%03d %6d %s %s/%03d %d %s %s %s %s%s/%s %s"
</PRE>
</P>
<P>Therefore, an <EM>access.log</EM> entry usually consists of (at least) 10
columns separated by one ore more spaces:</P>
<P>
<DL>
<DT><B>time</B><DD><P>A Unix timestamp as UTC seconds with a millisecond resolution. You
can convert Unix timestamps into something more human readable using
this short perl script:
<PRE>
#! /usr/bin/perl -p
s/^\d+\.\d+/localtime $&/e;
</PRE>
</P>
<DT><B>duration</B><DD><P>The elapsed time considers how many milliseconds the transaction
busied the cache. It differs in interpretation between TCP and UDP:</P>
<P>
<UL>
<LI>For HTTP/1.0, this is basically the time between <EM>accept()</EM>
and <EM>close()</EM>.</LI>
<LI>For persistent connections, this ought to be the time between
scheduling the reply and finishing sending it.</LI>
<LI>For ICP, this is the time between scheduling a reply and actually
sending it.</LI>
</UL>
</P>
<P>Please note that the entries are logged <EM>after</EM> the reply finished
being sent, <EM>not</EM> during the lifetime of the transaction.</P>
<DT><B>client address</B><DD><P>The IP address of the requesting instance, the client IP address. The
<EM>client_netmask</EM> configuration option can distort the clients for data
protection reasons, but it makes analysis more difficult. Often it is
better to use one of the log file anonymizers.</P>
<P>Also, the <EM>log_fqdn</EM> configuration option may log the fully qualified
domain name of the client instead of the dotted quad. The use of that
option is discouraged due to its performance impact.</P>
<DT><B>result codes</B><DD><P>
<A NAME="log-resultcode"></A> </P>
<P>This column is made up of two entries separated by a slash. This column
encodes the transaction result:</P>
<P>
<OL>
<LI>The cache result of the request contains information on the kind of
request, how it was satisfied, or in what way it failed. Please refer
to section
<A HREF="#cache-result-codes">Squid result codes</A>
for valid symbolic result codes.
<P>Several codes from older versions are no longer available, were
renamed, or split. Especially the <EM>ERR_</EM> codes do not seem to
appear in the log file any more. Also refer to section
<A HREF="#cache-result-codes">Squid result codes</A> for details
on the codes no longer available in Squid-2.</P>
<P>The NOVM versions and Squid-2 also rely on the Unix buffer cache, thus
you will see less <EM>TCP_MEM_HIT</EM>s than with a Squid-1.
Basically, the NOVM feature relies on <EM>read()</EM> to obtain an
object, but due to the kernel buffer cache, no disk activity is needed.
Only small objects (below 8KByte) are kept in Squid's part of main
memory.</P>
</LI>
<LI>The status part contains the HTTP result codes with some Squid specific
extensions. Squid uses a subset of the RFC defined error codes for
HTTP. Refer to section
<A HREF="#http-status-codes">status codes</A>
for details of the status codes recognized by a Squid-2.</LI>
</OL>
</P>
<DT><B>bytes</B><DD><P>The size is the amount of data delivered to the client. Mind that this does
not constitute the net object size, as headers are also counted. Also,
failed requests may deliver an error page, the size of which is also logged
here.</P>
<DT><B>request method</B><DD><P>The request method to obtain an object. Please refer to section
<A HREF="#request-methods">request-methods</A>
for available methods.
If you turned off <EM>log_icp_queries</EM> in your configuration, you
will not see (and thus unable to analyse) ICP exchanges. The <EM>PURGE</EM>
method is only available, if you have an ACL for ``method purge'' enabled
in your configuration file.</P>
<DT><B>URL</B><DD><P>This column contains the URL requested. Please note that the log file
may contain whitespaces for the URI. The default configuration for
<EM>uri_whitespace</EM> denies whitespaces, though.</P>
<DT><B>rfc931</B><DD><P>The eigth column may contain the ident lookups for the requesting
client. Since ident lookups have performance impact, the default
configuration turns <EM>ident_loookups</EM> off. If turned off, or no ident
information is available, a ``-'' will be logged.</P>
<DT><B>hierarchy code</B><DD><P>The hierarchy information consists of three items:</P>
<P>
<OL>
<LI>Any hierarchy tag may be prefixed with <EM>TIMEOUT_</EM>, if the
timeout occurs waiting for all ICP replies to return from the
neighbours. The timeout is either dynamic, if the
<EM>icp_query_timeout</EM> was not set, or the time configured there
has run up.</LI>
<LI>A code that explains how the request was handled, e.g. by
forwarding it to a peer, or going straight to the source. Refer to
section
<A HREF="#hier-codes">hier-codes</A>
for details on hierarchy codes and
removed hierarchy codes.</LI>
<LI>The IP address or hostname where the request (if a miss) was forwarded.
For requests sent to origin servers, this is the origin server's IP address.
For requests sent to a neighbor cache, this is the neighbor's hostname.
NOTE: older versions of Squid would put the origin server hostname here.</LI>
</OL>
</P>
<DT><B>type</B><DD><P>The content type of the object as seen in the HTTP reply
header. Please note that ICP exchanges usually don't have any content
type, and thus are logged ``-''. Also, some weird replies have content
types ``:'' or even empty ones.</P>
</DL>
</P>
<P>There may be two more columns in the <EM>access.log</EM>, if the (debug) option
<EM>log_mime_headers</EM> is enabled In this case, the HTTP request headers are
logged between a ``['' and a ``]'', and the HTTP reply headers are also
logged between ``['' and ``]''. All control characters like CR and LF are
URL-escaped, but spaces are <EM>not</EM> escaped! Parsers should watch out for
this.</P>
<H2><A NAME="cache-result-codes"></A> <A NAME="ss6.7">6.7</A> <A HREF="FAQ.html#toc6.7">Squid result codes</A>
</H2>
<P>The <B>TCP_</B> codes refer to requests on the HTTP port (usually 3128). The
<B>UDP_</B> codes refer to requests on the ICP port (usually 3130). If
ICP logging was disabled using the <EM>log_icp_queries</EM> option, no ICP
replies will be logged.</P>
<P>The following result codes were taken from a Squid-2, compare with the
<EM>log_tags</EM> struct in <EM>src/access_log.c</EM>:</P>
<P>
<DL>
<DT><B>TCP_HIT</B><DD><P>A valid copy of the requested object was in the cache.</P>
<DT><B>TCP_MISS</B><DD><P>The requested object was not in the cache.</P>
<DT><B>TCP_REFRESH_HIT</B><DD><P>The requested object was cached but <EM>STALE</EM>. The IMS query
for the object resulted in "304 not modified".</P>
<DT><B>TCP_REF_FAIL_HIT</B><DD><P>The requested object was cached but <EM>STALE</EM>. The IMS query
failed and the stale object was delivered.</P>
<DT><B>TCP_REFRESH_MISS</B><DD><P>The requested object was cached but <EM>STALE</EM>. The IMS query
returned the new content.</P>
<DT><B>TCP_CLIENT_REFRESH_MISS</B><DD><P>
<A NAME="tcp-client-refresh-miss"></A>
The client issued a "no-cache" pragma, or some analogous cache
control command along with the request. Thus, the cache has to
refetch the object.</P>
<DT><B>TCP_IMS_HIT</B><DD><P>
<A NAME="tcp-ims-hit"></A>
The client issued an IMS request for an object which was in the
cache and fresh.</P>
<DT><B>TCP_SWAPFAIL_MISS</B><DD><P>
<A NAME="tcp-swapfail-miss"></A>
The object was believed to be in the cache,
but could not be accessed.</P>
<DT><B>TCP_NEGATIVE_HIT</B><DD><P>Request for a negatively cached object,
e.g. "404 not found", for which the cache believes to know that
it is inaccessible. Also refer to the explainations for
<EM>negative_ttl</EM> in your <EM>squid.conf</EM> file.</P>
<DT><B>TCP_MEM_HIT</B><DD><P>A valid copy of the requested object was in the
cache <EM>and</EM> it was in memory, thus avoiding disk accesses.</P>
<DT><B>TCP_DENIED</B><DD><P>Access was denied for this request.</P>
<DT><B>TCP_OFFLINE_HIT</B><DD><P>The requested object was retrieved from the
cache during offline mode. The offline mode never
validates any object, see <EM>offline_mode</EM> in
<EM>squid.conf</EM> file.</P>
<DT><B>UDP_HIT</B><DD><P>A valid copy of the requested object was in the cache.</P>
<DT><B>UDP_MISS</B><DD><P>The requested object is not in this cache.</P>
<DT><B>UDP_DENIED</B><DD><P>Access was denied for this request.</P>
<DT><B>UDP_INVALID</B><DD><P>An invalid request was received.</P>
<DT><B>UDP_MISS_NOFETCH</B><DD><P>
<A NAME="udp-miss-nofetch"></A>
During "-Y" startup, or during frequent
failures, a cache in hit only mode will return either UDP_HIT or
this code. Neighbours will thus only fetch hits.</P>
<DT><B>NONE</B><DD><P>Seen with errors and cachemgr requests.</P>
</DL>
</P>
<P>The following codes are no longer available in Squid-2:</P>
<P>
<DL>
<DT><B>ERR_*</B><DD><P>Errors are now contained in the status code.</P>
<DT><B>TCP_CLIENT_REFRESH</B><DD><P>See:
<A HREF="#tcp-client-refresh-miss">TCP_CLIENT_REFRESH_MISS</A>.</P>
<DT><B>TCP_SWAPFAIL</B><DD><P>See:
<A HREF="#tcp-swapfail-miss">TCP_SWAPFAIL_MISS</A>.</P>
<DT><B>TCP_IMS_MISS</B><DD><P>Deleted,
<A HREF="#tcp-ims-hit">TCP_IMS_HIT</A> used instead.</P>
<DT><B>UDP_HIT_OBJ</B><DD><P>Hit objects are no longer available.</P>
<DT><B>UDP_RELOADING</B><DD><P>See:
<A HREF="#udp-miss-nofetch">UDP_MISS_NOFETCH</A>.</P>
</DL>
</P>
<H2><A NAME="http-status-codes"></A> <A NAME="ss6.8">6.8</A> <A HREF="FAQ.html#toc6.8">HTTP status codes</A>
</H2>
<P>These are taken from
<A HREF="ftp://ftp.isi.edu/in-notes/rfc2616.txt">RFC 2616</A> and verified for Squid. Squid-2 uses almost all
codes except 307 (Temporary Redirect), 416 (Request Range Not Satisfiable),
and 417 (Expectation Failed). Extra codes include 0 for a result code being
unavailable, and 600 to signal an invalid header, a proxy error. Also, some
definitions were added as for
<A HREF="ftp://ftp.isi.edu/in-notes/rfc2518.txt">RFC 2518</A> (WebDAV).
Yes, there are really two entries for status code
424, compare with <EM>http_status</EM> in <EM>src/enums.h</EM>:</P>
<P>
<PRE>
000 Used mostly with UDP traffic.
100 Continue
101 Switching Protocols
*102 Processing
200 OK
201 Created
202 Accepted
203 Non-Authoritative Information
204 No Content
205 Reset Content
206 Partial Content
*207 Multi Status
300 Multiple Choices
301 Moved Permanently
302 Moved Temporarily
303 See Other
304 Not Modified
305 Use Proxy
[307 Temporary Redirect]
400 Bad Request
401 Unauthorized
402 Payment Required
403 Forbidden
404 Not Found
405 Method Not Allowed
406 Not Acceptable
407 Proxy Authentication Required
408 Request Timeout
409 Conflict
410 Gone
411 Length Required
412 Precondition Failed
413 Request Entity Too Large
414 Request URI Too Large
415 Unsupported Media Type
[416 Request Range Not Satisfiable]
[417 Expectation Failed]
*424 Locked
*424 Failed Dependency
*433 Unprocessable Entity
500 Internal Server Error
501 Not Implemented
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout
505 HTTP Version Not Supported
*507 Insufficient Storage
600 Squid header parsing error
</PRE>
</P>
<H2><A NAME="request-methods"></A> <A NAME="ss6.9">6.9</A> <A HREF="FAQ.html#toc6.9">Request methods</A>
</H2>
<P>Squid recognizes several request methods as defined in
<A HREF="ftp://ftp.isi.edu/in-notes/rfc2616.txt">RFC 2616</A>. Newer versions of Squid (2.2.STABLE5 and above)
also recognize
<A HREF="ftp://ftp.isi.edu/in-notes/rfc2518.txt">RFC 2518</A> ``HTTP Extensions for Distributed Authoring --
WEBDAV'' extensions.</P>
<P>
<PRE>
method defined cachabil. meaning
--------- ---------- ---------- -------------------------------------------
GET HTTP/0.9 possibly object retrieval and simple searches.
HEAD HTTP/1.0 possibly metadata retrieval.
POST HTTP/1.0 CC or Exp. submit data (to a program).
PUT HTTP/1.1 never upload data (e.g. to a file).
DELETE HTTP/1.1 never remove resource (e.g. file).
TRACE HTTP/1.1 never appl. layer trace of request route.
OPTIONS HTTP/1.1 never request available comm. options.
CONNECT HTTP/1.1r3 never tunnel SSL connection.
ICP_QUERY Squid never used for ICP based exchanges.
PURGE Squid never remove object from cache.
PROPFIND rfc2518 ? retrieve properties of an object.
PROPATCH rfc2518 ? change properties of an object.
MKCOL rfc2518 never create a new collection.
COPY rfc2518 never create a duplicate of src in dst.
MOVE rfc2518 never atomically move src to dst.
LOCK rfc2518 never lock an object against modifications.
UNLOCK rfc2518 never unlock an object.
</PRE>
</P>
<H2><A NAME="hier-codes"></A> <A NAME="ss6.10">6.10</A> <A HREF="FAQ.html#toc6.10">Hierarchy Codes</A>
</H2>
<P>The following hierarchy codes are used with Squid-2:
<DL>
<DT><B>NONE</B><DD><P>For TCP HIT, TCP failures, cachemgr requests and all UDP
requests, there is no hierarchy information.</P>
<DT><B>DIRECT</B><DD><P>The object was fetched from the origin server.</P>
<DT><B>SIBLING_HIT</B><DD><P>The object was fetched from a sibling cache which replied with
UDP_HIT.</P>
<DT><B>PARENT_HIT</B><DD><P>The object was requested from a parent cache which replied with
UDP_HIT.</P>
<DT><B>DEFAULT_PARENT</B><DD><P>No ICP queries were sent. This parent was chosen because it was
marked ``default'' in the config file.</P>
<DT><B>SINGLE_PARENT</B><DD><P>The object was requested from the only parent appropriate for the
given URL.</P>
<DT><B>FIRST_UP_PARENT</B><DD><P>The object was fetched from the first parent in the list of
parents.</P>
<DT><B>NO_PARENT_DIRECT</B><DD><P>The object was fetched from the origin server, because no parents
existed for the given URL.</P>
<DT><B>FIRST_PARENT_MISS</B><DD><P>The object was fetched from the parent with the fastest (possibly
weighted) round trip time. </P>
<DT><B>CLOSEST_PARENT_MISS</B><DD><P>This parent was chosen, because it included the the lowest RTT
measurement to the origin server. See also the <EM>closests-only</EM>
peer configuration option.</P>
<DT><B>CLOSEST_PARENT</B><DD><P>The parent selection was based on our own RTT measurements.</P>
<DT><B>CLOSEST_DIRECT</B><DD><P>Our own RTT measurements returned a shorter time than any parent.</P>
<DT><B>NO_DIRECT_FAIL</B><DD><P>The object could not be requested because of a firewall
configuration, see also <EM>never_direct</EM> and related material,
and no parents were available.</P>
<DT><B>SOURCE_FASTEST</B><DD><P>The origin site was chosen, because the source ping arrived fastest.</P>
<DT><B>ROUNDROBIN_PARENT</B><DD><P>No ICP replies were received from any parent. The parent was
chosen, because it was marked for round robin in the config file
and had the lowest usage count.</P>
<DT><B>CACHE_DIGEST_HIT</B><DD><P>The peer was chosen, because the cache digest predicted a
hit. This option was later replaced in order to distinguish
between parents and siblings.</P>
<DT><B>CD_PARENT_HIT</B><DD><P>The parent was chosen, because the cache digest predicted a
hit.</P>
<DT><B>CD_SIBLING_HIT</B><DD><P>The sibling was chosen, because the cache digest predicted a
hit.</P>
<DT><B>NO_CACHE_DIGEST_DIRECT</B><DD><P>This output seems to be unused?</P>
<DT><B>CARP</B><DD><P>The peer was selected by CARP.</P>
<DT><B>ANY_PARENT</B><DD><P>part of <EM>src/peer_select.c:hier_strings[]</EM>.</P>
<DT><B>INVALID CODE</B><DD><P>part of <EM>src/peer_select.c:hier_strings[]</EM>.</P>
</DL>
</P>
<P>Almost any of these may be preceded by 'TIMEOUT_' if the two-second
(default) timeout occurs waiting for all ICP replies to arrive from
neighbors, see also the <EM>icp_query_timeout</EM> configuration option.</P>
<P>The following hierarchy codes were removed from Squid-2:
<PRE>
code meaning
-------------------- -------------------------------------------------
PARENT_UDP_HIT_OBJ hit objects are not longer available.
SIBLING_UDP_HIT_OBJ hit objects are not longer available.
SSL_PARENT_MISS SSL can now be handled by squid.
FIREWALL_IP_DIRECT No special logging for hosts inside the firewall.
LOCAL_IP_DIRECT No special logging for local networks.
</PRE>
</P>
<H2><A NAME="swaplog"></A> <A NAME="ss6.11">6.11</A> <A HREF="FAQ.html#toc6.11"><EM>cache/log</EM> (Squid-1.x)</A>
</H2>
<P>This file has a rather unfortunate name. It also is often called the
<EM>swap log</EM>. It is a record of every cache object written to disk.
It is read when Squid starts up to ``reload'' the cache. If you remove
this file when squid is NOT running, you will effectively wipe out your
cache contents. If you remove this file while squid IS running,
you can easily recreate it. The safest way is to simply shutdown
the running process:
<PRE>
% squid -k shutdown
</PRE>
This will disrupt service, but at least you will have your swap log
back.
Alternatively, you can tell squid to rotate its log files. This also
causes a clean swap log to be written.
<PRE>
% squid -k rotate
</PRE>
</P>
<P>For Squid-1.1, there are six fields:
<OL>
<LI><B>fileno</B>:
The swap file number holding the object data. This is mapped to a pathname on your filesystem.
</LI>
<LI><B>timestamp</B>:
This is the time when the object was last verified to be current. The time is a
hexadecimal representation of Unix time.
</LI>
<LI><B>expires</B>:
This is the value of the Expires header in the HTTP reply. If an Expires header
was not present, this will be -2 or fffffffe. If the Expires header was
present, but invalid (unparsable), this will be -1 or ffffffff.
</LI>
<LI><B>lastmod</B>:
Value of the HTTP reply Last-Modified header. If missing it will be -2,
if invalid it will be -1.
</LI>
<LI><B>size</B>:
Size of the object, including headers.
</LI>
<LI><B>url</B>:
The URL naming this object.
</LI>
</OL>
</P>
<H2><A NAME="ss6.12">6.12</A> <A HREF="FAQ.html#toc6.12"><EM>swap.state</EM> (Squid-2.x)</A>
</H2>
<P>In Squid-2, the swap log file is now called <EM>swap.state</EM>. This is
a binary file that includes MD5 checksums, and <EM>StoreEntry</EM> fields.
Please see the
<A HREF="../Prog-Guide/">Programmers Guide</A> for
information on the contents and format of that file.</P>
<P>If you remove <EM>swap.state</EM> while Squid is running, simply send
Squid the signal to rotate its log files:
<PRE>
% squid -k rotate
</PRE>
Alternatively, you can tell Squid to shutdown and it will
rewrite this file before it exits.</P>
<P>If you remove the <EM>swap.state</EM> while Squid is not running, you will
not lose your entire cache. In this case, Squid will scan all of
the cache directories and read each swap file to rebuild the cache.
This can take a very long time, so you'll have to be patient.</P>
<P>By default the <EM>swap.state</EM> file is stored in the top-level
of each <EM>cache_dir</EM>. You can move the logs to a different
location with the <EM>cache_swap_log</EM> option.</P>
<H2><A NAME="ss6.13">6.13</A> <A HREF="FAQ.html#toc6.13">Which log files can I delete safely?</A>
</H2>
<P>You should never delete <EM>access.log</EM>, <EM>store.log</EM>,
<EM>cache.log</EM>, or <EM>swap.state</EM> while Squid is running.
With Unix, you can delete a file when a process
has the file opened. However, the filesystem space is
not reclaimed until the process closes the file.</P>
<P>If you accidentally delete <EM>swap.state</EM> while Squid is running,
you can recover it by following the instructions in the previous
questions. If you delete the others while Squid is running,
you can not recover them.</P>
<P>The correct way to maintain your log files is with Squid's ``rotate''
feature. You should rotate your log files at least once per day.
The current log files are closed and then renamed with numeric extensions
(.0, .1, etc). If you want to, you can write your own scripts
to archive or remove the old log files. If not, Squid will
only keep up to <EM>logfile_rotate</EM> versions of each log file.
The logfile rotation procedure also writes a clean <EM>swap.state</EM>
file, but it does not leave numbered versions of the old files.</P>
<P>If you set <EM>logfile_rotate</EM> to 0, Squid simply closes and then
re-opens the logs. This allows third-party logfile management systems,
such as <EM>newsyslog</EM>, to maintain the log files.</P>
<P>To rotate Squid's logs, simple use this command:
<PRE>
squid -k rotate
</PRE>
For example, use this cron entry to rotate the logs at midnight:
<PRE>
0 0 * * * /usr/local/squid/bin/squid -k rotate
</PRE>
</P>
<H2><A NAME="ss6.14">6.14</A> <A HREF="FAQ.html#toc6.14">How can I disable Squid's log files?</A>
</H2>
<P><B>For Squid 2.4:</B></P>
<P>To disable <EM>access.log</EM>:
<PRE>
cache_access_log /dev/null
</PRE>
</P>
<P>To disable <EM>store.log</EM>:
<PRE>
cache_store_log none
</PRE>
</P>
<P>To disable <EM>cache.log</EM>:
<PRE>
cache_log /dev/null
</PRE>
</P>
<P><B>For Squid 2.5:</B></P>
<P>To disable <EM>access.log</EM>:
<PRE>
cache_access_log none
</PRE>
</P>
<P>To disable <EM>store.log</EM>:
<PRE>
cache_store_log none
</PRE>
</P>
<P>To disable <EM>cache.log</EM>:
<PRE>
cache_log /dev/null
</PRE>
</P>
<P><B>Note </B>: It is a bad idea to disable the <EM>cache.log</EM> because this
file contains many important status and debugging messages. However, if
you really want to, you can.</P>
<P><B>Warning </B>: If /dev/null is specified to any of the above log files,
<EM>logfile rotate</EM> must also be set to <EM>0</EM> or else risk Squid
rotating away /dev/null making it a plain log file.</P>
<P><B>Tip </B>: Instead of disabling the log files, it is advisable to use a
smaller value for <EM>logfile_rotate</EM> and properly rotating Squid's log
files in your cron. That way, your log files are more controllable and
self-maintained by your system.</P>
<H2><A NAME="log-large"></A> <A NAME="ss6.15">6.15</A> <A HREF="FAQ.html#toc6.15">My log files get very big!</A>
</H2>
<P>You need to <EM>rotate</EM> your log files with a cron job. For example:
<PRE>
0 0 * * * /usr/local/squid/bin/squid -k rotate
</PRE>
</P>
<H2><A NAME="ss6.16">6.16</A> <A HREF="FAQ.html#toc6.16">I want to use another tool to maintain the log files.</A>
</H2>
<P>If you set <EM>logfile_rotate</EM> to 0, Squid simply closes and then
re-opens the logs. This allows third-party logfile management systems,
such as <EM>newsyslog</EM>, to maintain the log files.</P>
<H2><A NAME="ss6.17">6.17</A> <A HREF="FAQ.html#toc6.17">Managing log files</A>
</H2>
<P>The preferred log file for analysis is the <EM>access.log</EM> file in native
format. For long term evaluations, the log file should be obtained at
regular intervals. Squid offers an easy to use API for rotating log files,
in order that they may be moved (or removed) without disturbing the cache
operations in progress. The procedures were described above.</P>
<P>Depending on the disk space allocated for log file storage, it is
recommended to set up a cron job which rotates the log files every 24, 12,
or 8 hour. You will need to set your <EM>logfile_rotate</EM> to a sufficiently
large number. During a time of some idleness, you can safely transfer the
log files to your analysis host in one burst.</P>
<P>Before transport, the log files can be compressed during off-peak time. On
the analysis host, the log file are concatinated into one file, so one file
for 24 hours is the yield. Also note that with <EM>log_icp_queries</EM>
enabled, you might have around 1 GB of uncompressed log information per day
and busy cache. Look into you cache manager info page to make an educated
guess on the size of your log files.</P>
<P>The EU project
<A HREF="http://www.desire.org/">DESIRE</A>
developed some
<A HREF="http://www.uninett.no/prosjekt/desire/arneberg/statistics.html">some basic rules</A>
to obey when handling and processing log files:</P>
<P>
<UL>
<LI>Respect the privacy of your clients when publishing results.</LI>
<LI>Keep logs unavailable unless anonymized. Most countries have laws on
privacy protection, and some even on how long you are legally allowed to
keep certain kinds of information. </LI>
<LI>Rotate and process log files at least once a day. Even if you don't
process the log files, they will grow quite large, see section
<A HREF="#log-large">log-large</A>
. If you rely on processing the log files, reserve
a large enough partition solely for log files.</LI>
<LI>Keep the size in mind when processing. It might take longer to
process log files than to generate them!</LI>
<LI>Limit yourself to the numbers you are interested in. There is data
beyond your dreams available in your log file, some quite obvious, others
by combination of different views. Here are some examples for figures to
watch:
<UL>
<LI>The hosts using your cache.</LI>
<LI>The elapsed time for HTTP requests - this is the latency the user
sees. Usually, you will want to make a distinction for HITs and MISSes
and overall times. Also, medians are preferred over averages.</LI>
<LI>The requests handled per interval (e.g. second, minute or hour). </LI>
</UL>
</LI>
</UL>
</P>
<H2><A NAME="ss6.18">6.18</A> <A HREF="FAQ.html#toc6.18">Why do I get ERR_NO_CLIENTS_BIG_OBJ messages so often?</A>
</H2>
<P>This message means that the requested object was in ``Delete Behind''
mode and the user aborted the transfer. An object will go into
``Delete Behind'' mode if
<UL>
<LI>It is larger than <EM>maximum_object_size</EM></LI>
<LI>It is being fetched from a neighbor which has the <EM>proxy-only</EM> option set.</LI>
</UL>
</P>
<H2><A NAME="ss6.19">6.19</A> <A HREF="FAQ.html#toc6.19">What does ERR_LIFETIME_EXP mean?</A>
</H2>
<P>This means that a timeout occurred while the object was being transferred. Most
likely the retrieval of this object was very slow (or it stalled before finishing)
and the user aborted the request. However, depending on your settings for
<EM>quick_abort</EM>, Squid may have continued to try retrieving the object.
Squid imposes a maximum amount of time on all open sockets, so after some amount
of time the stalled request was aborted and logged win an ERR_LIFETIME_EXP
message.</P>
<H2><A NAME="ss6.20">6.20</A> <A HREF="FAQ.html#toc6.20">Retrieving ``lost'' files from the cache</A>
</H2>
<P>
<BLOCKQUOTE>
<I>I've been asked to retrieve an object which was accidentally
destroyed at the source for recovery.
So, how do I figure out where the things are so I can copy
them out and strip off the headers?</I>
</BLOCKQUOTE>
</P>
<P>The following method applies only to the Squid-1.1 versions:</P>
<P>Use <EM>grep</EM> to find the named object (Url) in the
<A HREF="#swaplog">cache/log</A> file. The first field in
this file is an integer <EM>file number</EM>.</P>
<P>Then, find the file <EM>fileno-to-pathname.pl</EM> from the ``scripts''
directory of the Squid source distribution. The usage is
<PRE>
perl fileno-to-pathname.pl [-c squid.conf]
</PRE>
file numbers are read on stdin, and pathnames are printed on
stdout.</P>
<H2><A NAME="ss6.21">6.21</A> <A HREF="FAQ.html#toc6.21">Can I use <EM>store.log</EM> to figure out if a response was cachable?</A>
</H2>
<P>Sort of. You can use <EM>store.log</EM> to find out if a particular response
was <EM>cached</EM>.</P>
<P>Cached responses are logged with the SWAPOUT tag.
Uncached responses are logged with the RELEASE tag.</P>
<P>However, your
analysis must also consider that when a cached response is removed
from the cache (for example due to cache replacement) it is also
logged in <EM>store.log</EM> with the RELEASE tag. To differentiate these
two, you can look at the filenumber (3rd) field. When an uncachable
response is released, the filenumber is FFFFFFFF (-1). Any other
filenumber indicates a cached response was released.</P>
<HR>
<A HREF="FAQ-7.html">Next</A>
<A HREF="FAQ-5.html">Previous</A>
<A HREF="FAQ.html#toc6">Contents</A>
</BODY>
</HTML>
|
