File: grok.1

package info (click to toggle)
grok 1.20110708.1-4.3
  • links: PTS, VCS
  • area: main
  • in suites: buster
  • size: 2,460 kB
  • sloc: ansic: 3,469; ruby: 987; makefile: 276; sh: 124; yacc: 106
file content (340 lines) | stat: -rw-r--r-- 11,272 bytes parent folder | download | duplicates (5)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
.\" Automatically generated by Pod::Man 2.1801 (Pod::Simple 3.05)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.ie \nF \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    nr % 0
.    rr F
.\}
.el \{\
.    de IX
..
.\}
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "GROK 1"
.TH GROK 1 "2009-12-25" "" ""
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
grok \- parse logs, handle events, and make your unstructured text structured.
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
\&\fBgrok\fR [\fB\-d\fR] \fB\-f configfile\fR
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
Grok is software that allows you to easily parse logs and other files.
With grok, you can turn unstructured log and event data into structured data.
.PP
The grok program is a great tool for parsing log data and program output. You
can match any number of complex patterns on any number of inputs (processes and
files) and have custom reactions.
.SH "OPTIONS"
.IX Header "OPTIONS"
.IP "\fB\-d\fR or \fB\-\-daemon\fR" 4
.IX Item "-d or --daemon"
Daemonize after parsing the config file. Implemented with \fIdaemon\fR\|(3). The default is to stay in foreground.
.IP "\fB\-f configfile\fR" 4
.IX Item "-f configfile"
Specify a grok config file to use.
.SH "CONFIGURATION"
.IX Header "CONFIGURATION"
You can call the config file anything you want. A full example config follows
below, with documentation on options and defaults.
.PP
.Vb 7
\& # \-\-\- Begin sample grok config
\& # This is a comment. :)
\& #
\& # enable or disable debugging. Debug is set false by default.
\& # the \*(Aqdebug\*(Aq setting is valid at every level.
\& # debug values are copied down\-scope unless overridden.
\& debug: true
\&
\& # you can define multiple program blocks in a config file.
\& # a program is just a collection of inputs (files, execs) and
\& # matches (patterns and reactions),
\& program {
\&   debug: false
\&
\&   # file with no block. settings block is optional
\&   file "/var/log/messages"
\&
\&   # file with a block
\&   file "/var/log/secure" {
\&     # follow means to follow a file like \*(Aqtail \-F\*(Aq but starts
\&     # reading at the beginning of the file.  A file is followed
\&     # through truncation, log rotation, and append.
\&     follow: true
\&   }
\&
\&   # execute a command, settings block is optional
\&   exec "netstat \-rn"
\&
\&   # exec with a block
\&   exec "ping \-c 1 www.google.com" {
\&     # automatically rerun the exec if it exits, as soon as it exits.
\&     # default is false
\&     restart\-on\-exit: false
\&
\&     # minimum amount of time from one start to the next start, if we
\&     # are restarting. Default is no minimum
\&     minimum\-restart\-interval: 5
\&
\&     # run every N seconds, but only if the process has exited.
\&     # default is not to rerun at all.
\&     run\-interval: 60
\&
\&     # default is to read process output only from stdout.
\&     # set this to true to also read from stderr.
\&     read\-stderr: false
\&   }
\&
\&   # You can have multiple match {} blocks in your config.
\&   # They are applied, in order, against every line of input that
\&   # comes from your exec and file instances in this program block.
\&   match {
\&     # match a pattern. This can be any regexp and can include %{foo}
\&     # grok patterns
\&     pattern: "some pattern to match"
\&
\&     # You can have multiple patterns here, any are valid for matching.
\&     pattern: "another pattern to match"
\&
\&     # the default reaction is "%{@LINE}" which is the full line
\&     # matched.  the reaction can be a special value of \*(Aqnone\*(Aq which
\&     # means no reaction occurs, or it can be any string. The
\&     # reaction is emitted to the shell if it is not none.
\&     reaction: "%{@LINE}"
\&
\&     # the default shell is \*(Aqstdout\*(Aq which means reactions are
\&     # printed directly to standard output. Setting the shell to a
\&     # command string will run that command and pipe reaction data to
\&     # it.
\&     #shell: stdout
\&     shell: "/bin/sh"
\&
\&     # flush after every write to the shell. 
\&     # The default is not to flush.
\&     flush: true
\&
\&     # break\-if\-match means do not attempt any further matches on
\&     # this line.  the default is false.
\&     break\-if\-match: true
\&   }
\& }
\& # \-\- End config
.Ve
.SH "PATTERN FILES"
.IX Header "PATTERN FILES"
Pattern files contain lists of names and patterns for loading into grok.
.PP
Patterns are newline-delimited and have this syntax: 
 \fIpatternname\fR \fIexpression\fR
.PP
Any whitespace between the patternname and expression are ignored.
.IP "patternname" 4
.IX Item "patternname"
This is the name of your pattern which, when loaded, can be referenced in
patterns as %{patternname}
.IP "expression" 4
.IX Item "expression"
The expression here is, verbatim, available as a regular expression. You do not
need to worry about how to escape things.
.SS "\s-1PATTERN\s0 \s-1EXAMPLES\s0"
.IX Subsection "PATTERN EXAMPLES"
.Vb 2
\& DIGITS \ed+
\& HELLOWORLD \ebhello world\eb
.Ve
.SH "REGULAR EXPRESSIONS"
.IX Header "REGULAR EXPRESSIONS"
The expression engine underneath grok is \s-1PCRE\s0. Any syntax in \s-1PCRE\s0 is valid in grok.
.SH "REACTIONS"
.IX Header "REACTIONS"
Reactions can reference named patterns from the match. You can also access a few other special values, including:
.IP "%{@LINE}" 4
.IX Item "%{@LINE}"
The line matched.
.IP "%{@MATCH}" 4
.IX Item "%{@MATCH}"
The substring matched
.IP "%{@START}" 4
.IX Item "%{@START}"
The starting position of the match from the beginning of the string.
.IP "%{@END}" 4
.IX Item "%{@END}"
The ending position of the match.
.IP "%{@LENGTH}" 4
.IX Item "%{@LENGTH}"
The length of the match
.IP "%{@JSON}" 4
.IX Item "%{@JSON}"
The full set of patterns captured, encoded as a json dictionary as a structure
of { pattern: [ array of captures ] }. We use an array becuase you can use the
same named pattern multiple times in a match.
.IP "%{@JSON_COMPLEX}" 4
.IX Item "%{@JSON_COMPLEX}"
Similar to the above, but includes start and end position for every named
pattern. That structure is:
.Sp
.Vb 7
\& { "grok": [ 
\&    { "@LINE": { "start": ..., "end": ..., "value": ... } },
\&    { "@MATCH": { "start": ..., "end": ..., "value": ... } },
\&    { "patternname": { "start": startpos, "end": endpos, "value": "string" } },
\&    { "patternname2": { "start": startpos, "end": endpos, "value": "string" } },
\&    ...
\& ] }
.Ve
.SS "\s-1REACTION\s0 \s-1FILTERS\s0"
.IX Subsection "REACTION FILTERS"
Reaction filters allow you to mutate the captured data. The following filters are available:
.PP
An example of using a filter in a reaction is like this:
 reaction: \*(L"echo Matched: %{@MATCH|shellescape}\*(R"
.IP "shellescape" 4
.IX Item "shellescape"
Escapes all characters necessary to make the string safe in non-quoted a shell argument
.IP "shelldqescape" 4
.IX Item "shelldqescape"
Escapes characters necessary to be safe within doublequotes in a shell.
.IP "jsonencode" 4
.IX Item "jsonencode"
Makes the string safe to represent in a json string (escapes according to json.org recommendations)
.SH "SEE ALSO"
.IX Header "SEE ALSO"
\&\fIpcre\fR\|(3), \fIpcresyntax\fR\|(3),
.PP
Sample grok configs are available in in the grok samples/ directory.
.PP
Project site: <http://semicomplete.googlecode.com/wiki/Grok>
.PP
Google Code: <http://semicomplete.googlecode.com/>
.PP
Issue/Bug Tracker: <http://code.google.com/p/semicomplete/issues/list>
.SH "CONTACT"
.IX Header "CONTACT"
Please send questions to grok\-users@googlegroups.com. File bugs and feature requests at the following \s-1URL:\s0
.PP
Issue/Bug Tracker: <http://code.google.com/p/semicomplete/issues/list>
.SH "HISTORY"
.IX Header "HISTORY"
grok was originally in perl, then rewritten in \*(C+ and Xpressive (regex), then
rewritten in C and \s-1PCRE\s0.
.SH "AUTHOR"
.IX Header "AUTHOR"
grok was written by Jordan Sissel.