1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
|
.\"
.\" Argus-5.0 Software
.\" Copyright (c) 2000-2024 QoSient, LLC
.\" All rights reserved.
.\"
.\"
.TH RASTREAM 1 "12 August 2023" "rastream 5.0.3"
.SH NAME
\fBrastream\fP \- stream block processor for \fBargus(8)\fP data.
.SH SYNOPSIS
.B rastream
[[\fB\-M\fP \fIsplitmode\fP] [\fIsplitmode options\fP]]
[\fB\-f\fP \fIfile processing program\fP -B secs]
[\fBraoptions\fP] [\fB--\fP \fIfilter-expression\fP]
.SH DESCRIPTION
.IX "rastream command" "" "\fBrastream\fP \(em argus data"
.LP
\fBRastream\fP reads
.BR argus
data from an \fIargus-data\fP source, and splits the resulting
output into consecutive sections of records based on size, count
time, or flow event, writing the output into a set of output-files.
\fBRastream\fP provides the option to run a program against the output
files, \fBseconds\fP after the file is understood to be finished.
The program must be specified in a manner so that \fBrastream\fP can
find it, either within the system $PATH, or provided as a full pathname.
By default, \fBrastream\fP splits the stream by output file record
count, putting 10,000 records of input into each \fBargus\fP output
file, or standard out, as needed. The behavior is similar to the
unix split.1 command.
The output files' name consists of a prefix, which is specified using
the \fI-w\fP \fIra option\fP, and a suffix, which is created for each
resulting file. If no prefix is provided, then \fBrastream\fP will
use 'x' as the default prefix. The suffix that is used is determined
by the mode of operation. When \fBrastream\fP is using the default
count mode or the size mode, the suffix is a group of letters 'aa',
\'ab\', and so on, such that concatenating the output files in sorted
order by file name produces the original input file. If \fBrastream\fP
will need to create more output files than are allowed by the default
suffix strategy, more letters will be added, in order to accomodate
the needed files. When the mode is \fBtime\fP mode, the default
output filename suffix is '%Y.%m.%d.%h.%m.%s', which is used by
strftime() to create an output filename that is time oriented.
This default is overrided by adding a '%' extension to the name
provided on the commandline using the \fI-w\fP option.
When standard out is specified, using \fI-w -\fP, \fBrastream\fP
will output a single \fBargus-stream\fP with START and STOP argus management
records inserted appropriately to indicate where the output is split.
See \fBargus(8)\fP for more information on output stream formats.
When \fBrastream\fP is spliting on output record count (the default), the
number of records is specified as an ordinal counter, the default is
10,000 records. When \fBrastream\fP is spliting based on the maximum output
file size, the size is specified as bytes. The scale of the bytes can be
specified by appending 'b', 'k' and 'm' to the number provided.
When \fBrastream\fP is spliting based on time, the time period is specified
with the option, and can be any period based in seconds (s), minutes (m),
hours (h), days (d), weeks (w), months (M) or years (y). \fBRastream\fP
will create and modify records as required to split on prescribed time
boundaries. If any record spans a time boundary, the record is split
and the metrics are adjusted using a uniform distribution model to
distribute the statistics between the two records. Care is taken to
avoid records with zero packet and byte counts, that could result
from roundoff error.
When \fBrastream\fP is spliting based on flow event, the flow that acts
as the event marker is specified using a standard \fBra\fP filter
expression, that is bounded by quotes ("). Records that preceed the
first flow event in the data stream are written to the specified
output file, and then new files are generated with the flow event
record being the first record of the new file. This method will allow
you to use wire events as triggers for spliting data.
.SH RASTREAM SPECIFIC OPTIONS
Rastream, like all ra based clients, supports
a number of \fBra options\fP including remote data access, reading
from multiple files and filtering of input argus records through a
terminating filter expression.
\fBrastream(1)\fP specific options are:
.TP 4 4
.BI \-a "\| suffix length\^"
Starting append suffix length. The default is 2 characters.
.TP 4 4
.BI \-B "\| duration\^"
Buffer hold time before processing. This value is usually in
the 5-15 second range and provides time for rastream to sort records
and schedule outputfile processing. The number is derived from the
larges FAR status interval of all the argus data sources encountered.
.TP 4 4
.BI \-f "\| program\^"
Post processing program. \fBrastream\fP, will execute this program
just after closing the output file, passing the full path to the
closed output file as a parameter, using this convention:
.nf
program -r /full/path/to/closed/file
.fi
This allows you to post-process the output file in an automated fashion.
Generally, this program can do anything you like, such as aggregating
and correcting flow records, labeling records for semantic enhancement,
indexing the files, using programs like rasqltimeindex(), and compressing
the files. Traditionally, the program has been a shell-script, perl
program, or php script, so that it can be easily modified, on the fly,
but it can be any executable that can handle the "-r filename" parameter
convention. The program should provides its own accountability and
error logging, so that you know that things are working as you expect.
\fBrastream\fP must have a path to the program, the program must be
executable, and \fBrastream\fP must have permission to run the program
for this strategy to be successful.
An example rastream.sh is provided in the ./support/Config directory.
.TP 4 4
.BI \-M "\| splitmode\^"
Supported spliting modes are:
.nf
\fB count <num>\fP
\fB size <size>\fP
\fB time <period>\fP
\fB flow "filter-expression"\fP
.fi
.TP 4 4
.BI \-M "\| lock[=nonblock]\^"
\fBRastream\fP has additional functionality for file locking.
If the lock mode is "nonblock", a failed lock results in rastream writing
to a temporary file, instead.
When all input has been processed the temporary files are addressed.
For each temporary file rastream again attempts to lock the intended
output file, but this time it blocks until the file is available.
The contents of the temporary file are appended to the target file and
the temporary file removed.
No guarantees are made that the resulting output file contains all
records in start-time order and additional processing may be required.
Multiple instances of rastream can concurrently request locks for the
same set of files, but in different orders, resulting in deadlock.
Non-blocking locks address the deadlock scenario.
The blocking locks used during cleanup are aquired singly, avoiding
any opportunity for deadlock, and in the vast majority of cases will be
aquired for a small subset of output files.
.TP 4 4
.BI \-w "\| filename\^"
\fBRastream\fP supports an extended \fI-w\fP option that allows for
output record contents to be inserted into the output filename.
Specified using '$' (dollar) notation, any printable field can be used.
Care should be taken to honor any shell escape requirements when
specifying on the command line. See \fBra(1)\fP for the list of
printable fields.
Another extended feature, when using \fBtime\fP mode, \fBrastream\fP
will process the supplied filename using \fBstrftime(3)\fP, so that
time fields can be inserted into the resulting output filename.
.SH INVOCATION
This invocation reads \fBargus(8)\fP data from \fBinputfile\fP and splits
the \fBargus(8)\fP data stream based on output file size of no greater
than 1 Megabyte. The resulting output files have a prefix of \fIargus.\fP
and suffix that starts with 'aa'. The single trailing '.' is significant.
.nf
\fBrastream\fP -r inputfile -M size 1m -w argus.
.fi
This invocation splits \fBinputfile\fP based on hard 10 minute time boundaries.
The resulting output files are created with a prefix of \fI/archive/%Y/%m/%d/argus.\fP
and the suffix is \fI%H.%M.%S\fP. The values will be supplied based on the time in
the record being written out.
.nf
\fBrastream\fP -r * -M time 10m -w "/archive/%Y/%m/%d/argus.%H.%M.%S"
.fi
This invocation splits \fBinputfile\fP based on the argus source identifier.
The resulting output files are created with a prefix of \fI/archive/Source Identifier/argus.\fP
and the default suffix starting with "aa". The source identifier will be
supplied based on the contents of the record being exported.
.nf
\fBrastream\fP -r * -M time 10m -w "/archive/$srcid/argus."
.fi
This invocation splits \fBinputfile\fP based on a flow event marker.
The resulting output files are created with a prefix of 'outfile.' and
the default suffix starting with "aa". Whenever a ping to a specific
host is seen in the stream, a new output file is generated.
.nf
\fBrastream\fP -r * -M flow "echo and host 1.2.3.4" -w outfile.
.fi
.SH COPYRIGHT
Copyright (c) 2000-2024 QoSient. All rights reserved.
.SH SEE ALSO
.BR ra(1),
.BR rarc(5),
.BR argus(8),
.SH AUTHORS
.nf
Carter Bullard (carter@qosient.com).
.fi
|
