1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
.\"
.\" Argus-5.0 Software
.\" Copyright (c) 2000-2024 QoSient, LLC
.\" All rights reserved.
.\"
.\"
.TH RASQLINSERT 1 "8 April 2023" "rasqlinsert 5.0.3"
.SH NAME
\fBrasqlinsert\fP \- write \fBargus(8)\fP data into mysql database tables.
.SH SYNOPSIS
.B rasqlinsert
[\fBraoptions\fP] [\fB--\fP \fIfilter-expression\fP]
.SH DESCRIPTION
.IX "rasqlinsert command" "" "\fBrasqlinsert\fP \(em argus data"
.LP
\fBRasqlinsert\fP writes
.BR argus
data into a mysql database.
The principal function of \fBrasqlinsert\fP is to insert and update flow data attributes,
into a MySQL database table. Using the same syntax and strategies for all other
ra* programs, \fBrasqlinsert\fP creates databases and database tables, based on
the print specification on the either the command-line or the .rarc file.
The concept is that where a ra* program would print fields to standard out in
ascii, \fBrasqlinsert\fP will insert those fields into the database as attributes.
The flow key, as defined by the "-m fields" option, provides the definition of
any keys that would be used in the schema. A "-m none" option, will remove
the use of any DBMS keys for inserted data, and is the method to use when inserting
streaming, unprocessed, primitive argus data into a database table.
The schema is important for database utility and performance. You can use MySQL
querys against the attributes that you insert into the tables, such searching and
sorting on IP addresses, time, packet counts, etc.... While \fBrasqlinsert\fP
does not limit you to the number of attributes (columns) per record you provide, the
RDBMS performance will quide you as to how many fields are useful.
\fBRasqlinsert\fP by default, includes the actual binary argus 'record' in the
schema, and inserts and updates the binary record when needed. This enables
a large number of fucnctions that extend beyond simple RDBMS schema's that
are useful. Adding the 'record' is expensive, and some will elect to not
use this feature. This can be controlled using the option '-s -record' as a
print field option in the standard \fBra.1\fP command line. When the 'record'
attribute is present, \fBrasql.1\fP can read the records directly from the database,
to provide additional processing on the database table contents.
When keys are used, the database will enforce that any insertions meet the
relaitional requirements, i.e. that the keys be unique. This requirement demands
a sense of caching and key tracking, which \fBrasqlinsert\fP is specifically
designed to provide.
\fBRasqlinsert\fP by default, will append data to existing tables, without checking
the schema for consistency. If your schema has keys, and you attempt to append
new records to an existing table, there is a high likelyhood for error, as
\fBrasqlinsert\fP will attempt to insert a record that collides with an existing
flow key. Use the "-M cache" option to cause \fBrasqlinsert\fP to reference the
table contents prior to aggregation and insertion.
The binary data that \fBrasqlinsert\fP inserts by default is retrieved
with \fBrasql(1)\fP.
.LP
.SH RASQLINSERT SPECIFIC OPTIONS
\fBRasqlinsert\fP, like all ra based clients, supports a number of \fBra options\fP
including filtering of input argus records through a terminating filter expression.
\fBRasqlinsert(1)\fP specific options are:
.TP 4 4
.B \-M cache
This causes \fBrasqlinsert\fP to use the database table as its persistent
cache store. This mechanism is used to control memory use when dealing with large
amounts of data and flow keys.
.TP 4 4
.B \-M drop
This causes \fBrasqlinsert\fP to drop any pre-existing database table that
has the same name as the target table name, on startup.
.TP 4 4
.B \-M rewrite
Update SQL tables with a freshly processed record field.
This allows existing tables to be modified in place and is provided as
a way to reduce the stored flow data.
For example, when combined with the "-M dsrs=" mode it can be used to
remove certain DSRs from the in-table flow record.
Some commandline parameters are ignored in rewrite mode and some others
conflict.
The cache and drop modes cannot be used with rewrite.
No input files (-r) may be specified since the source of data is always
the table to be rewritten.
Lastly, the field specifier (-s) and flow model (-m) options will be
ignored since these parameters are derived from the table being rewritten.
See the INVOCATION section below for an example of the rewrite mode.
.SH INVOCATION
This invocation writes aggregated \fBargus(8)\fP data from the \fIfile\fP into
a database table. The standard 5-tuple fields, 'saddr daddr proto sport dport'
are used as keys for each entry. \fBrasqlinsert\fP will aggregate all the data
prior to inserting the data into the database:
.nf
\fBrasqlinsert\fP -r file -w mysql://user@localhost/db/table
.fi
Because aggregation can require a lot of memory, \fBrasqlinsert\fP provides an
option '-M cache' to have \fBrasqlinsert\fP use the database table as the persistent
cache store for the aggregation. With this example, the standard 5-tuple fields,
\&'saddr daddr proto sport dport' are still used as keys for each entry, but
\fBrasqlinsert\fP will aggregate data over short spans of time as it reads the data
from the file, and then commit the data to the database. If additional data arrives
that matches that unique flow, \fBrasqlinsert\fP will fetch the entry from the
database, aggregate, and then update the data entry in the database.
.nf
\fBrasqlinsert\fP -M cache -r file -w mysql://user@localhost/db/table
.fi
\fBrasqlinsert\fP can provide the same function for streaming data read directly from
an argus data source. This allows \fBrasqlinsert\fP to reassemble all status
records for an individual flow, such that the resulting table has only a single entry
for each communciation relationship seen.
.nf
\fBrasqlinsert\fP -M cache -S argus -w mysql://user@localhost/db/table
.fi
This invocation writes \fBargus(8)\fP data from the \fIfile\fP into a database
table, without aggregation, by specifying no relational key in the data.
.nf
\fBrasqlinsert\fP -m none -r file -w mysql://user@localhost/db/table
.fi
This invocation writes \fBargus(8)\fP data from the \fIstream\fP into a database
table, without modification.
.nf
\fBrasqlinsert\fP -m none -S argus -w mysql://user@localhost/db/table
.fi
This invocation writes \fBargus(8)\fP data from the \fIstream\fP into a daily
database table, without modification. \fBrasqlinsert\fP will generate table names
based on time and insert its data relative to the timestamps found in the flow
records it processes. In this specific example, "-M time 1d" specifies daily
tables.
.nf
\fBrasqlinsert\fP -m none -S argus -w mysql://user@localhost/db/table_%Y_%m_%d -M time 1d
.fi
Update tables by removing the source and destination user buffer DSRs
from the binary flow record column for the month of February, 2018.
.nf
\fBrasqlinsert\fP -M rewrite time 1d dsrs=-suser,-duser \\
-t 2018/02 \\
-w mysql://user@localhost/db/table_%Y_%m_%d
.fi
.SH COPYRIGHT
Copyright (c) 2000-2024 QoSient. All rights reserved.
.SH SEE ALSO
.BR rasql(1),
.BR ra(1),
.BR rarc(5),
.BR argus(8),
.SH AUTHORS
.nf
Carter Bullard (carter@qosient.com).
.fi
|
