File: Export.html

package info (click to toggle)
collectl 3.7.4-1
  • links: PTS
  • area: main
  • in suites: jessie, jessie-kfreebsd
  • size: 1,624 kB
  • ctags: 119
  • sloc: perl: 14,928; sh: 429; makefile: 11
file content (220 lines) | stat: -rw-r--r-- 12,073 bytes parent folder | download | duplicates (4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
<html>
<head>
<link rel=stylesheet href="style.css" type="text/css">
<title>Exporting Custom Output</title>
</head>

<body>
<center><h1>Exporting Custom Output</h1></center>
<p>
<h3>Introduction</h3>
As with <i>--import</i>, the <i>--export</i> option allows one to build
custom modules, but in this case for generating output.  Unlike <i>--import</i>, which
allows multiple modules to be specified, you only <i>--export</i>
using a single module, which in turn overrides all of collectl's output
formatting routines.
<p>
These modules all end in the extension <i>.ph</i> and are searched for first 
in the directory collectl is being executed from and then <i>/usr/share/collectl</i>.
If the module name is prepended with a directory name, collectl will search only
there.
<p>
The reason you might care about this is that now if you want to produce your 
own exportable form of output and be able to print it locally, make it 
available to another program over a socket or even write to a local file 
while still being able to log to <i>raw</i> and/or <i>plot</i> formats, 
you get that all that functionality for free.
<p>
<h3>How It Works</h3>
The interface to all this is really quite simple.  At the command line the user
types <i>collectl --export name[,options]</i> where:

<ul>
<li><b>name</b> both names a file to be <i>required</i> by collectl as well as
the name of the entry points to both an initialization routine as well as one
that produces the output</li>
<li><b>options</b> specifies an optional list of arguments that are passed to both
routines to do with whatever they wish</li>
</ul>

There are currently 5 different custom exports that are part of a standard collectl
distribution:
<ul>
<li><b><a href=Gexpr.html>gexpr</a></b> allows one to send collectl data to <a href=http://ganglia.info/>ganglia</a></li>
<li><b><a href=Graphite.html>graphite</a></b> allows one to send collectl data to <a href=http://graphite.wikidot.com/>graphite</a></li>
<li><b>lexpr</b> generates output in an easy to parse format</li>
<li><b><a href=Process.html>proctree</a></b> provides another way to look at process data</li>
<li><b>vmstat</b> is the standard unix <i>vmstat</i> command most people know and love,
so why reinvent it?  It's both a useful example of how to write an export as well as
provides the benefit of allowing one to play back collectl data in vmstat format!</li>
</ul>

The first four are the most interesting because they have been built to take advantage
of collectl's capability of sending their output over a socket and are therefore the ideal
vehicle for interfacing with other tools and environments.  All four share a common set
of options as described in the following table:
<p>
<center>
<table border=1 width=85%>
<tr><td width=10%><b>align</b></td><td>Used in conjunction with <i>i=</i> and when specified data samples will
                be aligned to whole minute boundaries.  In other words if used with <i>i=15</i>, data will be reported
                at the top of the minute, 15 seconds past, etc.  The first sample may therefore be partial.</td>
<tr><td width=10%><b>avg|max|min|tot</b></td><td>used in conjunction with <i>i=</i>, send the average, maximum,
                minimum or total of the data over the associated set of intervals specified by <i>i=</i>. If 
                none of these are specified, the values from the most recent monitoring interval will
                be reported.</td></tr>
<tr><td width=10%><b>co</b></td><td>this does not take a value and indicates <i>changes only</i> such
                that only data elements that have changed since they were last sampled are reported in
                an attempt to minimize processing and network bandwidth.  If not specified, samples for
                all reporting intervals will be sent.</td></tr>
<tr><td><b>d=mask</b></td><td>debugging mask, see beginning of the actual export file for details</td></tr>
<tr><td><b>f=file</b></td><td>names the output snapshot file, which applies to only lexpr.
                If this option is not used, -f must be and the snapshot file name which is set to the
                single character <i>L</i> filename and written into the directory 
                associated with -f
		<center><i><b>caution</b></i></center>
		If you are writing your own export module that doesn't use a snapshot file, you must explicity
		include error checking to assure it is not run without at least <i>-P</i> or <i>--rawtoo</i> in
                combination with <i>-f</i></td></tr>
<tr><td><b>h</b></td><td>shows help/usage</td></tr>
<tr><td><b>i=secs</b></td><td>specifies the reporting interval in seconds.  In other words, if you specify
                i=60, a sample will be reported every 60 seconds independent of collectl's monitoring
                interval.  The default is to report every sample.  This interval must be a multiple of 
                the base collectl interval.  <i>note that while collectl always rounds rates to the next whole
                value, when multiple intervals are added together only the totals are rounded.</i></td></tr>
<tr><td><b>s</b></td><td>specifies a subset of those subsystems specified with -s in the collectl command line
                and only data collected for that subset will be reported.  The default to report everything.
                In the case where you only want report data collected via <i>--import</i>, use s= with
                no args.</td></tr>
<tr><td><b>ttl</b></td><td> is the <i>time to live</i> in intervals for each piece of performance data.  If
                more than this number of intervals passes data will be sent regardless of whether it changed
                or not and the ttl countdown timer reset.  The default is 5.  This actually has a second use
                for gexpr and that is to set the gmond ttl to double this number multiplied by the interval.</td></tr>
</table>
</center>
<p>
<b>Logging</b>
<br>
Collectl can actually create up to 3 different type of log files and it's worth spending a little more
time enumerating how collectl decides where and when to create them.
<ul>
<li>In its simplest form, when one runs collectl with --export and no options, all output is sent to the 
terminal, noting <i>gexpr</i> and <i>graphite</i> always require an address.</li>
<li>If collectl is run with -f, and one is exporting with <i>lexpr</i> a snapshot
file is created in the directory associated with -f.</li>
<li>If a socket is to be used by specifying <i>collectl -A</i> and the export supports 
socket I/O:</li>
  <ul>
    <li>all output goes out over the socket and no snapshot file is involved</li>
    <li>if one specifies -f, a raw file is created in the directory specified by -f.
        If <i>-P</i> is also used, a plot file is created instead and if <i>-P</i> and <i>--rawtoo</i> are used,
        both types of files are created.  <i>caution:</i> this is different behavior than one sees when running
        without sockets in which case a snapshot file is written to the directory named by <i>-f</i>
    <li>Since <i>gexpr</i> and <i>graphite</i> do not use <i>-A</i> for specifying socket I/O and do their own
        communications, collectl's rules for non-socket logging apply, in which case <i>-f</i> by itself implies 
        a snapshot file is involved and so requires <i>-P</i> or <i>--rawtoo</i> or <i>both</i> to do any logging.</li>
  </ul>
<li>To create the snapshot file in a directory other then the one specified with -f, use the
<i>f=</i> option with <i>--export</i></li>
<li>If one tries an invalid combination of switches or something not supported, it is up to 
the export itself to determine that since by design collectl has no knowledge of those 
module's inner workings.</li>
</ul>

<h3>Example</h3>
Perhaps the best way to see how all this works is with a simple example and
it turns out that <i>vmstat.ph</i> is small enough to meet that need.  You
may also wish to refer to the others as well to see how some of the  more exotic
capabilities are implemented.
<p>
This first section gets called almost immediately by collectl after reading
in the various user switches.  This is the place to catch switch errors
and since this routine always requires <i>-scm</i> we'll just hardcode it 
to that and reject any user entered ones.  This initialization subroutine
<i>must</i> be named for our module followed by <i>Init</i>.
<div class=terminal-wide14>
<pre>
sub vmstatInit
{
  error("-s not allowed with 'vmstat'")          if $userSubsys ne '';
  error("-f requires either --rawtoo or -P")     if $filename ne '' && !$rawtooFlag && !$plotFlag;
  error("-P or --rawtoo require -f")             if $filename eq '' && ($rawtooFlag || $plotFlag);
  $subsys=$userSubsys='cm';
}
</pre></div>

Next we define the output routine, with the same base name as that
of our included file.
<p>
The <i>if statement</i> uses collectl's standard idiom for printing headers
based on the number of lines printed and whether or not the user wants only a
single header, no header or even to clear the screen between headers.

<div class=terminal-wide14>
<pre>
sub vmstat
{
  my $line;
  if (printHeader())
  {
    $line= "${cls}#${miniBlanks}procs ---------------memory (KB)--------------- --swaps-- -----io---- --system-- ----cpu-----\n";
    $line.="#$miniDateTime r  b   swpd   free   buff  cache  inact active   si   so    bi    bo   in    cs us sy  id wa\n";
  }
</pre></div>

Next comes the handling of optional date/time prefixes that I stole from
printTerm() in formatit.ph and which can be controlled
by various switch options.  Again, if you have no intent of supporting these
you can even put in error handling in your <i>initialization routine</i>
or simply ignore the switches.

<div class=terminal>
<pre>
  my $datetime='';
  if ($options=~/[dDTm]/)
  {
    ($ss, $mm, $hh, $mday, $mon, $year)=localtime($lastSecs);
    $datetime=sprintf("%02d:%02d:%02d", $hh, $mm, $ss);
    $datetime=sprintf("%02d/%02d %s", $mon+1, $mday, $datetime)                  if $options=~/d/;
    $datetime=sprintf("%04d%02d%02d %s", $year+1900, $mon+1, $mday, $datetime)   if $options=~/D/;
    $datetime.=".$usecs"                                                         if ($options=~/m/);
    $datetime.=" ";
  }
</pre></div>

Here we build the actual output, noting that we're not really printing
anything yet, but rather building up a string (which may contain the header) 
that we will print in one shot.
<div class=terminal-wide14>
<pre>
  my $i=$NumCpus;
  my $usr=$userP[$i]+$niceP[$i];
  my $sys=$sysP[$i]+$irqP[$i]+$softP[$i]+$stealP[$i];
  $line.=sprintf("%s %2d %2d %6s %6s %6s %6s %6s %6s %4d %4d %5d %5d %4d %5d %2d %2d %3d %2d\n",
                $datetime, $procsRun, $procsBlock,
                cvt($swapUsed,6,1,1),  cvt($memFree,6,1,1),  cvt($memBuf,6,1,1),
                cvt($memCached,6,1,1), cvt($inactive,6,1,1), cvt($active,6,1,1),
                $swapin/$intSecs, $swapout/$intSecs, $pagein/$intSecs, $pageout/$intSecs,
                $intrpt/$intSecs, $ctxt/$intSecs,
                $usr, $sys, $idleP[$i], $waitP[$i]);
</pre></div>

Finally comes the output.  There is actually a lot of latitude here and in this case
we're calling <i>printText()</i> which will send the output to the terminal or over a
socket.  It will not write to a local file as does <i>lexpr</i>, but if you
want to see how to do that, refer to its source.  As with all perl require files, they must
return <i>true</i> and therefore the final line is the digit 1.
<div class=terminal-wide14>
<pre>
  printText($line);
}
1;
</pre></div>

Try running it and you'll see all the pagination and time formats work just as they
do with standard output formats.

<table width=100%><tr><td align=right><i>updated November 9, 2012</i></td></tr></colgroup></table>

</body>
</html>