1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
Gmetad Python - Initial Version
This represents the initial version of the rewritten gmetad in python.
Files are:
- gmetad_config.py
- gmetad_daemon.py
- gmetad_data.py
- gmetad_element.py
- gmetad_gmondReader.py
- gmetad_notifier.py
- gmetad_plugin.py
- gmetad.py
- gmetad_random.py
- gmetad_xmlWriter.py
- gmetad_consistency_test.py (test script)
This version should function generally as a gmetad replacement for reading
gmond data, providing aggregate data on XML ports and writing individual
and summary metric RRD files.
Advertised capabilities of this version include:
- Support of current gmetad configuration files - should "just work" by
pointing it directly to an existing gmetad.conf file. However the RRD
support requires the addition of the RRDplugin and RRDSummary plugins.
These plugins require some additional configuration.
- Most current gmetad command-line options, plus a few others.
- Six log levels as per the python "logging" module. Two of these - FATAL and
CRITICAL - are functionally equivalent but indicated separately in the
command-line parameters. The default logging or debug level is 2, meaning
that all messages ERROR level and higher will be logged. A specified debug
level of 5 or higher means all debug messages are shown and that the
application stays in the foreground.
- Logging is done to syslog, optionally by appending to a log file in addition
to syslog. If the debug level is 5 or higher, logging is also done to the
console.
- At a logging level of 4 or below, the application will go through the
daemonizing process (setuid to another user, fork twice, set session id,
change working dir to "/" and set umask to 0, close all open file descriptors
except the logging file descriptors, and redirect all other I/O to /dev/null.
- Writes a PID file that can be used for graceful shutdowns later (kill -TERM).
- Creates a reader thread for every data source specified in the configuration.
All data is stored internally in a single data structure which is updated on
every read.
- Readers honor the polling interval specified in the data source configuration
but vary the interval by +- 5 seconds.
- Listens on the XML and Interactive ports. XML port is a dump of all stored
data in a single XML file. Interactive waits for input from the user then
dumps the XML.
- Interactive port supports basic pathing filters, such as /Gridname/
Clustername/Hostname/Metricname.
- Interactive port ignores filters that don't match the corresponding level in
the XML - in other words, all data at that level will be sent, and no
filtering applied, if the specified filter doesn't match any of the tags at
that level.
- Shuts down gracefully at SIGTERM (generally this is true, unless an exception
occurs - and even then, it usually works...).
- Plugin module support provides the ability to write gmetad plugins. These plugins
are handed a cluster transaction each time that data is read from a cluster. The
transaction contains the current data for all nodes in a cluster.
- RRD and RRD summary plugins that write the individual and summary metrics to
RRD files.
How To Write And Use A Gmetad Python Plugin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A new version of Gmetad has been rewritten completely in
the Python scripting language. This rewrite is functionally
equivalent to the original Gmetad daemon however it provides
a plugable interface so that Gmetad can be extended. The Gmetad
daemon itself is only responsible for reading cluster data and
storing the raw data in memory as in an internal data store. Plugins
are responsible for processing the raw cluster data and providing
Gmetad with extended storage and analysis capability. For example,
Gmetad itself no longer writes data to an RRD file. The RRD storage
functionality has been moved to an RRD plugin storage module. This
same plugin capability could also be used to process and analyze
the cluster in order to produce events or alerts based on the
metric data.
Writing a python Gmetad plugin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Every python Gmetad plugin must be derived from the GmetadPlugin
base class. The resulting .py file must also be placed in a
specified Gmetad plugin directory before Gmetad is started.
The plugin directory can be specified in the gmetad.conf file
or a default location will be used. When Gmetad is started, it
will attempt to load all .py files that are found in the plugin
directory and then determine if it is a valid gmetad plugin by
verifying that the implemented plugin is based on the GmetadPlugin
base class. Every plugin must implement an __init__(),
_parseConfig() and notify() methods. In addition, the plugin must
also supply a factory function that creates an instance of the
plugin class. Optionally, the plugin can also implement a start()
and stop() methods that can be used to start up or shutdown any
module specific functionality. The following is an explanation
of the various methods can be mandatory or optionally implemented
for each plugin:
def get_plugin():
Each plugin must implmenent a factory function that instantiates
the plugin class. The class instance must be passed the configuration
ID that corresponds to the plugin section of the configuration file
contains configuration information for the plugin. For example, the
follwoing would return an instance of the RRD plugin and specify the
configuration ID of the plugin:
return RRDPlugin('rrd')
def __init__(self, cfgid):
This function will be called once when the plugin class in
first instantiated. The cfgid parameter will contain the
configuration ID for the module. This configuration ID should
be passed to the at parent's __init__() method and represents
the ID that is used to locate the plugin specific configuration
section in the gmetad.conf file. As mentioned, the plugin's
__init__() must call the parent's __init__() (GmetadPlugin.__init__())
method with the cfgid.
def _parseConfig(self, cfgdata):
As a result of calling the the parent's __init__() method with
the cfgid paramenter, the base GmetadPlugin class will locate
and read the configuration parameter section that is
associated with the plugin's cfgid. It will then call this
method with the configuraton data. It is the plugin's
responsibility to evaluate the configuration data and
act on the data appropriately.
def notify(self, clusterNode):
Each plugin must implement a notify() method. The plugin's notify
method will be called each time that cluster data has been
read from a gmond daemon and updated in the Gmetad in-memory
master data store. The cluster data is formated as a heirarchical
tree of nodes that represent each host and metric contained in
the cluster. The node heirarchy is a copy of the same cluster
heirarchy in the master data store so changing any of the node
attributes will not have any affect on master data store and any
changes to the master data store during processing, will not affect
the plugin. However, since the same cluster node copy is passed
to all of the plugins in the chain, if any plugin makes a change
to the cluster heirarchy, it will affect downstream plugins.
def start(self):
The implementation of the start() method is optional. It provides the
plugin with the opportunity to start up or initialize any plugin
specific functionality or data.
def stop(self):
The implementation of the stop() method is optional. It provides the
plugin with the opportunity to stop or uninitialize any plugin
specific functionality or data.
Other than the implemenation of the mandatory methods and factory
function, the Gmetad plugin is free to do whatever it needs
in order to appropriately store or analyze the metric data.
Deploying a Gmetad plugin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once a python Gmetad plugin has been developed, deploying the plugin
is as easy as copying a file to a specific directory. The gmetad.conf
configuration should specify a path using the plugins_dir directory
where all gmetad python plugins should be located. To deploy a
gmetad plugin, simply copy the .py file to the specified plugin
directory. Once the python plugin has been copied, starting gmetad
will cause the plugin to be loaded and initialized.
Configuring a Gmetad plugin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The configuration of a Gmetad plugin requires that a plugin
specific section be added to the gmond.conf file. The format of
the section is as follows:
<cfgID> {
<directive> <value>
...
}
The <cfgID> must match the cfgID that is initially passed in to the
plugin instance through the plugin factory function. The cfgID will
be used by the base plugin class to get the plugin configuration data
from the master configuration store. A plugin can implement any
configuration directives that it needs. When the Gmetad configuration
parser encounters a plugin section, it will read each configuration
directive and store the values without any attempts to interpret
the configuration data. It is up to the plugin itself to interpret
the meaning of each configuration directive and value.
|
