1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
|
<html>
<head>
<!-- This file has been generated by unroff 1.0, 03/01/02 08:56:55. -->
<!-- Do not edit! -->
<STYLE TYPE="text/css">
<!--
A:link{text-decoration:none}
A:visited{text-decoration:none}
A:active{text-decoration:none}
-->
</STYLE>
<title>ploticus: proc processdata</title>
<body bgcolor=D0D0EE vlink=0000FF>
<br>
<br>
<center>
<table cellpadding=2 bgcolor=FFFFFF width=550 ><tr>
<td>
<table cellpadding=2 width=550><tr>
<td><br><h2>proc processdata</h2></td>
<td align=right>
<small>
<a href="../doc/Welcome.html"><img src="../doc/ploticus.gif" border=0></a><br>
<a href="../doc/Welcome.html">Welcome</a>
<a href="../gallery/index.html">Gallery</a>
<a href="../doc/Contents.html">Handbook</a>
<td></tr></table>
</td></tr>
<td>
<br>
<br>
<title>Manual page for proc_processdata(PL)</title>
</head>
<body>
<h2>DESCRIPTION</h2>
<b>proc processdata</b> may be used to perform various types
processing on a data set after it is read.
<br><br><br>
<h2>FEATURES</h2>
Various types of processing may be done, including
break processing, accumulation, tabulation and counting, rewriting as percents,
computation of totals, reversing record order, and rotation of row/column matrix.<tt> </tt>
Only one of these types of processing may be done per invocation of proc processdata,
and the type of processing is controlled by the <tt>action</tt> attribute.<tt> </tt>
<p>
In addition to any one of the actions, data may be further processed in
that fields may be filtered (<tt>keepfields</tt> or <tt>rejectfields</tt>).<tt> </tt>
<p>
The result may replace the original data set in memory or be kept in memory
along with the original data (<tt>stack</tt>).<tt> </tt>
In either case, after proc processdata is finished the result will be
considered the "current data set" which all plotting procs will use.<tt> </tt>
<p>
You can write the results to a file (<tt>outfile</tt>) for processing by
other programs, or to read back in and process further using <b>proc getdata</b>
<tt>filter</tt>.<tt> </tt>
<p>
For additional processing capabilities see also
<a href="getdata.html">
proc getdata
</a>
<tt>filter</tt>.<tt> </tt>
<br><br><br>
<h2>EXAMPLE</h2>
A Gallery example where this is used is
<a href="../gallery/stock.htm">
stock
</a>
where the available data is in reverse chronological order.<tt> </tt>
Since lineplot must work from left to right, <b>proc processdata</b>
is used to reverse the record order.<tt> </tt>
<br><br><br>
<h2>VARIABLES THAT ARE SET</h2>
<p>
<b>NRECORDS</b> = Number of rows in the data result.<tt> </tt>
<p>
<b>NFIELDS</b> = Number of fields per row in the data result.<tt> </tt>
<p>
<b>TOTALS</b> = If totals, percents, or accumulation are being done,
this variable will be set to hold the field total(s). If more than
one field is being operated on, this will be a comma-delimited
list of totals; individual totals may be accessed in your script
using something like the following, which would access the first
total in the list:
<tt>#set T = $nmember(1,@TOTALS)</tt>
<br>
<p>
<b>BREAKFIELD1 .. n</b> = Current contents of break fields, when
<tt>action</tt> is <b>breaks</b>.<tt> </tt>
<br><br><br>
<h2>MANDATORY ATTRIBUTES</h2>
The <tt>action</tt> attribute must be specified, unless
just keeping/rejecting fields.<tt> </tt>
<br><br><br>
<h2>ATTRIBUTES</h2>
<p>
<b>action</b> <i>a</i>
<dl>
<dt><dd><p>
The type of processing to perform.<tt> </tt>
Often the <tt>fields</tt> attribute is used to indicate which field(s)
are involved.<tt> </tt>
Legal values for <i>a</i> include:
<dt><dd><p>
<b>accumulate</b> Rewrite field as a cumulative series (accumulation).<tt> </tt>
The field(s) to operate on must be given in the <tt>fields</tt> attribute.<tt> </tt>
For example, the data set on the left would be transformed to the one
on the right (<tt>fields: 2</tt>):
<pre>
A21 3 A21 3
A22 5 --> A22 8
A23 2 A23 10
A24 1 A24 11
</pre>
<dt><dd><p>
<b>breaks</b> (new in version 1.39). Break processing. In data processing
terminology, "break processing" is the act of passing through a sorted
data set and taking some action when a key field or fields change.<tt> </tt>
Break processing is significantly more efficient than scanning a data
set multiple times with a select statement.<tt> </tt>
The data set must be sorted such that key fields are grouped.<tt> </tt>
The key field(s) must be specified using the <tt>fields</tt> attribute.<tt> </tt>
The <b>BREAKFIELD1 .. n</b> variable(s) will be set to the current contents
of the break field(s). Your script can detect when the entire data set has
been processed by checking
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).<tt> </tt>
A gallery example that uses this feature is
<a href="../gallery/mouse.htm">
mouse.<tt> </tt>
</a>
See MORE ON BREAK PROCESSING below for more information.<tt> </tt>
<dt><dd><p>
<b>breakreset</b> Reset the "current row" to the beginning of the data set,
for the occasional time when more than one pass through a data set will be done
using the <b>breaks</b> action.<tt> </tt>
<dt><dd><p>
<b>count</b> Collapse data by counting the number of instances of a key
field. Input data must be sorted (or at least grouped) on the key field.<tt> </tt>
Resulting data set will always have two fields.<tt> </tt>
One or two fields must be specified using the <tt>fields</tt> attribute.
If one field is specified, the result fields will be 1) key field, 2) count.<tt> </tt>
For example (<tt>fields: 1</tt>):
<pre>
062698 062698 2
062698 --> 062898 1
062898 070198 3
070198 070498 1
070198
070198
070498
</pre>
<dt><dd><p>
If two fields are specified, the result fields will be 1) key field, 2) sum
of the numeric contents found in the second specified field.<tt> </tt>
For example (<tt>fields: 1 2</tt>):
<pre>
062698 4 062698 10
062698 6 --> 062898 3
062898 3 070198 9
070198 2 070498 2
070198 4
070198 3
070498 2
</pre>
See also the gallery example
<a href="../gallery/hitcount.htm">
hitcount
</a>
<dt><dd><p>
<b>percent</b> Rewrite one or more fields as percentages of its field (column) total.<tt> </tt>
The field(s) to operate on must be given in the <tt>fields</tt> attribute.<tt> </tt>
For example (<tt>fields: 1</tt>):
<pre>
8 40
4 --> 20
3 15
5 25
</pre>
<dt><dd><p>
<b>reverse</b> The last record becomes the first one; the
record order is reversed. For example (<tt>fields: 2</tt>):
<pre>
AXB 34 DIF 14
BYA 22 --> CES 52
CES 52 BYA 22
DIF 14 AXB 34
</pre>
<dt><dd><p>
<b>rotate</b> First row becomes 1st field, 2nd row
becomes 2nd field, and so on. This may be useful
in that most of the plotting procs work from data fields,
but sometimes data is given (or is more intuitive) in rows.<tt> </tt>
For example:
<pre>
A 2 4 6 8 10 --> A B
B 3 6 9 12 15 2 3
4 6
6 9
8 12
10 15
</pre>
<dt><dd><p>
<b>select</b> Select certain records.<tt> </tt>
Result will be made up of records meeting the
condition given in the <tt>select</tt> attribute.<tt> </tt>
<dt><dd><p>
<b>total</b> Compute field total(s) only and place total(s) into the variable
TOTALS (see above). This action does not rewrite data.<tt> </tt>
The field(s) to operate on must be given in the <tt>fields</tt> attribute.<tt> </tt>
<br>
The decimal format of the total(s) is controlled by the <tt>resultformat</tt>
attribute. If total(s) are to be written in presentable notation
(a spacer for thousands, etc.) the <tt>resultformat</tt> attribute may be
preceded by a <tt>n</tt>, e.g. <tt>n%7.0f</tt>.<tt> </tt>
</dl>
<p>
<b>fields</b>
<a href="attributetypes.html#dfield">
dfield
</a>
list
<dl>
<dt><dd><p>
The field(s) to be operated on when doing percents, accumulations, or totals.<tt> </tt>
<br>
Example: <tt>fields: 2 5 6 7</tt>
</dl>
<p>
<b>keepfields</b>
<a href="attributetypes.html#dfield">
dfield
</a>
list
<dl>
<dt><dd><p>
If specified, only the listed fields in the original data set
will be kept. The others will be rejected.
<tt>action</tt> may be anything.<tt> </tt>
<br>
Example: <tt>keepfields: 4 5 6</tt>
</dl>
<p>
<b>rejectfields</b>
<a href="attributetypes.html#dfield">
dfield
</a>
list
<dl>
<dt><dd><p>
If specified, the listed fields in the original data set
will not be kept. The others will be.
<tt>action</tt> may be anything.<tt> </tt>
<br>
Example: <tt>rejectfields: 1 2 3 4</tt>
</dl>
<p>
<b>fieldnames</b> <i>namelist</i>
<dl>
<dt><dd><p>
If specified, the names given in <i>namelist</i> may be used
in any plotting proc to identify data fields. <i>namelist</i> is a
space- or comma- delimited list of names. Names may include any alphanumeric
characters with a maximum length of 38, and are case-insensitive.<tt> </tt>
<br>
Note that if field names are specified in <b>proc getdata</b>
and then <b>proc processdata</b> is used to alter the order of fields or
delete fields, then this <tt>fieldnames</tt> attribute <b>must</b> be
used in order to redefine the field names properly.<tt> </tt>
<br>
Example: <tt>fieldnames: date group n</tt>
</dl>
<p>
<b>resultformat</b>
<a href="attributetypes.html#printfspec">
printf-spec
</a>
<dl>
<dt><dd><p>
Controls the decimal format of rewritten percents, accumulations, totals.<tt> </tt>
</dl>
<p>
<b>keepall</b> <tt>yes</tt> | <tt>no</tt>
<dl>
<dt><dd><p>
If <tt>yes</tt>, original fields are preserved when doing accumulate, percent, or total.<tt> </tt>
Thus the result will have more fields than the original data set. Default is <tt>no</tt>.<tt> </tt>
</dl>
<p>
<b>select</b>
<a href="condex.html">
selection-expression
</a>
<dl>
<dt><dd><p>
Used when <tt>action</tt> is <tt>select</tt> in order to specify the selection
condition.
</dl>
<p>
<b>stack</b> <tt>yes</tt> | <tt>no</tt>
<dl>
<dt><dd><p>
The default is <tt>no</tt>, which causes the processing results replace the original data
set in memory. However, if <tt>stack</tt> is set to <tt>yes</tt>, the results become
the current data set and the original data set is preserved (it may
be restored as the 'current' data set by invoking
<a href="originaldata.html">
proc originaldata.<tt> </tt>
</a>
</dl>
<p>
<b>showresults</b> <tt>yes</tt> | <tt>no</tt>
<dl>
<dt><dd><p>
If <tt>yes</tt> the data are shown after processing, as a diagnostic aid.<tt> </tt>
</dl>
<p>
<b>outfile</b> <i>filename</i>
<dl>
<dt><dd><p>
If specified, results are written to <i>filename</i>, in tab-delimited format,
for processing by other programs, or to read back in and process further using
<b>proc getdata</b> <tt>filter</tt>.<tt> </tt>
This only works with <tt>stack: no</tt>
<br><br><br>
</dl>
<h2>MORE ON BREAK PROCESSING</h2>
<p>
The <b>breaks</b> action allows break processing.
In data processing terminology, "break processing" is the act of passing
through a sorted data set and taking some action when a key field or fields change.<tt> </tt>
For example, if we were processing a list of charges ordered by paying budget
number, we could use a break processing strategy to pause and generate
a statement for one budget number, when we reached the point in the
data set where the budget numbers changed. Then we would continue on.<tt> </tt>
The <b>breaks</b> action allows a similar thing to be done with plotting.<tt> </tt>
Break processing is significantly more efficient than scanning the entire data
set multiple times with a select statement, especially with larger data sets.<tt> </tt>
<p>
The data set must be sorted such that key fields are grouped.<tt> </tt>
The key field(s) must be specified in the <tt>fields</tt> attribute.<tt> </tt>
<b>proc processdata</b> is generally called within a #loop. When <b>proc processdata</b>
finishes, the "current data set" will be the block of data from the previous break
to the current break. Subsequent invocations of <b>proc processdata</b> will continue
from the previous location in the data set. The original data set is always assumed
(i.e. it is not necessary to use <b>proc originaldata</b>).
<p>
Your script can access the current contents of the break field(s) via the
<b>BREAKFIELD1 .. n</b> variable(s).<tt> </tt>
Your script can detect when the entire data set has been processed by checking
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).<tt> </tt>
The following is an example:
<br>
<pre>
#loop
#proc processdata
action: breaks
fields: 1 2 3
#proc endproc
#if @NRECORDS = 0
#break
#endif
#proc page
title: Account @BREAKFIELD1
#proc bars
...
#endloop
</pre>
<p>
Limits: up to 5 break fields may be used.<tt> </tt>
Comparisons for equality are limited to the first 50 characters.<tt> </tt>
<br>
<br>
</td></tr>
<td align=right>
<a href="../doc/Welcome.html">
<img src="../doc/ploticus.gif" border=0></a><br><small>data display engine <br>
<a href="../doc/Copyright.html">Copyright Steve Grubb</a>
<br>
<br>
<center>
<img src="../gallery/all.gif">
</center>
</td></tr>
</table>
<p><hr>
Markup created by <em>unroff</em> 1.0, <tt> </tt> <tt> </tt>March 01, 2002.
</body>
</html>
|
