File: processdata

package info (click to toggle)
ploticus-doc 2.33-1
links: PTS
area: main
in suites: etch, etch-m68k
size: 9,392 kB
ctags: 169
sloc: pascal: 469; makefile: 63; sh: 11
file content (849 lines) | stat: -rw-r--r-- 19,787 bytes
.ig >>
<STYLE TYPE="text/css">
<!--
        A:link{text-decoration:none}
        A:visited{text-decoration:none}
        A:active{text-decoration:none}
        OL,UL,P,BODY,TD,TR,TH,FORM { font-family: arial,helvetica,sans-serif;; font-size:small; color: #333333; }

        H1 { font-size: x-large; font-family: arial,helvetica,sans-serif; }
        H2 { font-size: large; font-family: arial,helvetica,sans-serif; }
        H3 { font-size: medium; font-family: arial,helvetica,sans-serif; }
        H4 { font-size: small; font-family: arial,helvetica,sans-serif; }
-->
</STYLE>
<title>ploticus: proc processdata</title>
<body bgcolor=D0D0EE vlink=0000FF>
<br>
<br>
<center>
<table cellpadding=2 bgcolor=FFFFFF width=550><tr>
<td>
  <table cellpadding=2 width=550><tr>
  <td><br><h2>proc processdata</h2></td>
  <td align=right>
  <small>
  <a href="../doc/welcome.html"><img src="../doc/ploticus.gif" border=0></a><br>
  Version 2.33 Jun'06
     </small><br><a href="../doc/scripthome.html">Scripts</a>
  <td></tr></table>
</td></tr>
<td>
<br>
<br>
.>>

.TH proc_processdata PL "02-JUN-2006   PL ploticus.sourceforge.net"

.LP
\fBproc processdata\fR performs various types of manipulations on the
.ig >>
<a href="dataformat.html#currentds">
.>>
\0current data set.
.ig >>
</a>
.>>
Here are some of the things this proc can do: 
.IP \0
- accumulate
.IP \0
- break processing
.IP \0
- count/summate on instances
.IP \0
- joins 
.IP \0
- row numbering
.IP \0
- compute percents
.IP \0
- reverse row order
.IP \0
- rotation of row/column matrix
.IP \0
- find mean, sd, min, max
.IP \0
- compute totals
.LP
The \fCaction\fR attribute controls the type of processing to be done.
The result is usually a new data set (but sometimes the result is simply
the setting of some variables, as with \fCaction: totals\fR).
.LP
When a new data set is created, it will be kept in memory along with the original data by default.
To discard the new data set and return to the original one, use 
.ig >>
<a href="usedata.html">
.>>
\0proc usedata.
.ig >>
</a>
.>>
For large data sets you can cause the result to overwrite the
original data set in memory by setting (\fCstack: no\fR).
In either case, after proc processdata is finished with an operation that creates a new data set 
the result will automatically become the "current data set".
.LP
You can write the results to a file (\fCoutfile\fR) for processing by
other programs, or to read back in and process further using \fBproc getdata\fR
\fCfilter\fR.
.LP
For additional processing capabilities see also
.ig >>
<a href="getdata.html">
.>>
\0proc getdata
.ig >>
</a>
.>>
\fCfilter\fR.

.ig >>
<br><br><br>
.>>

.SH Example
A Gallery example where this is used is
.ig >>
<a href="../gallery/stock.htm">
.>>
\0stock
.ig >>
</a>
.>>
where the available data is in reverse chronological order.
Since lineplot must work from left to right, \fBproc processdata\fR
is used to reverse the record order.

.ig >>
<br><br><br>
.>>

.SH Variables that are set by processdata
.LP
\fBNRECORDS\fR = Number of rows in the data result.
.LP
\fBNFIELDS\fR = Number of fields per row in the data result.
.LP
\fBTOTALS\fR = If totals, percents, or accumulation are being done, 
this variable will be set to hold the field total(s).  If more than
one field is being operated on, this will be a comma-delimited
list of totals; individual totals may be accessed in your script
using something like the following, which would access the first
total in the list: \fC#set T = $nmember(1,@TOTALS)\fR
.LP
\fBBREAKFIELD1 .. n\fR = Current contents of break fields with \fCaction: breaks\fR
.LP
\fCaction: stats\fR sets a number of other variables, described in the \fCstats\fR section below.


.ig >>
<br><br><br>
.>>

.SH The action attribute
The \fCaction\fR attribute controls the type of operation that will be done.
Often the \fCfields\fR attribute is used to indicate which field(s) are involved.  

.ig >>
<br><br>
.>>

.IP \0
\fBaction: accumulate\fR  
.ig >>
&nbsp; &nbsp;
.>>
Rewrite field as a cumulative series (accumulation).
The field(s) to operate on must be given in the \fCfields\fR attribute.
For example, the data set on the left would be transformed to the one 
on the right (\fCfields: 2\fR):
.nf
	A21 3			A21 3
	A22 5		-->	A22 8 
	A23 2			A23 10
	A24 1			A24 11
.fi

.IP \0
\fBaction: breaks\fR 
.ig >>
&nbsp; &nbsp;
.>>
Perform break processing (the act of passing through an ordered
data set and taking some action when a key field or fields change).
Break processing is significantly more efficient than scanning a data 
set multiple times with a \fCselect\fR statement.
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified using the \fCfields\fR attribute.
The \fBBREAKFIELD1 .. n\fR variable(s) will be set to the current contents
of the break field(s).  Your script can detect when the entire data set has 
been processed by checking 
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
For more discussion of break processing
.ig >>
<a href="#breaks">
.>>
\0see below.
.ig >>
</a>
.>>

.ig >>
<br><br>
.>>

.IP \0
\fBaction: breakreset\fR  
.ig >>
&nbsp; &nbsp;
.>>
Reset the "current row" to the beginning of the data set,
for the occasional time when more than one pass through a data set will be done
using \fCaction: breaks\fR .
This is automatically done if a new data set is read by proc getdata.



.ig >>
<br><br>
.>>
           
.IP \0
\fBaction: count\fR  
.ig >>
&nbsp; &nbsp;
.>>
Collapse data by counting the number of instances of a key field.
This action can also summate some other field based on a key 
field.  Input data must be sorted (or at least grouped) on the key field.
Resulting data set will always have two fields.
One or two \fCfields\fR must be specified using the \fCfields\fR attribute.  
If one field is specified, the result fields will be 1) key field, 2) count.
For example (\fCfields: 1\fR):
.nf
	062698	 		062698 2
	062698 		  -->	062898 1
	062898 			070198 3
	070198 			070498 1
	070198	 
	070198
	070498 
.fi
.IP \0
If two \fCfields\fR are specified, the result fields will be 1) key field, 2) sum
of the numeric contents found in the second specified field.
For example (\fCfields: 1 2\fR):
.nf
	062698 4		062698 10
	062698 6	  -->	062898 3
	062898 3		070198 9
	070198 2		070498 2
	070198 4
	070198 3
	070498 2
.fi
See also \fCaction: segment\fR and \fCaction: segmentb\fR described below, which are
similar.
A gallery example that uses \fCaction: count\fR is
.ig >>
<a href="../gallery/hitcount.htm">
.>>
\0hitcount
.ig >>
</a>
.>>

.ig >>
<br><br><br>
.>>

.IP \0
\fBaction: join\fR
.ig >>
&nbsp; &nbsp;
.>>
Perform a relational join operation. (2.30+)
All records come from the current data set; left side records are selected using the \fCleftselect\fR
attribute and right side records are selected using \fCrightselect\fR.
The records are joined on one or more fields specified in the \fCfields\fR attribute.
The data must be ordered on these fields (integer fields should be ordered numerically; alphanumeric
fields should be ordered alphabetically).
One (and only one) left side record and one (and only one) right side record must exist in order to be joined
(but see leftjoin and rightjoin below).
For example, suppose we want a scatterplot where the X component and Y component
are represented in different sets of records within the data set:
.nf
	001 X 4.3		001 X 4.3 001 Y 5.2
	001 Y 5.2	-->	002 X 3.2 002 Y 2.9
	002 X 3.2
	002 Y 2.9
.fi
In order to do a scatterplot of X vs. Y we need to do a join so that all data for case 001 are
on the same row, and all data for case 002 are on the same row.  We can do this using:
.nf
	#proc processdata
	showresults: yes
	action: join
	fields: 1
	leftselect: @@2 = X
	rightselect: @@2 = Y
.fi
Also available are \fBaction: leftjoin\fR and \fBaction: rightjoin\fR.  
With a left join, if a left side record exists with no right side companion, the right side 
is filled in with a missing data code (\fC=\fR by default; can be specified in \fCmissingdatacode\fR 
attribute).  A right join is the same thing but in reverse.

.ig >>
<br><br><br>
.>>

.IP \0
\fBaction: numberrows\fR
.ig >>
&nbsp; &nbsp;
.>>
Prepend a row number field to each row.  For example:
.nf
	A AA AAA 0		1 A AA AAA 0
	B BB BBB 0     --->  	2 B BB BBB 0
	C CC CCC 1		3 C CC CCC 1
	D DD DDD 1		4 D DD DDD 1
	E EE EEE 0		5 E EE EEE 0
	F FF FFF 1		6 F FF FFF 1
.fi

.ig >>
<br><br>
.>>

.IP \0
\fBaction: percent\fR  
.ig >>
&nbsp; &nbsp;
.>>
Rewrite one or more fields as percentages of its field (column) total.
The field(s) to operate on must be given in the \fCfields\fR attribute.
For example (\fCfields: 1\fR):
.nf
	8		40
	4	   -->	20
	3		15
	5		25
.fi

.ig >>
<br><br>
.>>
.IP \0
\fBaction: reverse\fR  
.ig >>
&nbsp; &nbsp;
.>>
The last record becomes the first one; the
record order is reversed. For example (\fCfields: 2\fR):
.nf
	AXB 34		DIF 14
	BYA 22	   -->	CES 52
	CES 52		BYA 22
	DIF 14		AXB 34 
.fi

.ig >>
<br><br>
.>>
.IP \0
\fBaction: rotate\fR  
.ig >>
&nbsp; &nbsp;
.>>
First row becomes 1st field, 2nd row
becomes 2nd field, and so on.  This may be useful
in that most of the plotting procs work from data fields,
but sometimes data is given (or is more intuitive) in rows.
For example:
.nf
	A 2 4 6 8 10	-->	A B
	B 3 6 9 12 15		2 3
				4 6
				6 9
				8 12
				10 15
.fi

.ig >>
<br><br>
.>>
.IP
\fBaction: segment\fR  
.ig >>
&nbsp; &nbsp;
.>>
Similar to \fCaction: count\fR, except that instead of a count result, a range is produced,
useful for plotting bar segments, etc.
Resulting data set will always have three fields: 1) key field, 2) start position, 3) end position.
.IP
If one \fCfield\fR is specified, the start and end position will be record numbers of the first
and last records in the range (where first record is 1).  For example (\fCfields: 1\fR):
.nf
	062698			062698 1 3
	062698          --->	070198 4 6
	062698			070498 7 7
	070198
	070198
	070198
	070498
.fi
If two \fCfields\fR are specified, the start and end position will be the contents of the second
specified field, for the first and last records in the range.  
For example (\fCfields: 1 2\fR):
.nf
	062698 A		062698 A C
	062698 B         --->	070198 D F
	062698 C		070498 G G
	070198 D
	070198 E
	070198 F
	070498 G
.fi
.IP
\fBaction: segmentb\fR 
.ig >>
&nbsp; &nbsp;
.>>
Same as \fCaction: segment\fR, except that groups will butt up against one another.
For example (\fCfields: 1 2\fR):
.nf
	062698 A		062698 A D
	062698 B         --->	070198 D G
	062698 C		070498 G G
	070198 D
	070198 E
	070198 F
	070498 G
.fi

.ig >>
<br><br>
.>>

.IP \0
\fBaction: stats\fR
.ig >>
&nbsp; &nbsp;
.>>
Find the mean, SD, min, max, etc. for data field(s), and put mean, SD, min, max (etc) into ploticus variables.  
No new data set is created. Versions 2.30+.
The data field(s) to operate on should specified in the \fCfields\fR attribute.  
Sets the following ploticus variables:
.br
\fBMEAN\fR = mean of all numeric values found
.br
\fBSD\fR = standard deviation
.br
\fBN\fR = number of numeric values found
.br
\fBMIN\fR = the lowest numeric value found
.br
\fBMIN_ID\fR = contents of identifier/abscissa field in data row where minima was found
.br
\fBMAX\fR = the highest numeric value found
.br
\fBMAX_ID\fR = contents of identifier/abscissa field in data row where maxima was found
.br
\fBNMISSING\fR = number of non-numeric observations
.br
\fBTOTAL\fR = sum of all numeric values in the field
.IP 
(The following is a change in functionality starting in 2.31:)
This action can operate on one data field or several.  
The data field(s) to operate on should specified in the \fCfields\fR attribute.  
If more than one data field is specified, the data values from all the fields will be examined as
a group and the result will still be a single mean, SD, etc. 
\fCtagfield\fR may be specified (see below) to indicate an identifier or abscissa field 
which will be used to identify min and max cases (MIN_ID and MAX_ID) and may be useful in annotating 
the min and/or max with a later invocation of #proc annotate.


.ig >>
<br><br><br>
.>>

.IP \0
\fBaction: total\fR  
.ig >>
&nbsp; &nbsp;
.>>
Compute field total(s) only and place total(s) into the variable
TOTALS (see above).  No new data set is created.
Field(s) to be totalled are specified in the \fCfields\fR attribute.
.br
The decimal format of the total(s) is controlled by the \fCresultformat\fR
attribute.  If total(s) are to be written in presentable notation
(a spacer for thousands, etc.) the \fCresultformat\fR attribute may be
preceded by a \fCn\fR, e.g. \fCn%7.0f\fR.

.ig >>
<br><br><br>
.>>

.SH Other attributes

.LP
\fBfields\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#dfield">
.>>
\0dfield
.ig >>
</a>
.>>
list
.IP \0
The field(s) to be operated on.  Required for any \fCaction\fR that involves
data fields.
.br
Example: \fCfields: 2 5 6 7\fR

.ig >>
<br><br>
.>>

.LP
\fBkeepfields\fR
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#dfield">
.>>
\0dfield
.ig >>
</a>
.>>
list
.IP \0
If specified, only the listed fields in the original data set 
will be kept.  The others will be rejected.  
\fCaction\fR may be anything.
.br
Example: \fCkeepfields:  4 5 6\fR

.ig >>
<br><br>
.>>
.LP
\fBrejectfields\fR
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#dfield">
.>>
\0dfield
.ig >>
</a>
.>>
list
.IP \0
If specified, the listed fields in the original data set 
will not be kept.  The others will be.  
\fCaction\fR may be anything.
.br
Example: \fCrejectfields:  1 2 3 4\fR

.ig >>
<br><br>
.>>
.LP
\fBtagfield\fR
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#dfield">
.>>
\0dfield
.ig >>
</a>
.>>
.IP \0
Used with \fCaction: stats\fR to specify an identifier or abscissa data field,
which will be used to identify min and max cases.

.ig >>
<br><br>
.>>
.LP
\fBfieldnames\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fInamelist\fR
.IP \0
If specified, result data fields will be assigned the given names.
These names can later be used in any plotting proc to identify data fields.  
\fInamelist\fR is a space- or comma- delimited list of names.  
Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.
.br
Note that if field names are specified in \fBproc getdata\fR 
and then \fBproc processdata\fR is used to alter the order of fields or 
delete fields, then this \fCfieldnames\fR attribute \fBmust\fR be
used in order to redefine the field names properly.
.br
Example: \fCfieldnames: date group n\fR

.ig >>
<br><br>
.>>

.LP
\fBresultformat\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#printfspec">
.>>
\0printf-spec
.ig >>
</a>
.>>
.IP \0
Controls the decimal format of rewritten percents, accumulations, totals.
Default is \fC%g\fR.
.br
Example: \fCresultformat: %f\fR
.br
Example for \fCaction: totals\fR (see above): \fCresultformat: n%g\fR

.ig >>
<br><br>
.>>

.LP
\fBkeepall\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
If \fCyes\fR, original fields are preserved when doing accumulate, percent, or total.
Thus the result will have more fields than the original data set.  Default is \fCno\fR.

.ig >>
<br><br>
.>>

.LP
\fBselect\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="select.html">
.>>
\0selection expression
.ig >>
</a>
.>>
.IP \0
Used when \fCaction\fR is \fCselect\fR in order to specify the selection 
condition.  Beginning in version 2.20 \fCselect\fR can be used along with
any \fCaction\fR.
.br
Example: \fCselect: @@4 = A\fR

.ig >>
<br><br>
.>>

.LP
\fBstack\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
The default is \fCyes\fR which causes
the result data set to be "stacked", and the original data set is preserved in memory.
If \fCno\fR, the result data set replaces the original data set in memory (this might
be desired for larger data sets).

.ig >>
<br><br>
.>>

.LP
\fBshowresults\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCyes | no\fR
.IP \0
If \fCyes\fR the data are shown after processing, as a diagnostic aid.

.ig >>
<br><br>
.>>

.LP
\fBoutfile\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fIfilename\fR
.IP \0
If specified, results are written to \fIfilename\fR, in tab-delimited format,
for processing by other programs, or to read back in and process further using 
\fBproc getdata\fR \fCfilter\fR.
\fIfilename\fR can be a pathname or \fCstdout\fR or \fCstderr\fR.
Use of this attribute implies \fCstack: no\fR

.ig >>
<br><br>
.>>

.LP
\fBleftselect\fR
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="select.html">
.>>
\0select expression
.ig >>
</a>
.>>
.br
\fBrightselect\fR
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="select.html">
.>>
\0select expression
.ig >>
</a>
.>>
.IP \0
Used with \fCaction: join\fR (and leftjoin, rightjoin).
Specifies selection conditions for left side of join and right side of join.

.ig >>
<br><br>
.>>

.LP
\fBmissingdatacode\fR
.ig >>
&nbsp; &nbsp;
.>>
string
.IP \0
Used with \fCaction: leftjoin\fR (and rightjoin) to specify a missing data code to use for
filling in missing fields.


.ig >>
<a name=breaks></a>
.>>
.ig >>
<br><br><br>
.>>

.SH More on break processing
.LP
\fCaction: breaks\fR performs break processing.  
In data processing terminology, "break processing" is the act of passing 
through a sorted data set and doing something special when a change is encountered in key field(s).
For example, if we were processing a list of charges ordered by paying budget
number, we could use a break processing strategy to pause and generate
a statement for one budget number, when we reached the point in the
data set where the budget numbers changed.  Then we would continue on.
\fCaction: breaks\fR allows a similar thing to be done with plotting.
Break processing is significantly more efficient than scanning the entire data 
set multiple times with a \fCselect statement\fR, especially with larger data sets.
.LP
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified in the \fCfields\fR attribute.
\fBproc processdata\fR is generally called within a #loop.  When \fBproc processdata\fR
finishes, the 
.ig >>
<a href="dataformat.html#currentds">
.>>
\0current data set
.ig >>
</a>
.>>
will be the block of data from the previous break to the current break.  
\fBproc usedata\fR must be used at the bottom of the loop, to set the 
.ig >>
<a href="dataformat.html#currentds">
.>>
\0current data set
.ig >>
</a>
.>>
back to the larger data set that we are passing thru.
Subsequent invocations of \fBproc processdata\fR then continue
from the most recent break location.
.LP
Your script can access the current contents of the break field(s) via the
\fBBREAKFIELD1 .. n\fR variable(s).
Your script can detect when the entire data set has been processed by checking 
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
The following is an example:
.br
.nf
\0	#loop
\0	   #proc processdata
\0	     action: breaks
\0	     fields: 1 2 3
\0	   #proc endproc
\0
\0	   #if @NRECORDS = 0
\0	     #break
\0	   #endif
\0
\0	   #proc page
\0	   title: Account @BREAKFIELD1
\0
\0	   #proc bars
\0	   ...
\0
\0         #proc usedata
\0           original: yes
\0
\0	 #endloop
.fi
.LP
Limits: up to 5 break fields may be used.
Comparisons for equality are limited to the first 50 characters.

.ig >>
<br>
<br>
</td></tr>
<td align=right>
<a href="../doc/welcome.html">
<img src="../doc/ploticus.gif" border=0></a><br><small>data display engine &nbsp; <br>
<a href="../doc/Copyright.html">Copyright Steve Grubb</a>
<br>
<br>
<center>
<img src="../gallery/all.gif"> 
</center>
</td></tr>
</table>
<br>
<center>
Ploticus is hosted at http://ploticus.sourceforge.net <br>
<img src="http://sourceforge.net/sflogo.php?group_id=38453" width="88" height="31" border="0" alt="SourceForge Logo">
</center>
.>>