File: getdata

package info (click to toggle)
ploticus-doc 2.33-1
links: PTS
area: main
in suites: etch, etch-m68k
size: 9,392 kB
ctags: 169
sloc: pascal: 469; makefile: 63; sh: 11
file content (723 lines) | stat: -rw-r--r-- 15,426 bytes
.ig >>
<STYLE TYPE="text/css">
<!--
        A:link{text-decoration:none}
        A:visited{text-decoration:none}
        A:active{text-decoration:none}
        OL,UL,P,BODY,TD,TR,TH,FORM { font-family: arial,helvetica,sans-serif;; font-size:small; color: #333333; }

        H1 { font-size: x-large; font-family: arial,helvetica,sans-serif; }
        H2 { font-size: large; font-family: arial,helvetica,sans-serif; }
        H3 { font-size: medium; font-family: arial,helvetica,sans-serif; }
        H4 { font-size: small; font-family: arial,helvetica,sans-serif; }
-->
</STYLE>
<title>ploticus: proc getdata</title>
<body bgcolor=D0D0EE vlink=0000FF>
<br>
<br>
<center>
<table cellpadding=2 bgcolor=FFFFFF width=550><tr>
<td>
  <table cellpadding=2 width=550><tr>
  <td><br><h2>proc getdata</h2></td>
  <td align=right>
  <small>
  <a href="../doc/welcome.html"><img src="../doc/ploticus.gif" border=0></a><br>
  Version 2.33 Jun'06
     </small><br><a href="../doc/scripthome.html">Scripts</a>
  <td></tr></table>
</td></tr>
<td>
<br>
<br>
.>>

.TH proc_getdata PL "02-JUN-2006   PL ploticus.sourceforge.net"
 
.LP
\fBproc getdata\fR gets data for plotting,
and must be invoked before plotting any data.
Data may be read from files, from commands, or specified directly, and various 
.ig >>
<a href="dataformat.html">
.>>
\0input data formats
.ig >>
</a>
.>>
are supported.
The data which are read become the 
.ig >>
<a href="dataformat.html#currentds">
.>>
\0current data set
.ig >>
</a>
.>>
that subsequent plotting procs will access,
and replace any data that were read in (or generated) previously.
.LP
\fBproc getdata\fR will set the variable \fBNRECORDS\fR to the number of data rows gotten.
The variable \fBNFIELDS\fR will be set to the number of fields per data row.

.ig >>
<br><br>
.>>
.SH See also
.ig >>
<a href="dataformat.html">
.>>
\0Discussion of input data formats and parsing rules
.ig >>
</a>
.>>
.LP
.ig >>
<a href="dataformat.html#currentds">
.>>
\0Discussion of working with multiple data sets
.ig >>
</a>
.>>

.ig >>
<br><br><br>
.>>
.SH Attributes
Default field delimitation method is \fCspacequoted\fR; \fCdelim\fR must be set for other
types such as tab-delimited or comma-quote delimited.


.ig >>
<br><br><br>
.>>

.SH Specifying a data source
One (and only one) of the following data sources \fBmust\fR be specified:

.LP
\fBfile\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fIfilename\fR
.IP \0
Shell-expandable name of a file containing plotting data.
This name will be used along with cat(1) in a shell command, thus
exported shell variables and metacharacters may be part of the name.
A dash (-) may be used if data is to be read from the standard input,
(or the \fCstandardinput\fR attribute may be used).
.IP
Security concern: user-supplied values, such as CGI user variables, should not be used to build \fCfilename\fR. 
.IP
On win32 platforms, or when operating in
.ig >>
<a href="cgi.html">
.>>
\0direct CGI mode
.ig >>
</a>
.>>
(2.31-08+)
\fCfilename\fR automatically maps to \fCpathname\fR .
.IP
Example: \fCfilename: myfile.dat\fR

.ig >>
<br><br>
.>>

.LP
\fBpathname\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fIfilename\fR
.IP \0
Name of a file containing plotting data.
The file will be opened directly.
.IP
Security concern: user-supplied values, such as CGI user variables, should not be used to build the \fCpathname\fR, 
unless proper measures are taken to detect and remove the \fC../\fR construct (used as a hack 
to see higher levels of the file system).

.ig >>
<br><br>
.>>

.LP
\fBdata\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#text">
.>>
\0multiline-text
.ig >>
</a>
.>>
.IP \0
Literal specification of plotting data.  Terminates at
first blank (empty) line.  Data may include a field name header.
Example:
.nf
.ft C
data:   "Case 1"   0   4   4.1   
        "Case 2"   1   5   4.4  
        "Case 3"   2   2   4.0 
        "Case 4"   3   9   4.8

.fi
.ig >>
<br><br>
.>>

.LP
\fBcommand\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fIshell command line\fR
.IP \0
An external shell command that will produce plot data on its standard output.
.IP
Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command.
If this must be done, use \fC#shell / #endshell\fR along with \fCdata:\fR, which provides some protection against
shell metacharacter hacks.
.IP
Example: \fCcommand: cat mydat | uniq -c\fR

.ig >>
<br><br>
.>>

.LP
\fBcommandmr\fR
.ig >>
&nbsp; &nbsp;
.>>
\fImulti-row shell command\fR
.IP \0
Same as \fCcommand\fR but shell command can occupy multiple rows, and is is terminated by a blank line.  
Command can be up to ~1000 characters long.
.IP
Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command.
If this must be done, use #shell / #endshell along with \fCdata:\fR, which provides some protection against
shell metacharacter hacks.
.IP
You can overlay flow-of-control directives as in this example:
.nf
 \0 commandmr:  cat foo.dat |
 \0             process_step1 |
 \0     #if @sortmode = 1
 \0             sort +2 -3
 \0     #else
 \0             cat -
 \0     #endif
 \0
.fi

.ig >>
<br><br>
.>>

.LP
\fBstandardinput\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
If \fCyes\fR, data stream is read from the ploticus standard input.


.ig >>
<br><br>
.>>

.LP
\fBsql\fR
.ig >>
&nbsp; &nbsp;
.>>
\fIsql command\fR
.IP \0
Submit the given sql command and capture the results as tabular data,
using sql result column names as the field names.
This is currently available only for builds related to the
.ig >>
<a href="http://quisp.sourceforge.net">
.>>
\0QUISP/SHSQL
.ig >>
</a>
.>>
package.

.ig >>
<br><br>
.>>

.LP
\fBsqlmr\fR
.ig >>
&nbsp; &nbsp;
.>>
\fImulti-row sql command\fR
.IP \0
Same as \fCsql\fR but SQL command can occupy multiple rows, and is is terminated by a blank line.
Command can be up to ~1000 characters long.
You can overlay flow-of-control directives as in this example:
.nf
 \0 sqlmr: select id, lastname, firstname, balance
 \0        from accounts
 \0   #if @mode = 1
 \0        where acct_type = g
 \0   #endif
 \0     
.fi
.LP
\fB#intrailer\fR
.IP \0
Indicates that a \fCdata\fR attribute
will be given in a #proc trailer, at the end of the script file.
See EXAMPLES, below.


.ig >>
<br><br><br>
.>>

.SH Characteristics of the data stream
.LP
\fBdelim\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCspacequoted\fR | \fCwhitespace\fR | \fCcomma\fR | \fCtab\fR
.IP \0
The type of delimiting method to be used when 
parsing the data.  \fCspacequoted\fR is the default (\fCspace\fR is
equivalent to \fCspacequoted\fR).  See
.ig >>
<a href="dataformat.html">
.>>
\0dataformat
.ig >>
</a>
.>>
for details.
.br
Example: \fCdelim: comma\fR

.ig >>
<br><br>
.>>

.LP
\fBnfields\fR  
.ig >>
&nbsp; &nbsp;
.>>
\fIn\fR
.IP \0
If specified, this sets the expected number of fields per record.
If a data row has more than the expected number of fields, extra fields are silently ignored. 
If a data row has less than the expected number of fields, blank fields are silently added.
This is applied after any \fCfilter\fR processing.
If not specified, the first non-comment non-header row encountered will set the expected number of fields.

.ig >>
<br><br>
.>>

.LP
\fBcommentchar\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#string">
.>>
\0string
.ig >>
</a>
.>>
.IP \0
A character or group of characters that is used to signify a comment in the data file.
Commented lines will be skipped.  Default is \fC//\fR.
Example: \fCcommentchar: #\fR



.ig >>
<br><br><br>
.>>

.SH Setting field names

.LP
\fBfieldnameheader\fR  
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
Allows field names to be embedded in the input data.
If \fCyes\fR, the first non-comment line in the data is expected to
contain field names.  This line is not considered part of the data.
The field name header should be delimited like the rest of the data.
Field names may not contain embedded white space, commas, or quote characters
but (2.30+) there is a way to encode spaces and commas... see
.ig >>
<a href="settings.html#encodenames">
.>>
\0proc settings encodenames.
.ig >>
</a>
.>>


.ig >>
<br><br>
.>>
 
.LP
\fBfieldnames\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fInamelist\fR
.IP \0
Specify field names for input data fields.
\fInamelist\fR is a list delimited by spaces and/or commas.
Names may include any alphanumeric characters with a maximum length of 48, and are case-insensitive.
Field names may not contain embedded spaces or commas, 
but (2.30+) you can encode them... see
.ig >>
<a href="settings.html#encodenames">
.>>
\0proc settings encodenames.
.ig >>
</a>
.>>
Note: if you are using a \fCfilter\fR (see below) you may want to use \fCpf_fieldnames\fR
(see below) to name the result fields.

.br
Example: \fCfieldnames: date group n\fR

.ig >>
<br><br>
.>>

.LP
\fBfieldnamerows\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#text">
.>>
\0multiline text
.ig >>
</a>
.>>
.IP \0
Same as \fCfieldnames\fR (see above), except that field names are given one per line.
Must be terminated by a blank line.
Example: 
.br
.nf
fieldnamerows:
  id
  type
  age
  sex
.fi


.LP
\fBpf_fieldnames\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fInamelist\fR
.IP \0
Assign new field names to \fCfilter\fR result.
See \fCfilter\fR attribute described below.
Useful when the filter result has a different logical record format than the input.
See also \fCfieldnames\fR above.
.br
Example: \fCpf_fieldnames: date z sum1 sum2\fR

.ig >>
<br><br><br>
.>>

.SH Development & debugging
.LP
\fBshowresults\fR 
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
If \fCyes\fR, the results, after any selecting and/or filtering, are
written to the diagnostic file,
which may be useful in debugging, etc.


.ig >>
<br><br><br>
.>>

.SH Selecting and manipulating input rows

.LP
\fBselect\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="select.html">
.>>
\0select expression
.ig >>
</a>
.>>
.IP \0
This allows certain data records to be selected for inclusion based
upon a 
.ig >>
<a href="select.html">
.>>
\0selection expression.
.ig >>
</a>
.>>
Incoming data fields are referenced by number,
using a double at-sign (\fC@@\fR) prefix.  
Hint: use the \fCshowresults\fR attribute when debugging.
.br
Example: \fCselect: @@3 = g\fR
.br
This would select all data records having 3rd field equal to \fCg\fR.

.ig >>
<br><br>
.>>

.LP
\fBrotate\fR  
.ig >>
&nbsp; &nbsp;
.>>
\fCyes\fR | \fCno\fR
.IP \0
Allows data to be given all in one row, even when plotting proc
expects one record per instance (which most do).
Only applicable data set has one row.
To rotate more than one row, use 
.ig >>
<a href="processdata.html">
.>>
\0proc processdata
.ig >>
</a>
.>>
with \fCaction: rotate\fR.

.ig >>
<br><br>
.>>

.LP
\fBfilter\fR 
.ig >>
&nbsp; &nbsp;
.>>
.ig >>
<a href="attributetypes.html#text">
.>>
\0multiline text
.ig >>
</a>
.>>
.IP \0
An embedded script which allows flexible processing to be applied to 
incoming data records one at a time.
Typical uses are for
concatenating or splitting fields, doing on-the-fly date conversions,
or generating derived fields such as the sum of several fields or the difference 
between two fields.
.br
.br
.IP \0
The embedded script will be applied once to every incoming data record.
The script should produce some "output"; generally the last statement is a
\fB##print\fR.  
The output must use the same delimitation method as the input.
The output may have a different logical record format than the input.
If you are using field names, 
the \fCpf_fieldnames\fR attribute (see above) may be used to name the
filter result fields when result record format differs from that of the input.
.IP \0
The script uses the same syntax as the greater ploticus script, except that:
.RS
.IP \(bu
directives must begin with two pound signs (\fC##\fR) instead of one
.IP \(bu
local variables begin with two at signs (\fC@@\fR) instead of one
.IP \(bu
fields on the incoming data record are accessed like this: \fC@@1\fR for
the first field, \fC@@2\fR for the second, etc.  If you are using field
names, these may be used as well, eg: \fC@@score\fR.
.IP \(bu
the only directives that may be used are 
\fB##set\fR, \fB##if\fR, \fB##elseif\fR, \fB##else\fR, \fB##print\fR, \fB##call\fR,
and \fB##exit\fR
.RE
.IP \0
Other things worth noting:
.RS
.IP \(bu
the filter script is terminated using a blank line.
.IP \(bu
use the \fCshowresults\fR attribute when debugging.
.IP \(bu
if \fCfilter\fR is used along with \fBselect\fR, the \fCselect\fR is applied first.  
.IP \(bu 
\fCfilter\fR cannot be used on data specified using the \fCdata\fR attribute.
.IP \(bu
\fCfilter\fR can only access fields from one data record at a time.
.IP \(bu
variables from the ploticus script may be referenced within
the \fCfilter\fR script (use one at-sign \fC@\fR).
Evaluation occurs before the filter script executes.
.IP \(bu
ploticus script #if/#else statements (single pound sign)
may be used to selectively execute portions of the filter script.
Interpretation occurs before the filter script executes.
.IP 
Example: This filters out data records
having field 2 or field 3 equal to M.  It then
calulates the difference in days between two dates
and puts this difference in the variable DIFF.
Finally it "prints" incoming field 1 along with DIFF.
Thus the result of this #proc getdata
will have be data records having two fields.
.nf
.ft C
\0filter:
\0     ##if @@2 = M || @@3 = M
\0       ##exit
\0     ##endif
\0     ##set DIFF = $daysdiff(@@3,@@2)
\0     ##print @@1 @@DIFF
.fi
.ft R
.IP 
There are several more \fCfilter\fR examples in the 
.ig >>
<a href="faq.html#filter">
.>>
\0FAQ
.ig >>
</a>
.>>

.ig >>
<br><br><br>
.>>

.SH More examples
.LP
Data specification may be located at the end of the script file
by using \fB#intrailer\fR and \fB#proc trailer\fR.  This may be
useful in "getting the data out of the way", or with automated building
of script files.
Here is how this is done:
.nf
.ft C
\0#proc getdata
\0#intrailer

\fIother #procs, etc.\fR

\0#proc trailer
\0Data:	0.3 0.5 2.3
	3.5 9.4 1.4
	\fI..etc..\fR
\fIend of file\fR

.ig >>
<br><br>
.>>

.SH Variables that are set by proc getdata
.LP
\fBNRECORDS\fR = the number of data rows gotten
.LP
\fBNFIELDS\fR = the number of fields per record

.ig >>
<br><br><br>
.>>

.SH Hints
.LP
It is possible to
.ig >>
<a href="dataformat.html#set">
.>>
\0set working variables from within the data file.
.ig >>
</a>
.>>
.LP
During debugging, set \fCshowresults: yes\fR in order to see the data after it is read and parsed.
Especially useful when working with \fCfilter\fR.
.LP
In dynamic content environments it's good to gracefully handle the 
situation of an empty data file or command that produced no output.  Example:
.IP \0
.nf
\0#proc getdata
\0   ...
\0
\0#proc endproc
\0#if @NRECORDS = 0
\0  #proc annotate
\0  location: 3 3
\0  text: No data found.
\0
\0  #exit
\0#endif
.fi



.ig >>
<br>
<br>
</td></tr>
<td align=right>
<a href="../doc/welcome.html">
<img src="../doc/ploticus.gif" border=0></a><br><small>data display engine &nbsp; <br>
<a href="../doc/Copyright.html">Copyright Steve Grubb</a>
<br>
<br>
<center>
<img src="../gallery/all.gif"> 
</center>
</td></tr>
</table>
<br>
<center>
Ploticus is hosted at http://ploticus.sourceforge.net <br>
<img src="http://sourceforge.net/sflogo.php?group_id=38453" width="88" height="31" border="0" alt="SourceForge Logo">
</center>
.>>