@c $Id: analyzing-text.texinfo,v 1.20 2001/06/04 15:28:56 m Exp m $
@node    Analyzing Text, Formatting Text, Grammar, Text
@comment node-name,     next,	previous,	up
@chapter Analyzing Text
@cindex analyzing text
@cindex text, analyzing

@noindent
There are many ways to use command-line tools to analyze text in various
ways, such as counting the number of words in a text, creating a
concordance, and comparing texts to see if (and where) they
differ. There are also other tricks you can do with text that count as
analysis, such as finding anagrams and palindromes, or cutting up text
to generate unexpected combinations of words. This chapter covers all
these topics.

@menu
* Counting Text::               Counting words, lines and characters.
* Text Concordance::            Making a text concordance.
* Relevant Text::               Finding similar or relevant text.
* Anagrams::                    Finding anagrams.
* Palindromes::                 Finding palindromes.
* Text Cutups::                 Cutting up text.
@end menu

@node Counting Text, Text Concordance, Analyzing Text, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Counting Text
@cindex counting text
@cindex text, counting
@pindex wc
@pindex style

@noindent
Use the ``word count'' tool, @code{wc}, to count characters, words, and
lines in text.

Give the name of a file as an argument; if none is given, @code{wc}
works on standard input. By default, @code{wc} outputs three
columns, displaying the counts for lines, words, and characters in the
text.

@itemize @bullet
@item
To output the number of lines, words, and characters in file
@file{outline}, type:

@example
$ @kbd{wc outline @key{RET}}
@end example
@end itemize

The following subsections describe how to specify just one kind of count
with @code{wc}, and how to count text in Emacs.

@sp .25
@noindent
@strong{NOTE:} You can get a count of how many @emph{different} words
are in a text, too---see @ref{Text Concordance, , Making a Concordance
of a Text}. To count the average @emph{length} of words, sentences, and
paragraphs, use @code{style} (@pxref{Writing Style, , Checking Text for
Readability}).

@menu
* Counting Characters::         Counting characters.
* Counting Words::              Counting words.
* Counting Lines::              Counting lines.
* Counting Number::             Counting the number of some text string.
* Counting Emacs::              Counting pages in Emacs.
@end menu


@node Counting Characters, Counting Words, Counting Text, Counting Text
@comment  node-name,  next,  previous,  up
@subsection Counting the Characters in a Text
@cindex counting the characters in a text
@cindex text, counting the characters in a
@cindex characters, counting in a text
@pindex wc

@noindent
Use @code{wc} with the @samp{-c} option to specify that just the number
of characters be counted and output.

@itemize @bullet
@item
To output the number of characters in file @file{classified.ad}, type:

@example
$ @kbd{wc -c classified.ad @key{RET}}
@end example
@end itemize


@node Counting Words, Counting Lines, Counting Characters, Counting Text
@comment  node-name,  next,  previous,  up
@subsection Counting the Words in a Text
@cindex counting the words in a text
@cindex text, counting the words in a
@cindex words, counting in a text
@pindex cat
@pindex wc

@noindent
Use @code{wc} with the @samp{-w} option to specify that just the number
of words be counted and output.

@itemize @bullet
@item
To output the number of words in the file @file{story}, type:

@example
$ @kbd{wc -w story @key{RET}}
@end example
@end itemize

To output counts for several files, first concatenate the files with
@code{cat}, and then pipe the output to @code{wc}.

@itemize @bullet
@item
To output the combined number of words for all the files with a
@file{.txt} file name extension in the current directory, type:

@example
$ @kbd{cat *.txt | wc -w @key{RET}}
@end example
@end itemize

@sp .25
@noindent
@strong{NOTE:} To read more about concatenation with @code{cat}, see
@ref{Concatenating Text, , Concatenating Text}. 


@node Counting Lines, Counting Number, Counting Words, Counting Text
@comment  node-name,  next,  previous,  up
@subsection Counting the Lines in a Text
@cindex counting the lines in a text
@cindex text, counting the lines in a
@cindex lines, counting in a text
@pindex wc

@noindent
Use @code{wc} with the @samp{-l} option to specify that just the number
of lines be counted and output.

@itemize @bullet
@item
To output the number of lines in the file @file{outline}, type:

@example
$ @kbd{wc -l outline @key{RET}}
@end example
@end itemize


@node Counting Number, Counting Emacs, Counting Lines, Counting Text
@comment  node-name,  next,  previous,  up
@subsection Counting the Occurrences of Something
@cindex counting the occurrences of something
@cindex occurrences of something, counting the
@pindex grep
@pindex wc

@noindent
To find the number of occurrences of some text string or pattern in a
file or files, use @code{grep} to search the file(s) for the text
string, and pipe the output to @code{wc} with the @samp{-l} option.

@itemize @bullet
@item
To find the number of lines in the file @file{outline} that contain the
string @samp{chapter}, type:

@example
$ @kbd{grep chapter outline | wc -l @key{RET}}
@end example
@end itemize

@sp .25
@noindent
@strong{NOTE:} For more recipes for searching text, and more about
@code{grep}, see @ref{Searching Text, , Searching Text}.


@node Counting Emacs, , Counting Number, Counting Text
@comment  node-name,  next,  previous,  up
@subsection Counting Lines per Page in Emacs
@cindex counting lines per page in Emacs
@cindex lines per page, counting in Emacs
@cindex Emacs, counting lines per page in
@pindex emacs

@noindent
The @code{count-lines-page} function in Emacs outputs in the minibuffer
the number of lines on the current @dfn{page} (as delimited by pagebreak
characters, if any---@pxref{Paginating Text, , Paginating Text}),
followed by the number of lines in the buffer before the line that point
is on, and the number of lines in the buffer after point.

@itemize @bullet
@item
To count the number of lines per page in the current buffer in Emacs,
type:

@example
@kbd{C-x l}
@end example
@end itemize

Emacs outputs the number of lines per page of the current buffer in the
echo area.

For example, if the output in the minibuffer is

@example
Page has 351 lines (69 + 283)
@end example
@noindent
this means that the current page contains 351 lines, and point is on
line number 70---there are 69 lines before this line, and 283 lines
after this line.


@node Text Concordance, Relevant Text, Counting Text, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Making a Concordance of a Text
@cindex making a concordance of a text
@cindex text, making a concordance of a
@cindex concordance, making of a text
@cindex text, counting unique words in
@pindex tr
@pindex sort
@pindex uniq
@pindex wc

@noindent
A @dfn{concordance} is an index of all the words in a text, along with
their contexts. A concordance-like functionality---an alphabetical
listing of all words in a text and their frequency---can be made fairly
easily with some basic shell tools: @code{tr}, @code{sort}, and
@code{uniq}.

@itemize @bullet
@item
To output a word-frequency list of the text file @file{naked_lunch},
type:

@example
$ @kbd{tr ' ' ' @key{RET}}
> @kbd{' < naked_lunch | sort | uniq -c @key{RET}}
@end example
@end itemize

These commands @emph{translate} all space characters to newline
characters, outputting the text with each word on its own line; this is
then sorted alphabetically, and that output is passed to @code{uniq},
which outputs only the @emph{unique} lines---that is, all non-duplicate
lines---while the @samp{-c} option precedes each line with its
@emph{count} (the number of times it occurs in the text).

To get a word frequency count---that is, the total number of
@emph{different} words in a text---just pipe the output of the frequency
list to @code{wc} with the @samp{-l} option. This counts all the lines
of its input, which in this case will be the list of unique words, one
per line.

@itemize @bullet
@item
To output a count of the number of unique words in the text file
@file{naked_lunch}, type:

@example
$ @kbd{tr ' ' ' @key{RET}}
> @kbd{' < naked_lunch | sort | uniq -c | wc -l @key{RET}}
@end example
@end itemize


@node Relevant Text, Anagrams, Text Concordance, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Text Relevance
@cindex text relevance
@cindex relevant text, finding
@pindex diff

@noindent
The following recipes show how to analyze a given text for its relevancy
to other text, either to keywords or to whole files of text.

You can also use the @code{diff} family of tools to analyze differences
in text; those tools are especially good for comparing different
revisions of the same file (@pxref{Comparing Files, , Comparing Files}).

@menu
* Relevance Sorting::           Sorting text by relevance.
* Remembrance Agent::           An Emacs tool for finding relevant text.
@end menu

@node Relevance Sorting, Remembrance Agent, Relevant Text, Relevant Text
@comment  node-name,  next,  previous,  up
@subsection Sorting Text in Order of Relevance
@cindex sorting text in order of relevance
@cindex text, sorting in order of relevance
@pindex rel

@flushleft
@sf{Debian}: @file{rel}
@sf{WWW}: @url{http://www.johncon.com/}
@end flushleft
@*
@noindent
Use @code{rel} to analyze text files for relevance to a given set of
keywords. It outputs the names of those files that are relevant to the
given keywords, ranked in order of relevance; if a file does not meet
the criteria, it is not output in the relevance listing.

@code{rel} takes as an option the keyword to search for in quotes; you
can build a boolean expression by grouping multiple keywords in
parentheses and using any of the following operators between them:

@multitable @columnfractions .30 .70
@item @sc{Operator}
@tab @sc{Description}
@item @code{|}
@tab Logical ``or.''

@item @code{&}
@tab Logical ``and.''

@item @code{!}
@tab Logical ``not.''
@end multitable

Give as arguments the names of the files to rank.

@itemize @bullet
@item
To rank the files @file{report.a}, @file{report.b}, and @file{report.c}
in order of relevance to the keywords @samp{saving} and @samp{profit},
type:

@example
$ @kbd{rel "(saving & profit)" report.a report.b report.c @key{RET}}
@end example
@end itemize

Give the name of a directory tree to analyze all files in the directory
tree.

@itemize @bullet
@item
To output a list of any files containing either @samp{invitation} or
@samp{request} in the @file{~/mail} directory, ranked in order of
relevancy, type:

@example
$ @kbd{rel "(invitation | request)" ~/mail @key{RET}}
@end example

@item
To output a list of any files containing @samp{invitation} and not
@samp{wedding} in the @file{~/mail} directory, ranked in order of
relevancy, type:

@example
$ @kbd{rel "(invitation ! wedding)" ~/mail @key{RET}}
@end example

@item
To output a list of any files containing @samp{invitation} and
@samp{party} in the @file{~/mail} directory, ranked in order of 
relevancy, type:

@example
$ @kbd{rel "(invitation & party)" ~/mail @key{RET}}
@end example
@end itemize


@node Remembrance Agent, , Relevance Sorting, Relevant Text
@comment  node-name,  next,  previous,  up
@subsection Listing Relevant Files in Emacs
@cindex listing relevant files in Emacs
@cindex Emacs, listing relevant files in 
@pindex remembrance-agent
@pindex Emacs

@flushleft
@sf{Debian}: @file{remembrance-agent}
@sf{WWW}: @url{http://www.media.mit.edu/~rhodes/RA/}
@end flushleft
@*
@noindent
The purpose of the Remembrance Agent is to analyze the text you type in
an Emacs session and, in the background, find similar or relevant
passages of text within your other files. It then outputs in a smaller
window a list of suggestions---those files that it has found---which you
can open in a new buffer.

When installing the Remembrance Agent, you create three databases of the
files to use when making relevance suggestions; when
@code{remembrance-agent} is running, it searches these three databases
in parallel, looking for relevant text. You could create, for example,
one database of saved email, one of your own writings, and one of saved
documents.

@itemize @bullet
@item
To toggle the Remembrance Agent in the current buffer, type:

@example
@kbd{C-c r t}
@end example
@end itemize

When @code{remembrance-agent} is running, suggested buffers will be
displayed in the small @file{*Remembrance*} buffer at the bottom of the
screen. To open a suggestion in a new buffer, type @kbd{C-c r
@var{number}}, where @var{number} is the number of the suggestion.

@itemize @bullet
@item
To open the second suggested file in a new buffer, type:

@example
@kbd{C-c r 2}
@end example
@end itemize


@node Anagrams, Palindromes, Relevant Text, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Finding Anagrams in Text
@cindex finding anagrams in text
@cindex anagrams, finding in text
@cindex text, finding anagrams in 
@pindex an

@flushleft
@sf{Debian}: @file{an}
@end flushleft
@*
@noindent
An @dfn{anagram} is a word or phrase whose characters consist entirely
of all the characters of a given word or phrase---for example,
@samp{stop} and @samp{tops} are both anagrams of @samp{pots}.

Use @code{an} to find and output anagrams. Give as an argument the word
or quoted phrase to use; @code{an} writes its results to the standard
output.

@itemize @bullet
@item
To output all anagrams of the word @samp{lake}, type:

@example
$ @kbd{an lake @key{RET}}
@end example

@item
To output all anagrams of the phrase @samp{lakes and oceans}, type:

@example
$ @kbd{an 'lakes and oceans' @key{RET}}
@end example
@end itemize

To limit the anagrams output to those containing a given string,
specify that string with the @samp{-c} option.

@itemize @bullet
@item
To output only anagrams of the phrase @samp{lakes and oceans} which
contain the string @samp{seas}, type:

@example
$ @kbd{an -c seas 'lakes and oceans' @key{RET}}
@end example
@end itemize

To print all of the words that some or all letters in a given word or
phrase can make, use the @samp{-w} option. This outputs words that are
not anagrams, since anagrams must contain @emph{all} of the letters of
the other word or phrase.

@itemize @bullet
@item
To output all of the words that can be made from the letters of the word
@samp{seas}, type:

@example
$ @kbd{an -w seas @key{RET}}
@end example
@end itemize

This command outputs all of the words that can be formed from all or
some of the characters in @samp{seas}, including @samp{see} and
@samp{as}.


@node Palindromes, Text Cutups, Anagrams, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Finding Palindromes in Text
@cindex finding palindromes in text
@cindex palindromes, finding in text
@cindex text, finding palindromes in 
@pindex perl

@noindent
A @dfn{palindrome} is a word that reads the same both forwards and
backwards; for example, ``Mom,'' ``madam,'' and ``nun'' are all
palindromes.

To find palindromes in a file, use this simple Perl ``one-liner,'' and
substitute @var{file} for the name of the file to check:

@example
perl -lne 'print if $_ eq reverse' @var{file}
@end example

To check for palindromes in the standard input, specify @samp{-} as the
file name to check. This is useful for putting at the end of a pipeline.

@itemize @bullet
@item
To output all of the palindromes in the system dictionary, type:

@example
$ @kbd{perl -lne 'print if $_ eq reverse' /usr/dict/words @key{RET}}
@end example
@end itemize


@node Text Cutups, , Palindromes, Analyzing Text
@comment  node-name,  next,  previous,  up
@section Text Cut-Ups
@cindex text cut-ups
@cindex cut-ups, text
@cindex Burroughs, William S.
@cindex Gysin, Brion
@cindex @cite{Third Mind, The}

@noindent
A @dfn{cut-up} is a random rearrangement of a physical layout of text,
made with the intention of finding unique or interesting phrases in the
rearrangement. Software for rearranging text in random ways has existed
since the earliest text-processing tools; the popularity of these tools
will never die.

The cut-up technique in literature was discovered by painter Brion Gysin
and American writer William S. Burroughs in 1959; they believed it
brought the montage technique of painting to the written word.

``All writing is in fact cut-ups,'' Burroughs wrote.@footnote{In
@cite{The Third Mind}, by William S. Burroughs and Brion Gysin.} ``A
collage of words read heard overheard @dots{} [u]se of scissors renders
the process explicit and subject to extension and variation.''

These recipes describe a few of the common ways to make text cut-ups;
more free software tools for making cut-ups are listed at
@url{http://dsl.org/comp/cutups.shtml}.

@menu
* Simple Cutups::               Simple text cutups.
* Advanced Cutups::             Advanced text cutups.
* Emacs Cutups::                Cutups in Emacs.
@end menu


@node Simple Cutups, Advanced Cutups, Text Cutups, Text Cutups
@comment  node-name,  next,  previous,  up
@subsection Making Simple Text Cut-Ups
@cindex making simple text cut-ups
@cindex cut-ups, making simple text
@pindex cutup

@flushleft
@sf{WWW}: @url{http://dsl.org/comp/tinyutils/}
@end flushleft
@*
@noindent
To perform a simple cut-up of a text, use @code{cutup}. It takes the
name of a file as input and cuts it both horizontally and vertically
along the middle, rearranges the four sections to their diagonally
opposite corners, and then writes that cut-up to the standard
output. The original file is not modified.  

@itemize @bullet
@item
To make a cut-up from a file called @file{nova}, type:

@example
$ @kbd{cutup nova @key{RET}}
@end example
@end itemize


@node Advanced Cutups, Emacs Cutups, Simple Cutups, Text Cutups
@comment  node-name,  next,  previous,  up
@subsection Making Random Word Cut-Ups
@cindex making random word cut-ups
@cindex cut-ups, making random word
@cindex Zawinski, Jamie
@pindex dadadodo

@flushleft
@sf{Debian}: @file{dadadodo}
@sf{WWW}: @url{http://www.jwz.org/dadadodo/}
@end flushleft
@*
@noindent
No simple cut-up filter, Jamie Zawinski's @code{dadadodo} uses the
computer to go one step beyond---it generates passages of random text
whose structure and characters are similar to the text input you give
it. The program works better on larger texts, where more subtleties can
be analyzed and hence more realistic-looking text is output.

Give as an argument the name of the text file to be used; by default,
@code{dadadodo} outputs text to standard output until you interrupt it
by typing @kbd{C-c}.

@itemize @bullet
@item
To output random text based on the text in the file @file{nova},
type:

@example
$ @kbd{dadadodo nova @key{RET}}
@end example
@end itemize

This command will output passages of random text based on the text in
the file @file{nova} until it is interrupted by the user.

You can analyze a text and save the analysis to a file of compiled data;
this analysis can then be used to generate random text when the original
input text is not present. The following table describes this and other
@code{dadadodo} options.

@multitable @columnfractions .30 .70
@item @sc{Option}
@tab @sc{Description}
@item @code{-c @var{integer}}
@tab Generate @var{integer} sentences (default is 0, meaning ``generate
an infinite amount until interrupted'').

@item @code{-l @var{file}}
@tab Load compiled data in @var{file} and use it to generate text.

@item @code{-o @var{file}}
@tab Output compiled data to file @var{file} for later use.

@item @code{-p @var{integer}}
@tab Pause for @var{integer} seconds between paragraphs.
@end multitable


@node Emacs Cutups, , Advanced Cutups, Text Cutups
@comment  node-name,  next,  previous,  up
@subsection Making Cut-Ups in Emacs
@cindex making cut-ups in Emacs
@cindex Emacs, making cut-ups in 
@cindex cut-ups, making in emacs
@pindex emacs
@pindex dissociated-press

@noindent
The @code{dissociated-press} function in Emacs makes random cut-ups of
the current buffer in a new buffer called @file{*Dissociation*}; the
original buffer is not modified. The text in the new buffer is generated
by combining random portions of the buffer by overlapping characters or
words, thus (usually) creating plausible-sounding sentences. It pauses
occasionally and asks whether or not you want to continue the
dissociation.

@itemize @bullet
@item
To generate a Dissociated Press cut-up from the current buffer, type:

@example
@kbd{M-x dissociated-press @key{RET}}
@end example
@end itemize

Give a positive argument to the @code{dissociated-press} function to
specify the number of characters to use for overlap; give a negative
argument to specify the number of @emph{words} for overlap.

@itemize @bullet
@item
To generate a Dissociated Press cut-up from the current buffer,
always overlapping by three characters, type:

@example
@kbd{C-u 3 M-x dissociated-press @key{RET}}
@end example

@item
To generate a Dissociated Press cut-up from the current buffer,
always overlapping by one word, type:

@example
@kbd{C-u -1 M-x dissociated-press @key{RET}}
@end example
@end itemize