<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article
      PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN"
      "http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd">

<article  lang="en" xml:lang="en" >
  <articleinfo>
    <title>XMerL - XML processing tools for Erlang</title>
    <subtitle>Reference Manual</subtitle>
    <authorgroup>
      <author>
	<firstname>Ulf</firstname>
	<surname>Wiger</surname>
      </author>
    </authorgroup>
    <revhistory>
      <revision>
      <revnumber>1.0</revnumber><date>2003-02-04</date>
      <revremark>Converted xml from html</revremark>
      </revision>
    </revhistory>
    <abstract>
      <para>XMerL tools contains xmerl_scan; a non-validating XML
      processor, xmerl_xpath; a XPath implementation, xmerl for export
      of XML trees to HTML, XML or text and xmerl_xs for XSLT like
      transforms in erlang. 
      </para>
    </abstract>
  </articleinfo>
  
  <section>
    <title>xmerl_scan - the XML processor</title>
    <para>The (non-validating) XML processor is activated through 
    <computeroutput>xmerl_scan:string/[1,2]</computeroutput> or 
    <computeroutput>xmerl_scan:file/[1,2]</computeroutput>.
    It returns records of the type defined in xmerl.hrl.
    </para>
  
    <para>As far as I can tell, xmerl_scan implements the complete XML
    1.0 spec, including:</para>
    <itemizedlist>
      <listitem><para>entity expansion</para></listitem>
      <listitem><para>fetching and parsing external DTDs</para></listitem>
      <listitem><para>contitional processing</para></listitem>
      <listitem><para>UniCode</para></listitem>
      <listitem><para>XML Names</para></listitem>
    </itemizedlist>
    <programlisting>
xmerl_scan:string(Text [ , Options ]) -> #xmlElement{}.
xmerl_scan:file(Filename [ , Options ]) -> #xmlElement{}. </programlisting>
    
    <para>The Options are basically to specify the behaviour of the
    scanner.  See the source code for details, but you can specify
    funs to handle scanner events (event_fun), process the document
    entities once identified (hook_fun), and decide what to do if the
    scanner runs into eof before the document is complete
    (continuation_fun).</para>

    <para>You can also specify a path (fetch_path) as a list of
    directories to search when fetching files. If the file in question
    is not in the fetch_path, the URI will be used as a file
    name.</para>


    <section>
      <title>Customization functions</title>
      <para>The XML processor offers a number of hooks for
      customization. These hooks are defined as function objects, and
      can be provided by the caller.</para>
      
      <para>The following customization functions are available. If
      they also have access to their own state variable, the access
      function for this state is identified within parentheses:</para>

      <itemizedlist>

	<listitem><para>event function (<computeroutput>
	xmerl_scan:event_state/[1,2]
	</computeroutput>)</para></listitem>

	<listitem><para>hook function (<computeroutput>
	xmerl_scan:hook_state/[1,2]
	</computeroutput>)</para></listitem>

	<listitem><para>fetch function (<computeroutput>
	xmerl_scan:fetch_state/[1,2] </computeroutput>)
	</para></listitem>

	<listitem><para>continuation function (<computeroutput>
	xmerl_scan:cont_state/[1,2] </computeroutput>)
	</para></listitem>

	<listitem><para>rules function (<computeroutput>
      xmerl_scan:rules_state/[1,2] </computeroutput>)
      </para></listitem>

	<listitem><para>accumulator function</para></listitem>

	<listitem><para>close function</para></listitem>

      </itemizedlist>

      <para>For all of the above state access functions, the function
      with one argument
      (e.g. <computeroutput>event_fun(GlobalState)</computeroutput>)
      will read the state variable, while the function with two
      arguments (e.g.: <computeroutput>event_fun(NewStateData,
      GlobalState)</computeroutput>) will modify it.</para>

      <para>For each function, the description starts with the syntax
      for specifying the function in the
      <computeroutput>Options</computeroutput> list. The general forms
      are <computeroutput>{Tag, Fun}</computeroutput>, or
      <computeroutput>{Tag, Fun, LocalState}</computeroutput>. The
      second form can be used to initialize the state variable in
      question.</para>

      <section>
	<title>User State</title>

	<para>All customization functions are free to access a
	&quot;User state&quot; variable. Care must of course be taken
	to coordinate the use of this state. It is recommended that
	functions, which do not really have anything to contribute to
	the &quot;global&quot; user state, use their own state
	variable instead. Another option (used in
	e.g. <computeroutput>xmerl_eventp.erl</computeroutput>) is for
	customization functions to share one of the local states (in
	<computeroutput>xmerl_eventp.erl</computeroutput>, the
	continuation function and the fetch function both access the
	<computeroutput>cont_state</computeroutput>.)</para>

	<para>Functions to access user state:</para>

	<itemizedlist>

	  <listitem><para><computeroutput>
	  xmerl_scan:user_state(GlobalState) </computeroutput>
	  </para></listitem>

	  <listitem><para><computeroutput>xmerl_scan:user_state(UserState',
	  GlobalState) </computeroutput></para></listitem>

	</itemizedlist>

      </section>
      <section>
	<title>Event Function</title>

	<para><computeroutput>{event_fun, fun()} | {event_fun, fun(),
	LocalState}</computeroutput></para>

	<para>The event function is called at the beginning and at the
	end of a parsed entity. It has the following format and
	semantics:</para>

<programlisting>
<![CDATA[
fun(Event, GlobalState) ->
   EventState = xmerl_scan:event_state(GlobalState),
   EventState' = foo(Event, EventState),
   GlobalState' = xmerl_scan:event_state(EventState', GlobalState)
end.
]]></programlisting>

      </section>
      <section>
	<title>Hook Function</title>
	<para> <computeroutput>{hook_fun, fun()} | {hook_fun, fun(),
	LocalState}</computeroutput></para>



<para>The hook function is called when the processor has parsed a complete
entity. Format and semantics:</para>

<programlisting>
<![CDATA[
fun(Entity, GlobalState) ->
   HookState = xmerl_scan:hook_state(GlobalState),
   {TransformedEntity, HookState'} = foo(Entity, HookState),
   GlobalState' = xmerl_scan:hook_state(HookState', GlobalState),
   {TransformedEntity, GlobalState'}
end.
]]></programlisting>

	<para>The relationship between the event function, the hook
	function and the accumulator function is as follows:</para>

	<orderedlist>
	  <listitem><para>The event function is first called with an
	  'ended' event for the parsed entity.</para></listitem>

	  <listitem><para>The hook function is called, possibly
	  re-formatting the entity.</para></listitem>

	  <listitem><para>The acc function is called in order to
	  (optionally) add the re-formatted entity to the contents of
	  its parent element.</para></listitem>

    </orderedlist>

      </section>
      <section>
	<title>Fetch Function</title>
<para>
<computeroutput>{fetch_fun, fun()} | {fetch_fun, fun(), LocalState}</computeroutput>
</para>
<para>The fetch function is called in order to fetch an external resource
(e.g. a DTD).</para>

<para>The fetch function can respond with three different return values:</para>

    <programlisting>
<![CDATA[
    Result ::=
      {ok, GlobalState'} |
      {ok, {file, Filename}, GlobalState'} |
      {ok, {string, String}, GlobalState'}
]]></programlisting>

<para>Format and semantics:</para>

    <programlisting>
<![CDATA[
fun(URI, GlobalState) ->
   FetchState = xmerl_scan:fetch_state(GlobalState),
   Result = foo(URI, FetchState).  % Result being one of the above
end.
]]></programlisting>

      </section>
      <section>
	<title>Continuation Function</title>
<para>
<computeroutput>{continuation_fun, fun()} | {continuation_fun, fun(), LocalState}</computeroutput>
</para>
<para>The continuation function is called when the parser encounters the end
of the byte stream. Format and semantics:</para>

    <programlisting>
<![CDATA[
fun(Continue, Exception, GlobalState) ->
   ContState = xmerl_scan:cont_state(GlobalState),
   {Result, ContState'} = get_more_bytes(ContState),
   GlobalState' = xmerl_scan:cont_state(ContState', GlobalState),
   case Result of
      [] ->
         GlobalState' = xmerl_scan:cont_state(ContState', GlobalState),
         Exception(GlobalState');
      MoreBytes ->
         {MoreBytes', Rest} = end_on_whitespace_char(MoreBytes),
         ContState'' = update_cont_state(Rest, ContState'),
         GlobalState' = xmerl_scan:cont_state(ContState'', GlobalState),
         Continue(MoreBytes', GlobalState')
   end
end.
]]></programlisting>
      </section>
      <section>
	<title>Rules Functions</title>
	<para>
<computeroutput>
{rules, ReadFun : fun(), WriteFun : fun(), LocalState} |
{rules, Table : ets()}</computeroutput>
</para>
	<para>The rules functions take care of storing scanner
	information in a rules database. User-provided rules functions
	may opt to store the information in mnesia, or perhaps in the
	user_state(LocalState).</para>

	<para>The following modes exist:</para>

	<itemizedlist>

	  <listitem><para>If the user doesn't specify an option, the
	  scanner creates an ets table, and uses built-in functions to
	  read and write data to it. When the scanner is done, the ets
	  table is deleted.</para></listitem>

	  <listitem><para>If the user specifies an ets table via the 
	<computeroutput>{rules, Table}</computeroutput> option, the
	scanner uses this table. When the scanner is done, it does
	<emphasis>not</emphasis> delete the table.</para></listitem>
	  
	  <listitem><para>If the user specifies read and write
	  functions, the scanner will use them instead.</para></listitem>

	</itemizedlist>
	
	<para>The format for the read and write functions are as
	follows:</para>


<programlisting>
<![CDATA[
WriteFun(Context, Name, Definition, ScannerState) -> NewScannerState.
ReadFun(Context, Name, ScannerState) -> Definition | undefined.
]]></programlisting>

	<para>Here is a summary of the data objects currently being
	written by the scanner:</para>
	
	<table>
	  <title>Scanner data objects</title>
	  <tgroup cols="3">
	    <thead>
	      <row>
		<entry>Context</entry>
		<entry>Key Value</entry>
		<entry>Definition</entry>
	      </row>
	    </thead>
	    <tbody>
	      <row>
		<entry>notation</entry>
		<entry>NotationName</entry>
		<entry><computeroutput>{system, SL} | {public, PIDL, SL}</computeroutput></entry>
	      </row>
	      <row>
		<entry>elem_def</entry>
		<entry>ElementName</entry>
		<entry><computeroutput>#xmlElement{content = ContentSpec}</computeroutput></entry>
	      </row>
	      <row>
		<entry>parameter_entity</entry>
		<entry>PEName</entry>
		<entry><computeroutput>PEDef</computeroutput></entry>
	      </row>
	      <row>
		<entry>entity</entry>
		<entry>EntityName</entry>
	  <entry><computeroutput>EntityDef</computeroutput></entry>
	      </row>
	    </tbody>
	  </tgroup>
	</table>
      
	
<programlisting>
<![CDATA[
ContentSpec ::= empty | any | ElemContent
ElemContent ::= {Mode, Elems}
Mode        ::= seq | choice
Elems       ::= [Elem]
Elem        ::= '#PCDATA' | Name | ElemContent | {Occurrence, Elems}
Occurrence  ::= '*' | '?' | '+'
]]></programlisting>
	<note><para>When &lt;Elem&gt; is not wrapped with
&lt;Occurrence&gt;, (Occurrence = once) is implied.</para></note>

      </section>
      <section>
	<title>Accumulator Function</title>
	<para><computeroutput>{acc_fun, fun()} | {acc_fun, fun(),
	LocalState}</computeroutput></para>

	<para>The accumulator function is called to accumulate the
	contents of an entity.When parsing very large files, it may
	not be desirable to do so.In this case, an acc function can
	be provided that simply doesn't accumulate.</para>

	<para>Note that it is possible to even modify the parsed
	entity before accumulating it, but this must be done with
	care. <computeroutput>xmerl_scan</computeroutput> performs
	post-processing of the element for namespace management. Thus,
	the element must keep its original structure for this to
	work.</para>

	<para>The acc function has the following format and
	semantics:</para>

	<programlisting>
<![CDATA[
%% default accumulating acc fun
fun(ParsedEntity, Acc, GlobalState) ->
   {[X|Acc], GlobalState}.

%% non-accumulating acc fun
fun(ParsedEntity, Acc, GlobalState) ->
   {Acc, GlobalState}.
]]></programlisting>
      </section>
      <section>
	<title>Close Function</title>

	<para>The close function is called when a document (either the
	main document or an external DTD) has been completely
	parsed. When xmerl_scan was started using
	<computeroutput>xmerl_scan:file/[1,2]</computeroutput>, the
	file will be read in full, and closed immediately, before the
	parsing starts, so when the close function is called, it will
	not need to actually close the file. In this case, the close
	function will be a good place to modify the state
	variables.</para>

	<para>Format and semantics:</para>

	<programlisting>
<![CDATA[
fun(GlobalState) ->
   GlobalState' = ....  % state variables may be altered
]]></programlisting>
      </section>

    </section>

  </section>

  <section>
    <title>XPATH</title>

    <programlisting>
<![CDATA[
xmerl_xpath:string(QueryString, #xmlElement{}) ->
	[DocEntity]

DocEntity :	#xmlElement{} 
		| #xmlAttribute{} 
		| #xmlText{} 
		| #xmlPI{}
		| #xmlComment{}
]]></programlisting>

    <para>The xmerl_xpath module does seem to handle the entire XPATH
    1.0 spec, but I haven't tested that much yet. The grammar is
    defined in
    <computeroutput>xmerl_xpath_parse.yrl</computeroutput>.  The core
    functions are defined in
    <computeroutput>xmerl_xpath_pred.erl</computeroutput>.</para>
  </section>
  <section>
    <title>Some useful shell commands for debugging the XPath parser</title>
<para>
    <command>
<![CDATA[
c(xmerl_xpath_scan).
yecc:yecc("xmerl_xpath_parse.yrl", "xmerl_xpath_parse", true, []).
c(xmerl_xpath_parse).

xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("position() > -1")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6 div 2")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 + 6 mod 2")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("-----6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("parent::node()")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("descendant-or-self::node()")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("parent::processing-instruction('foo')")).]]></command></para>
  </section>
  <section>
    <title>Erlang Data Structure Export</title>

    <para>The idea as follows:</para>

    <para>The Erlang data structure should look like this:</para>
    <programlisting>
<![CDATA[
Element:	{Tag, Attributes, Content}
Tag :		atom()
Attributes:	[{Key, Value}]
Content:	[String | Element]
String:		[char() | binary() | String]
]]></programlisting>

    <para>Some short forms are allowed:</para>
    <programlisting>
<![CDATA[
{Tag, Content}	-> {Tag, [], Content}
Tag		-> {Tag, [], []}
]]></programlisting>

    <para>Note that content lists must be flat, but strings can be
    deep.</para>

    <para>It is also allowed to include normal
    <computeroutput>#xml...</computeroutput> elements in the simple
    format.</para>

    <para><computeroutput>xmerl:export_simple(Data,
    Callback)</computeroutput> takes the above data structure and
    exports it, using the callback module
    <computeroutput>Callback</computeroutput>.</para>

    <para>The callback module should contain hook functions for all
    tags present in the data structure. The hook function must have
    the format:</para>
    <para><computeroutput> Tag(Data, Attrs, Parents, E)
    </computeroutput></para>

    <para>where E is an <computeroutput>#xmlElement{}</computeroutput>
    record  (see <computeroutput>xmerl.hrl</computeroutput>).</para>

    <para>Attrs is converted from the simple <computeroutput>[{Key,
    Value}]</computeroutput> to
    <computeroutput>[#xmlAttribute{}]</computeroutput></para>

    <para>Parents is a list of <computeroutput>[{ParentTag,
    ParentTagPosition}]</computeroutput>.</para>

    <para>The hook function should return either the Data to be
    exported, or the tuple <computeroutput>{'#xml-redefine#',
    NewStructure}</computeroutput>, where
    <computeroutput>NewStructure</computeroutput> is an element (which
    can be simple), or a (simple-) content list wrapped in a 1-tuple
    as <computeroutput>{NewContent}</computeroutput>.</para>

    <para>The callback module can inherit definitions from other
    callback modules, through the required function
    <computeroutput>'#xml-interitance#() ->
    [ModuleName]</computeroutput>. </para>

    <para>As long as a tag is represented in one of the callback
    modules, things will work. It is of course also possible to
    redefine a tag.</para>
      <section>
      <title>XSLT like transforms</title>
	<para>See separate document <ulink url="xmerl_xs.html" >xmerl_xs.html
	</ulink></para>.
      </section>
  </section>

</article>