File: python-runtime.html

package info (click to toggle)
antlr 2.7.7%2Bdfsg-14
links: PTS, VCS
area: main
in suites: forky, sid, trixie
size: 10,016 kB
sloc: java: 54,649; cs: 12,537; makefile: 8,854; cpp: 7,359; pascal: 5,273; sh: 4,333; python: 4,297; lisp: 1,969; xml: 220; lex: 192; ansic: 127
file content (575 lines) | stat: -rw-r--r-- 22,565 bytes
parent folder | download | duplicates (9)
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
  <head>
    <title>Notes for using the ANTLR Python Code Generator</title>
  </head>
  <body bgcolor="#FFFFFF">
    <h1><strong>Python Code Generator for ANTLR 2.7.7</strong></h1>
    <p>As of ANTLR 2.7.5, you can generate your Lexers,
    Parsers and TreeParsers in Python. This feature extends the benefits of
    ANTLR's predicated-LL(k) parsing technology to the Python language and
    platform.</p>
    <p>To be able to build and use the Python language Lexers, Parsers and
    TreeParsers, you will need to have the ANTLR Python runtime library
    installed in your Python path. The Python runtime model is based on the
    existing runtime model for <a href="runtime.html">Java</a> and is thus
    immediately familiar. The Python runtime and the Java runtime are very
    similar although there a number of subtle (and not so subtle)
    differences. Some of these result from differences in the respective
    runtime environments.</p>
    <p>ANTLR Python support was contributed (and is to be maintained) by
    Wolfgang Haefelinger and Marq Kole.</p>
    <h2><a name="#building">Building the ANTLR Python Runtime</a></h2>
    <p>The ANTLR Python runtime source and build files are completely
    integrated in the ANTLR build process.The ANTLR runtime support module
    for Python is located in the <b><code>lib/python</code></b> subdirectory
    of the ANTLR distribution. Installation of the Python runtime support
    is enabled automatically if Python can be found on your system by the
    <tt>configure</tt> script.</p>
    <p>With Python support enabled the current distribution will look for
    the presence of a python executable of version 2.2 or higher. If it has
    found such a beast, it will generate and install the ANTLR Python
    runtime as part of the overall ANTLR building and installation
    process.</p>
    <p>If the python distribution you are using is at an unusual location,
    perhaps because you are using a local installation instead of a
    system-wide one, you can provide the location of that python executable
    using the <tt>--with-python=&lt;path&gt;</tt> option for the configure
    script, for instance:</p>
    <blockquote>
<pre>
./configure --with-python=$HOME/bin/python2.3
</pre>
    </blockquote>
    <p>Also, if the python executable is at a regular location, but has a
    name that differs from &quot;python&quot;, you can specify the correct
    name through the <tt>--with-python=&lt;path&gt;</tt>, as shown above, or through
    environment variable <tt>$PYTHON</tt></p>
    <blockquote>
<pre>
PYTHON=python2.3
export PYTHON
./configure
</pre>
    </blockquote>
    <p>All the example grammars for the ANTLR Python runtime are built when
    ANTLR itself is built. They can be run in one go by running <tt>make
    test</tt> in the same directory where you ran the configure script in
    the ANTLR distribution. So after you've run <tt>configure</tt> you can
    do:</p>
    <blockquote>
<pre>
# Build ANTLR and all examples
make
# Run them
make test
# Install everything
make install
</pre>
</blockquote>

<p>Note that <tt>make install</tt> will not add the ANTLR Python
runtime (i.e. antlr.py) to your Python installation but rather install
antlr.py in <tt>${prefix}/lib</tt>. To be able to use antlr.py you
would need to adjust Python's <tt>sys.path<tt>.</p>

<p>However, there a script is provided that let's you easily add antlr.py
as module to your Python installation. After installation just run</p>
<blockquote>
<pre>
${prefix}/sbin/pyantlr.sh install
</pre>
</blockquote>
<p>Note that usually you need to be superuser in order to succeed. Also
note that you can run this command later at any time again, for
example, if you have a second Python installation etc. Just make sure
that <tt>python</tt> is in your $PATH when running pyantlr.sh.</p>

<p>Note further that you can also do this to install ANTLR Python
runtime immediatly after having called <tt>./configure</tt>:</p> 
<blockquote>
<pre>
scripts/pyantlr.sh install
</pre>
</blockquote>

    <h2><a name="#codegen">Specifying Code Generation</a></h2>
    <p>You can instruct ANTLR to generate your Lexers, Parsers and
    TreeParsers using the Python code generator by adding the following
    entry to the global options section at the beginning of your grammar
    file.</p>
    <blockquote>
<pre>
{
    language="Python";
}
</pre>
    </blockquote>
    <p>After that things are pretty much the same as in the default
    <b>java</b> code generation mode. See the examples in
    <code>examples/python</code> for some illustrations.</p>
    <p>One particular issue that is worth mentioning is the handling of
    comments in ANTLR Python. Java, C++, and C# all use the same lexical
    structures to define comments: <code>//</code> for single-line
    comments, and <code>/* ... */</code> for block comments. Unfortunately,
    Python does not handle comments this way. It only knows about
    single-line comments, and these start off with a <code>#</code>
    symbol.</p>
    <p>Normally, all comments outside of actions are actually comments in
    the ANTLR input language. These comments, and that is both block
    comments and single-line comments are translated into Python
    single-line comments.</p>
    <p>Secondly, all comments inside actions should be comments in the
    target language, Python in this case. Unfortunately, if the actions
    contain ANTLR actions, such as <code>$getText</code>, the code
    generator seems to choke on Python comments as the <code>#</code> sign
    is also used in tree construction. The solution is to use Java/C++-style
    comments in all actions; these will be translated into Python comments
    by the ANTLR as it checks these actions for the presence of predefined
    action symbols such as <code>$getText</code>.</p>
    <p>So, as a general issue: all comments in an ANTLR grammar for the
    Python target should be in Java/C++ style, not in Python style.</p>
    <h2><a name="#sections">Python-Specific ANTLR Sections</a></h2>
    <ul>
      <li>
        <b>header - specify additional <code>import</code> directives</b> 
        <p>You can instruct the ANTLR Python code generator to import
        additional Python packages in your generated
        Lexer/Parser/TreeParser by adding code to the header section which
        must be the first section at the beginning of your ANTLR grammar
        file, apart from any other header sections.</p>
        <blockquote>
<pre>
header {
   import os, sys
}
</pre>
        </blockquote>
      </li>
      <li>
        <b>header "__init__" - specify additional code in the
        <code>__init__</code> method</b> 
        <p>You can instruct the ANTLR Python code generator to include
        additional Python code in your generated Lexer/Parser/TreeParser by
        adding code to the <code>init</code> header section which must
        be the first section at the beginning of your ANTLR grammar file,
        apart from any other header sections. The code in the header is
        appended to the end of the <code>__init__</code> method.</p>
        <blockquote>
<pre>
header "__init__" {
   self.message = "This is the default message"
}
</pre>
        </blockquote>
        <p>If your grammar file contains both a Lexer and a Parser (or any
        other multiple of definitions), the code in the
        <code>__init__</code> header will be reproduced in the
        <code>__init__</code> methods of all of these definitions without
        change. If you really want to update only one of the definitions,
        for instance, the <code>__init__</code> method of the Lexer class
        you are creating, use</p>
        <blockquote>
<pre>
header "&lt;LexerGrammar&gt;.__init__" {
   self.message = "This is the default message"
}
</pre>
        </blockquote>
        <p>where <tt>&lt;LexerGrammar&gt;</tt> is the name of the Lexer
        grammar. The same construction also works with the Parsers and
        TreeParsers, of course.</p>
        <p>In the case both a generic init header and a grammar-specific
        header are present, the grammar-specific one will override the
        generic one.</p>
      </li>
      <li>
        <b>header "__main__" - specify additional code after the class
        definition</b> 
        <p>You can instruct the ANTLR Python code generator to add
        additional Python code at the end of your generated
        Lexer/Parser/TreeParser, so after the class definition itself by
        adding code to the <code>__main__</code> header section which must
        be the first section at the beginning of your ANTLR grammar file,
        apart from any other header sections.</p>
        <blockquote>
<pre>
header "__main__" {
    print "You cannot execute this file!"
}
</pre>
        </blockquote>
        <p>If your grammar file contains both a Lexer and a Parser (or any
        other multiple of definitions), the code in the <code>__main__</code>
        header will be reproduced at the end of all of the generated class
        definitions. If you really want to add code after only one of the
        definitions, for instance, after the Lexer class, use</p>
        <blockquote>
<pre>
header "&lt;LexerGrammar&gt;.__main__" {
    print "You cannot execute this file!"
}
</pre>
        </blockquote>
        <p>where <tt>&lt;LexerGrammar&gt;</tt> is the name of the Lexer
        grammar. The same construction also works with the Parsers and
        TreeParsers, of course.</p>
        <p>In the case both a generic init header and a grammar-specific
        header are present, the grammar-specific one will override the
        generic one. If no <code>__main__</code> headers are present and the
        grammar is for a Lexer,  automated test code for that lexer is
        automatically added at the end of the generated module. This can be
        prevented by providing an empty <code>__main__</code> header. In the
        latter case it is good practise to provide a comment explaining why
        an empty header is present.</p>
        <blockquote>
<pre>
header "&lt;LexerGrammar&gt;.__main__" {
    // Empty main header to prevent automatic test code from being added
    // to the generated lexer module.
}
</pre>
        </blockquote>
        <p>This automated test code can be executed by running Python with
        the generated lexer file (<tt>&lt;LexerGrammar&gt;.py</tt>where
        <tt>&lt;LexerGrammar&gt;</tt> is the name of the Lexer grammar) and
        providing some test input on <tt>stdin</tt>:</p>
        <blockquote>
<pre>
python &lt;LexerGrammar&gt;.py &lt; test.in
</pre>
        </blockquote>
      </li>
    </ul>
    <h2><a name="#options">Python-Specific ANTLR Options</a></h2>
    <ul>
      <li>
        <b>className - change the default name of the generated class</b> 
        <p></p>
        <blockquote>
<pre>
options {
    className="Scanner";
}
</pre>
        </blockquote>
        <p>If you are using the <tt>className</tt> option conjunction with the
        Python specific header options, there will be no collisions. The
        <tt>className</tt> option changes the class name, while the
        <code>main</code> headers require the use of the grammar name which
        will become the module name after code generation.</p>
        <blockquote>
<pre>
header "ParrotSketch.init" {
    self.state = JohnCleese.select("dead", "pushing up daisies", \
                                   "no longer", "in Parrot Heaven")
    print "This parrot is", self.state
}

class ParrotSketch extends Lexer;

options {
    className="Scanner";
}
</pre>
        </blockquote>
      </li>
    </ul>
    <h2><a name="#template">A Template Python ANTLR Grammar File</a></h2>
    <p>As the handling of modules &emdash; packages in Java speak &emdash;
    in Python differs from that in Java, the current approach in ANTLR to
    call both the file and the class they contain after the name of the
    grammar is kind of awkward. Instead, a different approach is chosen that
    better reflects the handling of modules in Python. The name of the
    generated Python file is still derived from the name of the grammar, but
    the name of the class is fixed to the particular kind of grammar. A
    lexer grammar will be used to generate a class <tt>Lexer</tt>; a parser
    grammar will be used to generate a class <tt>Parser</tt>; and a
    treeparser grammar will be used to generate a class <tt>Walker</tt>.</p>
    <blockquote>
<pre>
header {
    // gets inserted in the Python source file before any generated
    // declarations
    ...
}
header "__init__" {
    // gets inserted in the __init__ method of each of the generated Python
    // classes
    ...
}
header "MyParser.__init__" {
    // gets inserted in the __init__ method of the generated Python class
    // for the MyParser grammar
    ...
}
header "__main__" {
    // gets inserted at the end of each of the generated Python files in an
    // indented section preceeded by the conditional:
    // if __name__ == "__main__":
    ...
}
header "MyLexer.__init__" {
    // gets inserted at the end of the generated Python file for the MyLexer
    // grammar in an indented section preceeded by the conditional:
    // if __name__ == "__main__":
    // and preventing the insertion of automatic test code in the same place.
    ...
}
options {
    language  = "Python";
}
{
    // global code stuff that will be included in the 'MyParser.py' source
    // file just before the 'Parser' class below
    ...
}
class MyParser extends Parser;
options {
   exportVocab=My;
}
{
   // additional methods and members for the generated 'Parser' class
   ...
}
... generated RULES go here ...
{
   // global code stuff that will be included in the 'MyLexer' source file
   // just before the 'Lexer' class below
   ...
}
class MyLexer extends Lexer;
options {
   exportVocab=My;
}
{
   // additional methods and members for the generated 'Lexer' class
   ...
}
... generated RULES go here ...
{
   // global code stuff that will be included in the 'MyTreeParser' source
   // file just before the 'Walker' class below
   ...
}
class MyTreeParser extends TreeParser;
options {
   exportVocab=My;
}
{
   // additional methods and members for the generated 'Walker' class
   ...
}
... generated RULES go here ...
</pre>
      <p>Version number in parentheses shows the tool version used to
      develop and test. It may work with older versions as well. Python 2.2
      or better is required as some recent Python features (like
      <tt>super()</tt> for example) are being used.</p>
      <h2><a name="#notes">More notes on using ANTLR Python</a></h2>
      <ul>
        <li>
          <p>The API of the generated lexers, parsers, and treeparsers is
          supposed to be similar to the Java ones. However, calling a lexer
          is somewhat simplified:</p> 
          <blockquote>
<pre>
### class "calcLexer extends Lexer" will generate python
### module "calcLexer" with class "Lexer". 
import calcLexer
### read from stdin ..
L = calcLexer.Lexer() 
### read from file "test.in" ..
L = calcLexer.Lexer("test.in")
### open a file and read from it ..
f = file("test.in", "r")
L = calcLexer.Lexer(f)
### this works of course as well
import sys
L = calcLexer.Lexer(sys.stdin)
### use a shared input state
L1 = calcLexer.Lexer(...)
state = L1.inputState
L2 = calcLexer.Lexer(state)
</pre>
          </blockquote>
        </li>
        <li>
          <p>The loop for the lexer to retrieve token by token can be written
          as:</p> 
          <blockquote>
<pre>
lexer = calcLexer.Lexer()          ### create a lexer for calculator
for token in lexer:
    ## do something with token
    print token
</pre>
          </blockquote>
          or even: 
          <blockquote>
<pre>
for token in calcLexer.Lexer():    ### create a lexer for calculator
    ## do something with token
    print token
</pre>
          </blockquote>
          <p>As an iterator is available for all <tt>TokenStreams</tt>, you
          can apply  the same technique with a <tt>TokenStreamSelector</tt>.</p>
        </li>
        <li>
          <p>However, writing this particular lexer loop is rarely necessary as it
          is generated by default in each generated lexer. Just run:</p> 
          <blockquote>
<pre>
python calcLexer.py &lt; calc.in
</pre>
          </blockquote>
          to test the generated lexer.<br>
        </li>
        <li>
          <p>Symbolic token number, table of literals bitsets and bitset data
          functions are generated on file (module) scope instead of class
          scope. For example:</p> 
          <blockquote>
<pre>
import calcLexer      # import calc lexer module
  
calcLexer.EOF_TYPE    # prints 1
calcLexer.literals    # { ';': 11, 'end': 12, 'begin': 10 }
</pre>
          </blockquote>
        </li>
        <li>
          <p>Comments in action should be in Java/C++ formats, ie. <tt>//</tt>
          and <tt>/* ... */</tt> are valid comments. However, make sure
          that you put a comment before or after a statement, but not
          within. For example, this will not work:</p> 
          <blockquote>
<pre>
x = /* one */ 1
</pre>
          </blockquote>
          <p>The reason is that Python only supports single-line comments. Such
          a Python comment skips everything till end-of-line. Therefore in
          the translation of the comment a newline will be introduced on
          reaching <tt>*/</tt>. The code above would result in the following
          Python code in the generated file:</p>
          <blockquote>
<pre>
x = # one
1
</pre>
          </blockquote>
          <p>which is probably not what you want.</p>
        </li>
        <li>The Lexer actions <tt>$newline</tt>, <tt>$nl</tt> and
        <tt>$skip</tt> have been introduced as language independent
        shortcuts for calling <tt>self.newline()</tt> (<tt>$newline</tt>,
        <tt>$nl</tt>) and <tt>_ttype = SKIP</tt> (<tt>$skip</tt>).<br>
        </li>
        <li>
           <p>In Python arguments to function and method calls do not have a
          declared type. Also, functionns and methdos do not have to declare
          a return type. If you want to pass a value to a rule in your grammar,
          you can do so by providing simply the name of a variable.</p>
          <blockquote>
<pre>
ident [symtable]
    :   ( 'a'..'z' | '0'..'9' )+
    ;
</pre>
          </blockquote>
          <p>Similarly, is you want a rule to pass a return value, you do not
          have to provide a type either. It is possible to provide a default
          value.</p> 
          <blockquote>
<pre>
sign returns [isPos = False]
    :    '-' { /* default value is OK */ }
    |    '+' { isPos = True }
    ;
</pre>
          </blockquote>
        </li>
        <li>
          <p>The <tt>__init__</tt> method of the generated Lexer, Parser, or
          TreeParser has the following heading:</p>
          <blockquote>
<pre>
def __init__(self, *args, **kwargs):
    ...
</pre>
          </blockquote>
          <p>So if you need to pass special arguments to your generated class,
          you can use the <tt>**kwargs</tt> to check for a particular keyword
          argument, irrespective of any non-keyword arguments that you did
          provide. So if you have a <tt>TokenStreamSelector</tt> that you want
          to access locally, you can pass it to the Lexer in the following
          call:</p>
          <blockquote>
<pre>
MySpecialLexer.Lexer(sys.stdin, selector=TokenStreamSelector())
</pre>
          </blockquote>
          <p>while in the <tt>__init__</tt> header of this particular grammar
          you can specify the handling of the <tt>selector</tt> keyword
          argument in the following way:</p>
          <blockquote>
<pre>
header "MyParser.__init__" {
    self.selector = None
    if kwargs.has_key("selector"):
        self.selector = kwargs["selector"]
        assert(isinstance(self.selector, TokenStreamSelector))

}
</pre>
          </blockquote>
        </li>
        <li>
          <p>Because of limitations in the lexer of the ANTLR compiler
          generator  itself, you cannot use single quoted strings of more than
          one character  in your Python code.<br>
          So if you use a Python string like <tt>'wink, wink, nudge,
          nudge'</tt>  in one of your actions, ANTLR will give a parse error
          when you try to compile this grammar. Instead you should use double
          quotes: <tt>"wink, wink, nudge, nudge"</tt>.</p>
        </li>
        <li>
          <p>Unicode is supported but it's easy to run into errors if your
          terminal(output device) is not able to handle unicode chars.<br>
          Here are some rules when using Unicode input:</p> 
          <ol>
            <li>
              You need to wrap your input stream by a stream reader which
              translates bytes into unicode chars. This requires usually
              knowledge about your input's encoding. Assume for example
              that your input is 'latin1', you would do this: 
              <blockquote>
<pre>
### replace  stdin  with  a  wrapper that spits out
### unicode chars.       
sys.stdin = codecs.lookup('latin1')[-2](sys.stdin)
</pre>
              </blockquote>
              Here reading from stdin gets wrapped.
            </li>
            <li>
              When printing tokens etc containing Unicode chars it appears
              to be best to translate explicit to a unicode string before
              printing. Consider: 
              <blockquote>
<pre>
for token in unicode_l.Lexer() :
    print unicode(token)   ## explict cast
</pre>
              </blockquote>
              This explicit cast appears to be a bug in Python found during
              development (discussion still in progress).
            </li>
          </ol>
        </li>
      </ul>
    </blockquote>
  </body>
</html>