=======================
PyBindGen Tutorial
=======================



What is PyBindGen ?
===================

PyBindGen is a tool which can be used to generate python bindings
for C or C++ APIs. It is similar in scope to tools such as boost::python,
SWIG, and a few others but has a number of specific features which make
it especially useful in a number of cases:

  - PyBindGen is implemented in python and is used and controlled
    through python;
  - PyBindGen error messages do not involve c++ template deciphering
    (as in boost::python);
  - PyBindGen generates highly-readable C or C++ code so it is
    possible to step into and debug the bindings;
  - In simple cases, PyBindGen is really easy to use. In more
    complicated cases, it does offer all the flexibility you need to
    wrap complex C or C++ APIs;
  - PyBindGen also provides an optional tool to parse C and C++
    headers and generate automatically bindings for them, potentially
    using extra inline or out-of-line annotations.  This tool is based
    on gccxml and pygccxml: it can be used to generate the first
    version of the bindings and tweak them by hand later or as a fully
    automated tool to continuously generate bindings for changing
    C/C++ APIs.

This tutorial will show how to build bindings for a couple of common C and C++ API idioms
and, then, will proceed to show how to use the automatic binding generator.


Supported Python versions
=========================

PyBindGen officially supports Python versions 2.6, 2.7, and >= 3.3[*] (tested in 3.3 and 3.4).

PyBindGen does not support Python versions 3.0, 3.1, and 3.2.

Note that C files generated by PyBindGen transparently support
multiple Python versions.  In particular, the generated code can build
in either Python 2.x or Python 3.x.

[*] The automatic header file scanning feature of PyBindGen (submodule
`pybindgen.gccxmlparser`) does not work in any Python 3.x version,
since the pygccxml package has not been ported to Python 3.



Work flows
==========

PyBindGen is only a Python module.  The programmer must write a Python
script that uses the module in order to generate the bindings.  There
are several ways that PyBindGen can be used:

  1. Basic mode, script that directly generates the bindings;

  .. image:: figures/work-flow-basic.*

  2. Automatically scan header files and generate the bindings
  directly (see :ref:`Header file scanning with (py)gccxml<header-file-scanning>`);

  .. image:: figures/work-flow-gccxml.*

  3. Automatically scan header files to generate API defs file, that
  file can later be used to generate the bindings.  See (see
  :ref:`Header file scanning with (py)gccxml: python intermediate
  file<header-file-scanning-apidefs>`);

  .. image:: figures/work-flow-gccxml-apidefs.*


A simple example
================
The best way to get a feel for what PyBindGen looks like is to go through a 
simple example.

Code generation script
----------------------

Let's assume that we have a simple C API as shown below
declared in a header my-module.h:

  .. code-block:: c++

    void MyModuleDoAction (void);

What we want to do is call this C function from python and be able to write 
python code such as::

  import MyModule

  MyModule.MyModuleDoAction ()

Getting there is, hopefully, not very complicated: we just need to write a small
python program whose job is to generate the C code which will act as a bridge
between our user's python program and the underlying C function. First, we import
the pybindgen and the sys modules::

  import pybindgen
  import sys

Then, we create an object to represent the module we want to generate::

  mod = pybindgen.Module('MyModule')

add our C header::

  mod.add_include('"my-module.h"')

and register our function which returns no value (hence, the second
argument 'None'), and, takes no arguments (hence, the third argument,
the empty list '[]')::

  mod.add_function('MyModuleDoAction', None, [])

Finally, we generate code for this binding directed to standard output::

  mod.generate(sys.stdout)

The final program is pretty short::

  import pybindgen
  import sys

  mod = pybindgen.Module('MyModule')
  mod.add_include('"my-module.h"')
  mod.add_function('MyModuleDoAction', None, [])
  mod.generate(sys.stdout)

Building it using Python setup.py (distutils)
---------------------------------------------

This very small example is located in the :download:`first-example
directory <_static/first-example.zip>`, together with a small makefile which
will build our extension module::

  $ cd first-example/
  $ python setup.py build

The `setup.py` is mostly a standard Python distutils driver script.
Please refer to the `Python documentation
<http://docs.python.org/3/extending/building.html>`_ for more
information.

The unusual part about this `setup.py` is that it imports
`mymodulegen` and calls the generate function, with the
`build/my-module-binding.c` as argument::

 from mymodulegen import generate
 module_fname = os.path.join("build", "my-module-binding.c")
 with open(module_fname, "wt") as file_:
     print("Generating file {}".format(module_fname))
     generate(file_)

After `build/my-module-binding.c` having been generated, with the help
of PyBindGen (as seen in the previous section), we can use it as one
of the source files for our extension module::

 mymodule = Extension('mymodule',
                      sources = [module_fname, 'my-module.c'],
                      include_dirs=['.'])

The rest is standard setup.py code.


Testing it
----------

Once all of that code is built, we obviously want to run it. Setting up
your system to make sure that the python module is found by the python runtime
is outside the scope of this tutorial but, for most people, the following session
should be self-explanatory::

 $ cd first-example/
 $ python setup.py buid
 Generating file build/my-module-binding.c
 running build
 running build_ext
 building 'mymodule' extension
 creating build/temp.linux-x86_64-2.7
 creating build/temp.linux-x86_64-2.7/build
 [...]
 $ export PYTHONPATH=build/lib.linux-x86_64-2.7/
 $ python
 Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
 [GCC 4.8.1] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import mymodule
 >>> mymodule.MyModuleDoAction ()
 You called MyModuleDoAction !
 >>> 


Wrapping types by value
=======================

Primitive types
---------------

The first example showed how to call a function which takes no
arguments and returns no values which, obviously, is not especially
interesting so, let's look at how we can give meaningfull arguments
to our function:

.. code-block:: c++

   int MyModuleDoAction (int v1, int v2);

and the corresponding bit from the code generation script: the second
argument to add_function specifies that our function returns a value of type
'int' and the third argument specifies that our function takes as a
single argument an 'int' of name 'value'::

  mod.add_function('MyModuleDoAction', 
                    pybindgen.retval ('int'), 
                   [pybindgen.param ('int', 'v1'),
                    pybindgen.param ('int', 'v2')])

The above then allows you to write::

  >>> import MyModule
  >>> v = MyModule.MyModuleDoAction (10, -1)
  You called MyModuleDoAction: 10
  >>> print v
  10
  >>> v = MyModule.MyModuleDoAction (v2=5, v1=-2)
  You called MyModuleDoAction: -2
  >>> print v
  -2

Which shows how the argument name can be used to avoid
using positional arguments.


Of course, the above example could be rewritten to the more compact and readable::

  from pybindgen import *
  mod.add_function('MyModuleDoAction', retval ('int'), 
                   [param ('int', 'v1'),
                    param ('int', 'v2')])

In the following examples, this is what we will do to avoid extra typing.


Enum types
----------

Enums are often used to define C and C++ constants as shown below:

.. code-block:: c++

  enum MyEnum_e
  {
    CONSTANT_A,
    CONSTANT_B,
    CONSTANT_C
  };
  void MyModuleDoAction (enum enum_e value);

And wrapping them is also pretty trivial::

  from pybindgen import *
  import sys

  mod = Module('MyModule')
  mod.add_include('"my-module.h"')
  mod.add_enum('MyEnum_e', ['CONSTANT_A', 'CONSTANT_B', 'CONSTANT_C'])
  mod.add_function('MyModuleDoAction', None, [param('MyEnum_e', 'value')])
  mod.generate(sys.stdout)

With the resulting python-visible API::

  >>> import MyModule
  >>> print MyModule.CONSTANT_A
  0
  >>> print MyModule.CONSTANT_B
  1
  >>> print MyModule.CONSTANT_C
  2
  >>> MyModule.MyModuleDoAction (MyModule.CONSTANT_B)
  MyModuleDoAction: 1

Compound types
--------------

Passing a structure to and from C is not really more complicated than
our previous example. The API below:

.. code-block:: c++

  struct MyModuleStruct
  {
    int a;
    int b;
  };
  struct MyModuleStruct MyModuleDoAction (struct MyModuleStruct value);

can be bound to python using the following script::

  from pybindgen import *
  import sys

  mod = Module('MyModule')
  mod.add_include('"my-module.h"')
  struct = mod.add_struct('MyModuleStruct')
  struct.add_instance_attribute('a', 'int')
  struct.add_instance_attribute('b', 'int')
  mod.add_function('MyModuleDoAction', retval ('MyModuleStruct'), [param ('MyModuleStruct', 'value')])
  mod.generate(sys.stdout)

The most obvious change here is that we have to define the new structure type::

  struct = mod.add_struct('MyModuleStruct')

and register the names and types of each of the members we want to make accessible
from python::

  struct.add_instance_attribute('a', 'int')
  struct.add_instance_attribute('b', 'int')

The name of the method called here, 'add_instance_attribute' reflects the fact that
PyBindGen can wrap both C and C++ APIs: in C++, there exist both instance and static
members so, PyBindGen provides two methods: add_instance_attribute and add_static_attribute
to register these two kinds of members.

Our C API then becomes accessible from python::
  >>> import MyModule
  >>> st = MyModule.MyModuleStruct ()
  >>> st.a = 10
  >>> st.b = -20
  >>> st.c = -10
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  AttributeError: 'MyModule.MyModuleStruct' object has no attribute 'c'
  >>> v = MyModule.MyModuleDoAction (st)
  You called MyModuleDoAction: 10
  >>> print v
  <MyModule.MyModuleStruct object at 0x2b5ef522b150>
  >>> print v.a
  10
  >>> print v.b
  -20


C++ classes
-----------

Wrapping C++ classes is very similar to wrapping a C struct with a few functions: we will thus
start by extending our C API with a C++ class declaration:

.. code-block:: c++

  class MyClass
  {
  public:
    void SetInt (int value);
    int GetInt (void) const;
  };

We first need to declare a C++ class::

  mod = Module('MyModule')
  klass = mod.add_class('MyClass')

and, then, specify that it has a constructor::

  klass.add_constructor([])

We can declare the setter method which is really
a straightforward extension from the add_function function::

  klass.add_method('SetInt', None, [param('int', 'value')])

The getter is also pretty straightforward except for the declaration
of constness::

  klass.add_method('GetInt', retval('int'), [], is_const=True)

Using this API is also very similar to the struct example we went through
in the previous section::

  >>> my = MyModule.MyClass()
  >>> my.SetInt(10)
  >>> v = my.GetInt()
  >>> print v
  10

It is also possible to bind inner classes and enums such
as these:

.. code-block:: c++

  class Outer
  {
  public:
    void Do (void);
    // an inner enum
    enum inner_e
    {
      INNER_A,
      INNER_B,
      INNER_C
    };
    // an inner class
    class Inner
    {
    public:
      void Do (enum Outer::inner_e value);
    };
  };

We just need to bind the outer class::

  outer = mod.add_class('Outer')
  outer.add_constructor([])
  outer.add_method('Do', None, [])

Then, bind its inner enum::

  mod.add_enum('inner_e', ['INNER_A', 'INNER_B', 'INNER_C'], outer_class=outer)

and, finally, bind its inner class::

  mod.add_class('Inner', outer_class=outer)
  inner.add_constructor([])

The only slightly tricky part is binding the Do method of the Inner
class since it refers to the enum type defined in the Outer class: we
simply need to carefully use the fully scoped name of the enum.::

  inner.add_method('Do', None, [param('Outer::inner_e', value)])

The resulting python API reflects the underlying C++ API very closely::

  >>> import MyModule
  >>> print MyModule.Outer.INNER_A
  0
  >>> print MyModule.Outer.INNER_B
  1
  >>> outer = MyModule.Outer()
  >>> outer.Do()
  >>> inner = MyModule.Outer.Inner()
  >>> inner.Do(MyModule.Outer.INNER_A)


C++ namespaces
--------------

Wrapping multiple nested namespaces is, of course, possible and represents
no special challenge. Let's look at an example:

.. code-block:: c++

  namespace Outer {
    void Do (void);
    class MyClass 
    {};
    namespace Inner {
      void Do (void);
      class MyClass 
      {};
    } // namespace Inner
  } // namespace Outer

First, we need to define the Outer namespace::

  mod = Module('MyModule')
  outer = mod.add_cpp_namespace('Outer')

Then, register its classes and functions::

  outer.add_class('MyClass')
  outer.add_function('Do', None, [])

and, finally, define the Inner namespace and its associated
functions and methods::

  inner = outer.add_cpp_namespace('Inner')
  inner.add_class('MyClass')
  inner.add_function('Do', None, [])

The resulting API, again, sticks to the underlying C++ API by
defining one python module for each C++ namespace and making
sure that the hierarchy of python modules matches the hierarchy
of C++ namespaces::

  >>> import MyModule
  >>> o = MyModule.Outer.MyClass()
  >>> i = MyModule.Outer.Inner.MyClass()
  >>> from MyModule.Outer.Inner import *
  >>> i = MyClass()


Memory management for pointer types
===================================

Until then, we have shown how to pass back and forth data through C/C++ APIs
only by value but, a large fraction of real-world APIs use raw pointers
(and, in the case of C++, smart pointers) as arguments or return values 
of functions/methods.

Rather than try to explain the detail of every option offered by PyBindGen
to deal with pointers, we will go through a couple of very classic memory
management schemes and examples.

Function returns pointer
------------------------

The API to bind:

.. code-block:: c++

  class MyClass;
  MyClass *DoSomethingAndReturnClass (void);

First, we declare the MyClass type::

  mod.add_class('MyClass')
  ...

Then, if we assume that the function returns ownership of the pointer to the caller, we
can write::

  mod.add_function('DoSomethingAndReturnClass', retval('MyClass *', caller_owns_return=True), [])

The above will tell PyBindGen that the caller (the python runtime) becomes
responsible for deleting the instance of MyClass returned by the function
DoSomethingAndReturnClass when it is done with it.

Of course, it is possible to not give back ownership of the returned pointer
to the caller::

  mod.add_function('DoSomethingAndReturnClass', retval('MyClass *', caller_owns_return=False), [])

Which would make the python runtime assume that the lifetime of the returned pointer
is longer than the associated python object.

Function takes pointer
----------------------

The API to bind:

.. code-block:: c++

  class MyClass;
  void DoWithClass (MyClass *cls);

If we assume that the callee takes ownership of the input pointer, we can write::

  mod.add_function('DoWithClass', None, [param('MyClass *', 'cls', transfer_ownership=True)])

Which will make python keep a handle on the MyClass instance but never destroy it himself
and rely on the callee to destroy it at the right time. This kind of scheme is obviously
a bit dangerous because python has no way of knowing when the underlying MyClass instance
is really destroyed so, if you try to invoke methods on it _after_ it has been destroyed,
bad things will obviously happen.

If, instead, we assume that the caller keeps ownership of the pointer, we can write
the much safer version::

  mod.add_function('DoWithClass', None, [param('MyClass *', 'cls', transfer_ownership=False)])

Which will allow python to delete the MyClass instance only when the associated python wrapper
disappears.

A reference-counted object
--------------------------

A nice way to avoid some of the ambiguities of the above-mentioned API bindings is to 
use reference-counted C or C++ objects which must provide a pair of functions or methods
to increase or decrease the reference count of the object. For example, a classic
C++ reference-counted class:

.. code-block:: c++

  class MyClass 
  {
  public:
    void Ref (void);
    void Unref (void);
    uint32_t PeekRef (void);
  };

And the associated function which takes a pointer:

.. code-block:: c++

   void DoSomething (MyClass *cls);

To wrap this class, we first need to declare our class::

  from pybindgen import cppclass
  [...]
  mod.add_class('MyClass', memory_policy=cppclass.ReferenceCountingMethodsPolicy( 
                    incref_method='Ref', 
                    decref_method='Unref', 
                    peekref_method='PeekRef'))

The above allows PyBindGen to maintain and track the reference count
of the MyClass object while the code below shows how we can declare
a function taking a pointer as input::

  mod.add_function('DoSomething', None, [param('MyClass *', 'cls', transfer_ownership=False)]

Here, the meaning of transfer_ownership changes slightly.
Whithout reference counting, transfer_ownership refers to the
transfer of the object as a whole, i.e. either the caller or
callee will own the object in the end, but not both.  With
reference counting, transfer_ownership refers to the transfer of a
_reference_.  In this example, transfer_ownership=False means that
the caller will not "steal" our reference, i.e. it will either not
keep a reference to our object for itself, or if it does it
creates its own reference to the object by calling the incref
method.  If transfer_ownership=True it would mean that the caller
would keep the passed in reference to itself, and if the caller
wants to keep the reference it must call the incref method first.

A more interesting case is that of returning such a reference counted 
object from a function:

.. code-block:: c++

  MyClass *DoSomething (void);

While classic reference counting rules require that the callee returns
a reference to the caller (i.e., it calls Ref on behalf of the caller
before returning the pointer), some APIs will undoubtedly return a pointer
and expect the caller to acquire a reference to the returned object by
calling Ref himself. PyBindGen hopefully can be made to support this
case too::

  mod.add_function('DoSomething', retval('MyClass *', caller_owns_return=False), [])

Which instructs PyBindGen that DoSomething is not to be trusted and that it should
acquire ownership of the returned pointer if it needs to keep track of it.


A STL container
---------------

If you have a function that takes a STL container, you have to
tell PyBindGen to wrap the container first:

.. code-block:: c++

    void DoSomething (std::list<std::string> const &listOfStrings);

Is wrapped by::

    module.add_container('std::list<std::string>', 'std::string', 'list') # declare a container only once
    [...]
    mod.add_function('DoSomething', None, [param('std::list<std::string> const &', 'listOfStrings')])


.. Subclassing a C++ class from python
.. ===================================

.. Extending a C++ class or namespace from python
.. ==============================================


Advanced usage
==============


Basic interface with error handling
-----------------------------------

It is also possible to declare a error handler.  The error handler
will be invoked for API definitions that cannot be wrapped for some
reason::

  #! /usr/bin/env python

  import sys

  import pybindgen
  from pybindgen import Module, FileCodeSink, retval, param

  import pybindgen.settings
  import warnings

  class ErrorHandler(pybindgen.settings.ErrorHandler):
      def handle_error(self, wrapper, exception, traceback_):
          warnings.warn("exception %r in wrapper %s" % (exception, wrapper))
          return True
  pybindgen.settings.error_handler = ErrorHandler()


  def my_module_gen(out_file):
      pybindgen.write_preamble(FileCodeSink(out_file))

      mod = Module('a')
      mod.add_include('"a.h"')

      mod.add_function('ADoA', None, [])
      mod.add_function('ADoB', None, [param('uint32_t', 'b')])
      mod.add_function('ADoC', retval('uint32_t'), [])

      mod.generate(FileCodeSink(out_file) )

  if __name__ == '__main__':
      my_module_gen(sys.stdout)

In this example, we register a error handler that allows PyBindGen
to simply ignore API definitions with errors, and not wrap them, but
move on.

The difference between is Parameter.new(...) and param(...), as well
as between ReturnValue.new(...) and retval(...) is to be noted here.
The main difference is not that param(...) and retval(...) are
shorter, it is that they allow delayed error handling.  For example,
when you put Parameter.new("type that does not exist", "foo") in
your python script, a TypeLookupError exception is raised and it is
not possible for the error handler to catch it.  However, param(...)
does not try to lookup the type handler immediately and instead lets
Module.add_function() do that in a way that the error handler can be
invoked and the function is simply not wrapped if the error handler
says so.

.. _header-file-scanning:

Header file scanning with (py)gccxml
------------------------------------

If you have gccxml and pygccxml installed, PyBindGen can use them to
scan the API definitions directly from the header files::

  #! /usr/bin/env python

  import sys

  import pybindgen
  from pybindgen import FileCodeSink
  from pybindgen.gccxmlparser import ModuleParser

  def my_module_gen():
      module_parser = ModuleParser('a1', '::')
      module = module_parser.parse([sys.argv[1]])
      module.add_include('"a.h"')

      pybindgen.write_preamble(FileCodeSink(sys.stdout))
      module.generate(FileCodeSink(sys.stdout))

  if __name__ == '__main__':
      my_module_gen()

The above script will generate the bindings for the module directly.
It expects the input header file, a.h, as first command line
argument.

.. _header-file-scanning-apidefs:

Header file scanning with (py)gccxml: python intermediate file
--------------------------------------------------------------

The final code generation flow supported by PyBindGen is a hybrid of
the previous ones.  One script scans C/C++ header files, but instead
of generating C/C++ binding code directly it instead generates a
PyBindGen based Python script::

  #! /usr/bin/env python

  import sys

  from pybindgen import FileCodeSink
  from pybindgen.gccxmlparser import ModuleParser

  def my_module_gen():
      module_parser = ModuleParser('a2', '::')
      module_parser.parse([sys.argv[1]], includes=['"a.h"'], pygen_sink=FileCodeSink(sys.stdout))

  if __name__ == '__main__':
      my_module_gen()

The above script produces a Python program on stdout.  Running the
generated Python program will, in turn, generate the C++ code
binding our interface.