.. _ug-introduction-to-xso:

An Introduction to XSO
######################

This document shall serve as an introduction to the :mod:`aioxmpp.xso`
subpackage. This is intentionally separate from the API documentation and
the glossary, since it should provide a high- and user-level introduction for
those who first get into using it.


About :mod:`aioxmpp.xso`
========================

Let us give you an introduction to the :mod:`aioxmpp.xso` package in form of
answers to a few quick questions:

What is :mod:`aioxmpp.xso`?
---------------------------

It is a mapping layer between XML structured data (elements, text and
attributes) and python objects. It is built with streaming in mind.
SAX-compatible events are interpreted and converted to python objects
on-the-fly. No DOM is needed or used.

If you have ever worked with Object-Relational Mappers for databases, such as
:mod:`sqlalchemy`, XSO will feel familiar.

What is :mod:`aioxmpp.xso` **not**?
-----------------------------------

A replacement for a full-blown XML library. If you need the full XML 1.0+ DOM,
XPath, XQuery and/or possibly XSLT to work with your data, XSO is not the right
thing for you.

Specifically, the following XML 1.0 features are decidedly **not supported** in
:mod:`aioxmpp.xso`:

- Non-namespace-well-formed documents: All documents processed and generated by
  XSO are namespace well-formed.
- Processing Insturctions
- Comments
- Document Type Declarations
- Preservation of qualified names / namespace prefixes. They are semantically
  irrelevant: only the :term:`Namespace URI` and :term:`Local Name` matter.
- Preservation of ordering between some elements. The following relative orders
  are specifically violated:

  - Text nodes vs. non-text nodes within the same parent element
  - Child elements which are handled by different descriptors. Often, only
    elements with the same :term:`Namespace URI` and :term:`Local Name` are
    handled by the same descriptor (see also
    :ref:`ug-introduction-to-xso-descriptors`).

There may be other edge-case features we do not support.

How much `Magic`_ is inside :mod:`aioxmpp.xso`?
-----------------------------------------------

Hopefully not too much, but there’s still a bit. I’ll let you know that there
is at least one metaclass involved, to handle the processing of descriptors at
class-definition time and enforcing invariants during inheritance. Sorry for
that.

Oh, and the use of generators as suspendable functions to make the parsing code
easier to read.

Other than that, I think, it’s pretty standard Python though.

.. _Magic: http://www.catb.org/jargon/html/M/magic.html


Into the Deep End / Very Quick Start
====================================

Let us jump right in:

.. code-block:: python

    >>> data = \
    ...   b"<node xmlns='urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734'" \
    ...   b" a1='foo'>" \
    ...   b"<child>some text</child>" \
    ...   b"</node>"
    >>> namespace = "urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734"
    >>> class Node(aioxmpp.xso.XSO):
    ...   TAG = namespace, "node"
    ...   attr = aioxmpp.xso.Attr("a1")
    ...   data = aioxmpp.xso.ChildText((namespace, "child"))
    ...
    >>> buf = io.BytesIO(data)
    >>> n = aioxmpp.xml.read_single_xso(buf, Node)
    >>> isinstance(n, Node)
    True
    >>> n.attr
    'foo'
    >>> n.data
    'some text'

Look, you just parsed your first XSO!

.. note::

    The :mod:`aioxmpp.xml` module, which is technically not part of
    :mod:`aioxmpp.xso`, was also involved. This is because *driving* the XSO
    parser with SAX events from a bytes object requires quite some setup, and
    there are shorthands for that in :mod:`aioxmpp.xml`.

Let us walk through this step-by-step.

1. ``data = ...``: We simply set up a blob of data for us to parse. There
   should be nothing or at least not much special in there. It is simply an
   XML fragment with an element which has a single child element.
2. ``class Node``: This declares the XSO class. Inheriting from
   :class:`aioxmpp.xso.XSO` is how you say "I want this to be parseable and
   serialisable from/to XML". It is required for the descriptors to work.

   1. ``TAG = ...``: This sets the :term:`namespace-uri`/:term:`local-name`
      pair which identifies this XSO. The identification is not global; thus,
      it is allowed to declare multiple XSO descendant classes with the same
      TAG.
   2. ``attr = aioxmpp.xso.Attr(...)``: :class:`aioxmpp.xso.Attr` is a
      descriptor. It is understood by the :class:`aioxmpp.xso.XSO` class and
      collected into bookkeeping attributes at class definition time. When
      an element needs to be parsed and it has attributes, the parsing function
      looks up the attribute tag in the bookkeeping and delegates processing of
      the attribute to the descriptor.
   3. ``data = aioxmpp.xso.ChildText(...)``: :class:`aioxmpp.xso.ChildText` is
      another descriptor. In contrast to the :class:`~aioxmpp.xso.Attr`
      descriptor, this one handles child element events (and not attribute
      events). If a child element event matching the tag given as first
      argument to this descriptor, the parser delegates parsing of that element
      to the descriptor.
3. ``buf = ...``: Create a file-like from which the parser function can read.
4. ``n = aioxmpp.xml.read_single_xso``: Read a single XSO from a file-like
   object and save it into ``n``.
5. The following attribute accesses show how data has arrived in the instance
   of ``Node``.

Again, if you have used an ORM before, how we declared `Node` should be very
familiar to you.


.. _ug-introduction-to-xso-terminology:

A Bit of XSO Terminology
========================

Now after the plunge into the deep end, let us get a bit of terminology
straight so that it is clear what we're talking about:

Character Data
    Text or CDATA nodes in the XML document. Text and CDATA are treated the
    same by XSO (after the decoding handled by the XML library).

Element
    An element node in an XML tree. An element node may hold child nodes,
    such as text nodes, other elements and attributes.

Tag
    A tag is a pair consisting of a :term:`namespace-uri` and a
    :term:`local-name`. It is a fully-qualified name for an XML element. A
    common notation for tags is
    `Clark’s Notation <http://www.jclark.com/xml/xmlns.htm>`_. For example
    ``{uri:foo}bar`` for a local name ``bar`` and a namespace URI
    ``uri:foo``.

    In XSO, tags are represented as tuples with two strings, reflecting the
    structure of the aforementioned pair.

XSO Type
    Describes how to map XML data (character data or element subtrees) to
    python types and vice versa. Examples are :class:`aioxmpp.xso.Integer`
    and :class:`aioxmpp.xso.EnumElementType`.

    XSO types can be categorized in two classes:

    1. :term:`Character Data Types <Character Data Type>`, which map character
       data to python data structures (e.g. :class:`aioxmpp.xso.Integer`).
    2. :term:`Element Types <Element Type>`, which map XML subtrees to python
       data structures and vice versa (e.g.
       :class:`aioxmpp.xso.EnumElementType`).

    Not to be confused with a descendant of :mod:`aioxmpp.xso`.


Writing XSO classes
===================

To write your own XSO class, you simply need a class which inherits (directly
or indirectly) from :class:`aioxmpp.xso.XSO`. Inheriting from that class allows
the descriptors to work.

.. note::

    Despite its intricacy, inheritance involving :class:`aioxmpp.xso.XSO`
    descendants is fully supported. There are a few invariants which have to be
    maintained, however. Violating those invariants will raise an error at
    class definition time. In general, those invariants are common sense, but
    if you want to dig into the details, see
    :class:`aioxmpp.xso.model.XMLStreamClass`.

.. _ug-introduction-to-xso-descriptors:

XSO descriptors
---------------

The descriptors are the main component a user will come in contact with. They
can be categorized into four categories:

*Attribute Descriptors*
    which handle attribute nodes, i.e. attributes on the element which the XSO
    describes.


*Text Descriptors*
    which handle text nodes, i.e. text content (including CDATA sections)
    inside the element which the XSO describes.

*Scalar Child Descriptors*
    which handle (possibly different) child elements, but at most one of them.

    For example, a scalar descriptor which captures one child element of either
    of two different types will at any time hold at most one child element; it
    cannot hold one of each type. Two different descriptors, or a non-scalar
    descriptor is needed for that.

*Non-scalar Child Descriptors*
    which handle multiple child elements. These are then aggregated in
    different types of containers depending on the specific descriptor.

An overview of all descriptors, grouped by their category, follows. Please
click through to the full classes at one point, because the one-liner
description shown in this summary (as well as the abbreviated argument list)
cannot describe the full potential.

Attribute Descriptors
^^^^^^^^^^^^^^^^^^^^^

.. autosummary::

    ~aioxmpp.xso.Attr
    ~aioxmpp.xso.LangAttr

Text Descriptors
^^^^^^^^^^^^^^^^

.. autosummary::

    ~aioxmpp.xso.Text

Scalar Child Descriptors
^^^^^^^^^^^^^^^^^^^^^^^^

.. autosummary::

    ~aioxmpp.xso.Child
    ~aioxmpp.xso.ChildTag
    ~aioxmpp.xso.ChildFlag
    ~aioxmpp.xso.ChildText
    ~aioxmpp.xso.ChildValue

Non-scalar Child Descriptors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. autosummary::

    ~aioxmpp.xso.ChildList
    ~aioxmpp.xso.ChildMap
    ~aioxmpp.xso.ChildValueList
    ~aioxmpp.xso.ChildValueMap
    ~aioxmpp.xso.ChildValueMultiMap
    ~aioxmpp.xso.ChildLangMap
    ~aioxmpp.xso.ChildTextMap
    ~aioxmpp.xso.Collector

Handling of unexpected attributes and child elements
----------------------------------------------------

The handling of unexpected attributes and child elements on an XSO can be
controlled at class definition time using two special attributes:

* :attr:`aioxmpp.xso.XSO.UNKNOWN_CHILD_POLICY` to control how unknown children
  are handled. The possible values are :attr:`.UnknownChildPolicy.DROP` (the
  default), which simply ignores such child elements and
  :attr:`.UnknownChildPolicy.FAIL` which raises an exception.
* :attr:`aioxmpp.xso.XSO.UNKNOWN_ATTR_POLICY` to control how unknown attributes
  are handled. The possible values are :attr:`.UnknownAttrPolicy.DROP` (the
  default), which simply ignores such attributes and
  :attr:`.UnknownAttrPolicy.FAIL` which raises an exception.

.. note::

    Unexpected text is always treated as an error.


Character Data Types
--------------------

XML data (beyond the structure) is strings only. However, most protocols built
on top of XML will have types which are used for attributes and text content
more specific than "string".

For example, you’ll commonly find attributes which are integers or booleans and
character data payloads which are base64-encoded binary. For the common types,
:mod:`aioxmpp.xso` ships with type definitions:

.. autosummary::

    aioxmpp.xso.String
    aioxmpp.xso.Float
    aioxmpp.xso.Integer
    aioxmpp.xso.Bool
    aioxmpp.xso.Base64Binary
    aioxmpp.xso.HexBinary
    aioxmpp.xso.LanguageTag
    aioxmpp.xso.JSON

Some more XMPP specific types are:

.. autosummary::

    aioxmpp.xso.DateTime
    aioxmpp.xso.Date
    aioxmpp.xso.Time
    aioxmpp.xso.JID
    aioxmpp.xso.ConnectionLocation

.. note::

    "What is XMPP-specific about the date types?" you may very well ask. They
    do not implement the full syntax of xml schema date, datetime and time
    data type definitions.

    They should work for most of those values, but some edge-cases (such as
    years outside of the range 0..9999) are not handled. See also :xep:`82`.

The types above can be used anywhere where XSO character data types are needed.
Which in turn is every place where XSO handles XML character data, so that’s
attributes (:class:`~aioxmpp.xso.Attr`) and text nodes (e.g. :class:`~aioxmpp.xso.ChildText` and :class:`~aioxmpp.xso.Text`).


Combining the Above in an Example
---------------------------------

We’ve given you lots of theoretical stuff to chew on. Let us put this in
practice with a more sophisticated example.

Hopefully, with the above explanations and the links into the reference
documentation, you will be able to understand this example. If you are not, I
did a bad job at writing this documentation. In that case, I very much would
like to `hear about it <https://github.com/horazont/aioxmpp/issues/new>`_ to
improve it in the future!

Take this bit of code:

.. code-block:: python

    import aioxmpp.xso

    namespace = "urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490"

    class Chapter(aioxmpp.xso.XSO):
        TAG = namespace, "chapter"

        title = aioxmpp.xso.ChildTextMap((namespace, "title"))
        start_page = aioxmpp.xso.Attr(
            "start-page",
            type_=aioxmpp.xso.Integer()
        )


    class TableOfContents(aioxmpp.xso.XSO):
        TAG = namespace, "toc"

        chapters = aioxmpp.xso.ChildList([Chapter])


    class Book(aioxmpp.xso.XSO):
        TAG = namespace, "book"

        id_ = aioxmpp.xso.Attr("id")
        author = aioxmpp.xso.ChildText((namespace, "author"))
        npages = aioxmpp.xso.ChildText(
            (namespace, "pages"),
            type_=aioxmpp.xso.Integer(),
        )
        published = aioxmpp.xso.ChildText(
            (namespace, "published"),
            type_=aioxmpp.xso.Date(),
        )
        title = aioxmpp.xso.ChildTextMap((namespace, "title"))
        toc = aioxmpp.xso.Child([TableOfContents])


    class Library(aioxmpp.xso.XSO):
        TAG = namespace, "library"

        books = aioxmpp.xso.ChildList([Book])


It declares one of the classic examples of XML teaching: a book collection.
Save the above snippet as ``library_demo.py``. Then we can read an XML file
with a ``Library`` shaped root element using the following snippet:

.. code-block:: python

    import sys
    import aioxmpp.xml
    import library_demo

    with open(sys.argv[1], "r") as f:
        library = aioxmpp.xml.read_single_xso(f, library_demo.Library)

    for book in library.books:
        print("book (id = {!r}):".format(book.id_))
        print("  author:", book.author)
        print("  published:", book.published)
        print("  npages:", book.npages)
        print("  title:")
        for lang, title in book.title.items():
        print("    [{!s}] {!r}".format(lang, title))
        print("  table of contents:")
        for i, chapter in enumerate(book.toc.chapters, 1):
            print("    {}.    (page {})".format(i, chapter.start_page))
            for lang, title in chapter.title.items():
                print("      [{!s}] {!r}".format(lang, title))

Save that file as ``library_load.py`` and try it on the following XML file
(``library_test.xml``):

.. code-block:: xml

    <?xml version="1.0"?>
    <library xmlns="urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490">
      <book id="foo">
        <title xml:lang="en">The Amazing Life of Foo</title>
        <title xml:lang="de">Das Faszinierende Leben des Foo</title>
        <author>F. Nord</author>
        <published>2099-01-01</published>
        <pages>23</pages>
        <toc>
          <chapter start-page="1">
            <title xml:lang="en">The Birth of Foo</title>
            <title xml:lang="de">Die Geburt des Foo</title>
          </chapter>
          <chapter start-page="3">
            <title xml:lang="en">The Death of Foo</title>
            <title xml:lang="de">Der Tod des Foo</title>
          </chapter>
        </toc>
      </book>
      <book id="pink-flamingos">
        <title xml:lang="en">The Relevance of Pink Flamingos to Computer Science</title>
        <title xml:lang="de">Die Relevanz von rosa Flamingos für die Informatik</title>
        <author>O. L. Bilderrahmen</author>
        <published>2007-01-01</published>
        <pages>42</pages>
        <toc/>
      </book>
    </library>

Try it:

.. code-block:: console

    $ python3 library_load.py library_test.xml
    book (id = 'foo'):
      author: F. Nord
      published: 2099-01-01
      npages: 23
      title:
        [en] 'The Amazing Life of Foo'
        [de] 'Das Faszinierende Leben des Foo'
      table of contents:
        1.    (page 1)
          [en] 'The Birth of Foo'
          [de] 'Die Geburt des Foo'
        2.    (page 3)
          [en] 'The Death of Foo'
          [de] 'Der Tod des Foo'
    book (id = 'pink-flamingos'):
      author: O. L. Bilderrahmen
      published: 2007-01-01
      npages: 42
      title:
        [en] 'The Relevance of Pink Flamingos to Computer Science'
        [de] 'Die Relevanz von rosa Flamingos für die Informatik'
      table of contents:

Reference Documentation
=======================

To learn more about XSO and how to use it, check out the reference
documentation in :mod:`aioxmpp.xso`. The remainder of this documentation will
now dive deeper into the details on how XSO works.