.. index:: internals

Traits internals
================

This section of the documentation is intended both for those developing Traits
itself, and for those using Traits who would like a better understanding of
the Traits internals.


Fundamentals
------------

The runtime behavior of Traits is governed by two key classes: |CTrait| and
|HasTraits|. Each of these classes inherits from a superclass implemented in
the |ctraits| C extension module: |CTrait| inherits from |cTrait|, while
|HasTraits| inherits from |CHasTraits|.

The |CHasTraits| and |cTrait| base classes are not intended for direct use and
do not provide a complete, coherent API: that API is provided by their
respective subclasses |HasTraits| and |CTrait|. The existence and precise
semantics of the base classes should be regarded as implementation details.
In what follows, we'll frequently refer to behavior of |HasTraits| and |CTrait|
even when the behavior we're referring to is actually implemented by
|CHasTraits| or |cTrait|.

We'll examine both of these classes, along with some of their hidden state, in
more detail below.


Key class: |HasTraits|
~~~~~~~~~~~~~~~~~~~~~~

The primary purpose of the |HasTraits| class is to override Python's normal
rules for attribute access. It does this by overriding the
``__getattribute__``, ``__setattr__`` and ``__delattr__`` special methods. At C
level, this is done through providing ``tp_getattro`` and ``tp_setattro``
"slots" to the |CHasTraits| ``PyTypeObject``: ``tp_getattro`` controls
attribute retrieval operations, while ``tp_setattro`` is called for both
attribute set and attribute deletion operations.

To support trait get and set operations, a |HasTraits| object (that is, in
normal use, an instance of a user-defined subclass of |HasTraits|) has two key
pieces of state: a dictionary of **class traits**, and a second dictionary of
**instance traits**. Each dictionary is a mapping from trait attribute names to
|CTrait| objects. A |CTrait| object encapsulates the rules for getting and
setting the corresponding attribute, as well as providing an attachment point
for trait notifications.

In addition to these two dictionaries, a |HasTraits| object has a ``__dict__``
that stores attribute values for that object in the normal way.

Analogously to class variables and instance variables in a normal Python class,
the class traits dictionary for a given |HasTraits| subclass is shared between
instances of that class, while the instance traits dictionary is specific to a
particular |HasTraits| instance. (As an internal optimization, the instance
traits dictionary is created on demand when first needed, while the class
traits dictionary is always present for each instance.)

For introspection and debugging purposes, both of these dictionaries can
be retrieved directly, using the |_instance_traits| and |_class_traits|
methods. However, it's not recommended to use either of these methods in
production code, and you should be especially careful when modifying either
of these dictionaries.

For example, consider the following |HasTraits| subclass::

    from traits.api import Float, HasTraits, Str

    class Ingredient(HasTraits):
        """ An ingredient in a recipe listing. """

        #: The name of the ingredient.
        name = Str()

        #: Ingredient quantity.
        quantity = Float()

Here's what happens when we create an instance of this ingredient and inspect
the instance and class trait dictionaries::

    >>> eggs = Ingredient(name="eggs", quantity=12)
    >>> eggs._instance_traits()
    {}
    >>> eggs._class_traits()
    {
        'name': <traits.ctrait.CTrait object at 0x1020cd400>,
        'quantity': <traits.ctrait.CTrait object at 0x1020cd360>,
        'trait_added': <traits.ctrait.CTrait object at 0x1020cd4a0>,
        'trait_modified': <traits.ctrait.CTrait object at 0x1020079a0>,
    }
    >>> eggs.__dict__
    {'name': 'eggs', 'quantity': 12.0}

Note that the actual values for the ingredient are stored in the ``__dict__``
as usual, not in the ``CTrait`` objects.

If we create a second ingredient, it shares class traits (but not
instance traits) with the first one::

    >>> flour = Ingredient(name="flour", quantity=3.5)
    >>> flour._class_traits() is eggs._class_traits()
    True
    >>> flour._instance_traits() is eggs._instance_traits()
    False


Key class: |CTrait|
~~~~~~~~~~~~~~~~~~~

A |CTrait| object has two main purposes:

- It encapsulates the rules for getting and setting a traited attribute on
  a |HasTraits| object.
- It provides an attachment point for trait notifiers.

The hidden state for a |CTrait| instance is encapsulated in the
``trait_object`` C ``struct`` in the |ctraits| source. There are several
interesting fields, not all of which are exposed at Python level.

Of particular interest are the ``getattr`` and ``setattr`` fields, which
hold pointers to C functions that act as the entry points for attribute
access via the given trait. See below for a fuller description of attribute
access mechanics.


Attribute retrieval
~~~~~~~~~~~~~~~~~~~

When evaluating ``obj.name`` for a ``HasTraits`` object ``obj``, the following
sequence of steps occurs (see ``has_traits_getattro`` in the C source):

- The ``name`` is looked up in ``obj.__dict__``. If found, the corresponding
  value is returned immediately.
- If ``name`` is not found in ``obj.__dict__``, we look first for an instance
  trait named ``name``, and then for a class trait named ``name``. Thus an
  instance trait with a given name will shadow a class trait with the same
  name.
- If a matching trait is found, its ``getattr`` field is invoked to retrieve
  the trait's value for the given object.
- If no matching trait is found, we try to access the attribute value
  using Python's usual attribute rules (via the ``PyObject_GenericGetAttr``
  C-API call).
- Finally, if the ``PyObject_GenericGetAttr`` call fails, we invoke the
  **prefix trait** machinery to get a new ``CTrait`` object, and use that
  new trait to get a value.

Note that the above sequence of steps applies to method access as well as
attribute access. Note also that there's no mechanism to automatically
search for ``CTrait`` objects in superclasses of the immediate ``HasTraits``
subclass.


Attribute set operations
~~~~~~~~~~~~~~~~~~~~~~~~

The rules for setting an attribute (evaluating ``obj.name = value`` for a
``HasTraits`` object ``obj``) are analogous to those for attribute retrieval.
The starting point is ``has_traits_setattro`` in the source.

- First we look for the name ``name`` in ``obj._instance_traits()``,
  and retrieve the corresponding ``CTrait`` instance if present.
- If no matching entry is found, we then look up ``name`` in
  ``obj._class_traits()``, and again retrieve the corresponding ``CTrait``.
- If still not found, we invoke the **prefix trait** machinery to get a new
  ``CTrait`` object. By default, this goes through the
  ``HasTraits.__prefix_trait__`` method (which is implemented in Python), and
  this may still fail with an exception.
- If one of the above steps succeeded, we now have a ``CTrait`` object, and
  its ``setattr`` function is invoked (passing along the trait object, ``obj``,
  ``name`` and ``value``) to perform the actual attribute set operation.


Attribute deletion
~~~~~~~~~~~~~~~~~~

Attribute deletion (``del obj.name``) goes through the same code path as
attribute set operations. Most ``CTrait`` types do not permit deletion.

Traits Containers
-----------------

Traits has the ability to watch for changes in standard Python containers:
lists, dictionaries and sets.  To achieve this Traits provides special
subclasses of the standard Python classes that can validate elements and can
fire trait notifications when the contents change.  These classes are,
respectively, |TraitList|, |TraitDict| and |TraitSet| (not to be confused with
the deprecated Trait Handlers of the same names).

In addition to being able to take an appropriate value to initialize the
container (such as a sequence or mapping), these container subclasses also
take keyword-only arguments for validators (either a single item validator for
|TraitList| and |TraitSet|, or key and value validators for |TraitDict|) and
notifiers.

These classes were introduced in Traits 6.1 and the implementation details
described here are provisional and may change.

Backwards Compatibility
~~~~~~~~~~~~~~~~~~~~~~~

In practice, most traits containers are instances of |TraitListObject|,
|TraitDictObject| and |TraitSetObject| which are subclasses that are
created by the validators of |List|, |Dict| and |Set| traits respectively and
supplied with appropriate element validators that wrap the standard trait
validators so that constructs like ``List(Str)`` can work.

These objects are designed to be API compatible with the classes of the
same name from Traits 6.0 and before, and so have different constructors and
may behave slightly differently from the base classes in some cases.

In particular |TraitListObject| classes can have restrictions on their
length, which is not part of the base |TraitList| API.

All of these backward compatibility classes are strongly tied to a particular
object and trait, and are not designed to operate as stand-alone entities.
These relationships are required to support the `object.*_items`-style event
traits, but complicate the implementation, for example by having to use
weak references to the object, and having to take this additional structure
into account when serializing and deserializing.

It is a long-term goal to phase out the use of these backwards compatibility
classes.

Container Validators
~~~~~~~~~~~~~~~~~~~~

Container validators are callables that are expected to take a single value
and either return a validated value or raise a TraitError.  The default
validators simply pass the input value directly through, but validators can
potentially do much more.   Validators are expected to be idempotent:
``validate(validate(x)) == validate(x)`` should hold as long as
``validate(x)`` does not raise an error.

For example, the following validator casts the list items to integers,
raising a TraitError if that fails::

    def int_validator(value):
        try:
            return int(value)
        except ValueError:
            raise TraitError(
                "List items must be castable to an int, but a value %r was specified."
                % value
            )

So if we were to use this as the validator for a |TraitList| we would get the
following behaviour::

    >>> int_list = TraitList(item_validator=int_validator)
    >>> int_list.append("5")
    >>> int_list
    [5]
    >>> int_list.extend([3, 5, "aaaarrrghh"])
    TraitError: List items must be castable to an int, but a value 'aaaarrrghh' was specified.

In Traits 6.1 validation is not done uniformly before performing operations
to keep behaviour the same as Traits 6.0 and earlier, so results of operations can
sometimes be surprising::

    >>> int_list.append("6")
    >>> int_list.remove("6")
    ValueError: list.remove(x): x not in list

This is expected to be resolved in a future traits version by providing clear
guidance to users about when validation is performed, and possibly changing the
behaviour.

Container Notifiers
~~~~~~~~~~~~~~~~~~~

Container objects also have a list of notifiers that fire when the contents of
the container change.  Like trait notifiers, container notifiers are low-level
callbacks that are used by the higher-level, more user-friendly observer and
listener systems.  They have a fixed signature, which is slightly different for
lists, dicts and sets, but in all cases starts with the container object itself.
Notifiers are called in the order that they appear in the notifiers list and
should not mutate the parameters that they have been passed.

List notifiers must take 4 arguments: the **trait_list** object, the **index**
value or slice that identifies where the change occurred, a list of
**removed** elements, and a list of **added** elements.  The |TraitList|
methods make an attempt to normalize indices and slices to make things easier
for notification writers.

Dict notifiers expect 4 arguments: the **trait_dict** object, a dictionary of
**removed** items, a dictionary of **added** elements, and a dictionary of
**changed** items, where the values are the old values held in the keys.

Set notifiers expect 3 arguments: the **trait_set** object, the set of
**removed** elements, and the set of **added** elements.

Users should not usually need to interact with the container notifiers directly,
just as they do not usually need to interact with trait notifiers.


..
   # substitutions

.. |_class_traits| replace:: :meth:`~traits.ctraits.CHasTraits._class_traits`
.. |_instance_traits| replace:: :meth:`~traits.ctraits.CHasTraits._instance_traits`
.. |cTrait| replace:: :class:`~traits.ctraits.cTrait`
.. |CTrait| replace:: :class:`~traits.ctrait.CTrait`
.. |ctraits| replace:: :mod:`~traits.ctraits`
.. |CHasTraits| replace:: :class:`~traits.ctraits.CHasTraits`
.. |HasTraits| replace:: :class:`~traits.has_traits.HasTraits`
.. |TraitList| replace:: :class:`~traits.trait_list_object.TraitList`
.. |TraitDict| replace:: :class:`~traits.trait_dict_object.TraitDict`
.. |TraitSet| replace:: :class:`~traits.trait_dict_object.TraitSet`
.. |TraitListObject| replace:: :class:`~traits.trait_list_object.TraitListObject`
.. |TraitDictObject| replace:: :class:`~traits.trait_dict_object.TraitDictObject`
.. |TraitSetObject| replace:: :class:`~traits.trait_dict_object.TraitSetObject`
.. |List| replace:: :class:`~traits.trait_types.List`
.. |Dict| replace:: :class:`~traits.trait_types.Dict`
.. |Set| replace:: :class:`~traits.trait_types.Set`