API Microversions
=================

Background
----------

Cinder uses a framework we called 'API Microversions' for allowing changes
to the API while preserving backward compatibility. The basic idea is
that a user has to explicitly ask for their request to be treated with
a particular version of the API. So breaking changes can be added to
the API without breaking users who don't specifically ask for it. This
is done with an HTTP header ``OpenStack-API-Version`` which
is a monotonically increasing semantic version number starting from
``3.0``.

Each OpenStack service that uses microversions will share this header, so
the Volume service will need to prefix the semantic version number with the
word ``volume``::

  OpenStack-API-Version: volume 3.0

If a user makes a request without specifying a version, they will get
the ``_MIN_API_VERSION`` as defined in
``cinder/api/openstack/api_version_request.py``. This value is currently
``3.0`` and is expected to remain so for quite a long time.

The Nova project was the first to implement microversions. For full
details please read Nova's `Kilo spec for microversions
<https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html>`_

When do I need a new Microversion?
----------------------------------

A microversion is needed when the contract to the user is
changed. The user contract covers many kinds of information such as:

- the Request

  - the list of resource URLs which exist on the server

    Example: adding a new shares/{ID}/foo which didn't exist in a
    previous version of the code

  - the list of query parameters that are valid on URLs

    Example: adding a new parameter ``is_yellow`` servers/{ID}?is_yellow=True

  - the list of query parameter values for non free form fields

    Example: parameter filter_by takes a small set of constants/enums "A",
    "B", "C". Adding support for new enum "D".

  - new headers accepted on a request

- the Response

  - the list of attributes and data structures returned

    Example: adding a new attribute 'locked': True/False to the output
    of shares/{ID}

  - the allowed values of non free form fields

    Example: adding a new allowed ``status`` to shares/{ID}

  - the list of status codes allowed for a particular request

    Example: an API previously could return 200, 400, 403, 404 and the
    change would make the API now also be allowed to return 409.

  - changing a status code on a particular response

    Example: changing the return code of an API from 501 to 400.

  - new headers returned on a response

The following flow chart attempts to walk through the process of "do
we need a microversion".


.. graphviz::

   digraph states {

    label="Do I need a microversion?"

    silent_fail[shape="diamond", style="", label="Did we silently
   fail to do what is asked?"];
    ret_500[shape="diamond", style="", label="Did we return a 500
   before?"];
    new_error[shape="diamond", style="", label="Are we changing what
    status code is returned?"];
    new_attr[shape="diamond", style="", label="Did we add or remove an
    attribute to a payload?"];
    new_param[shape="diamond", style="", label="Did we add or remove
    an accepted query string parameter or value?"];
    new_resource[shape="diamond", style="", label="Did we add or remove a
   resource URL?"];


   no[shape="box", style=rounded, label="No microversion needed"];
   yes[shape="box", style=rounded, label="Yes, you need a microversion"];
   no2[shape="box", style=rounded, label="No microversion needed, it's
   a bug"];

   silent_fail -> ret_500[label="no"];
   silent_fail -> no2[label="yes"];

    ret_500 -> no2[label="yes [1]"];
    ret_500 -> new_error[label="no"];

    new_error -> new_attr[label="no"];
    new_error -> yes[label="yes"];

    new_attr -> new_param[label="no"];
    new_attr -> yes[label="yes"];

    new_param -> new_resource[label="no"];
    new_param -> yes[label="yes"];

    new_resource -> no[label="no"];
    new_resource -> yes[label="yes"];

   {rank=same; yes new_attr}
   {rank=same; no2 ret_500}
   {rank=min; silent_fail}
   }

If a patch that will require a microversion increment is proposed
having similar intention and code with a previously merged patch
given the previous merged patch hasn't been released, then the
previously merged patch could be modified to include the new patch
code under the same microversion.

**Footnotes**

[1] - When fixing 500 errors that previously caused stack traces, try
to map the new error into the existing set of errors that API call
could previously return (400 if nothing else is appropriate). Changing
the set of allowed status codes from a request is changing the
contract, and should be part of a microversion.

The reason why we are so strict on contract is that we'd like
application writers to be able to know, for sure, what the contract is
at every microversion in Cinder. If they do not, they will need to write
conditional code in their application to handle ambiguities.

When in doubt, consider application authors. If it would work with no
client side changes on both Cinder versions, you probably don't need a
microversion. If, on the other hand, there is any ambiguity, a
microversion is probably needed.


In Code
-------

In ``cinder/api/openstack/wsgi.py`` we define an ``@api_version`` decorator
which is intended to be used on top-level Controller methods. It is
not appropriate for lower-level methods. Some examples:

Adding a new API method
~~~~~~~~~~~~~~~~~~~~~~~

In the controller class::

    @wsgi.Controller.api_version("3.4")
    def my_api_method(self, req, id):
        ....

This method would only be available if the caller had specified an
``OpenStack-API-Version`` of >= ``3.4``. If they had specified a
lower version (or not specified it and received the default of ``3.1``)
the server would respond with ``HTTP/404``.

Removing an API method
~~~~~~~~~~~~~~~~~~~~~~

In the controller class::

    @wsgi.Controller.api_version("3.1", "3.4")
    def my_api_method(self, req, id):
        ....

This method would only be available if the caller had specified an
``OpenStack-API-Version`` of <= ``3.4``, and >= ``3.1``. If ``3.5`` or later
is specified or if ``3.0`` or earlier (/v2 or /v1 endpoint), the server will
respond with ``HTTP/404``

Changing a method's behaviour
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the controller class::

    @wsgi.Controller.api_version("3.1", "3.3")
    def my_api_method(self, req, id):
        .... method_1 ...

    @my_api_method.api_version("3.4")
    def my_api_method(self, req, id):
        .... method_2 ...

If a caller specified ``3.1``, ``3.2`` or ``3.3`` (or received the
default of ``3.1``) they would see the result from ``method_1``,
``3.4`` or later ``method_2``.

We could use ``wsgi.Controller.api_version`` decorator on the second
``my_api_method`` as well, but then we would have to add ``# noqa`` to that
line to avoid failing flake8's ``F811`` rule.  So the recommended approach is
to use the ``api_version`` decorator from the first method that is defined, as
illustrated by the example above, and then use ``my_api_method`` decorator for
subsequent api versions of the same method.

The two methods may be different in any kind of semantics (schema validation,
return values, response codes, etc.).

A method with only small changes between versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A method may have only small changes between microversions, in which
case you can decorate a private method::

    @wsgi.Controller.api_version("3.1", "3.4")
    def _version_specific_func(self, req, arg1):
        pass

    @_version_specific_func.api_version(min_ver="3.5")
    def _version_specific_func(self, req, arg1):
        pass

    def show(self, req, id):
        .... common stuff ....
        self._version_specific_func(req, "foo")
        .... common stuff ....

When not using decorators
~~~~~~~~~~~~~~~~~~~~~~~~~

When you don't want to use the ``@api_version`` decorator on a method
or you want to change behaviour within a method (say it leads to
simpler or simply a lot less code) you can directly test for the
requested version with a method as long as you have access to the api
request object (commonly called ``req``). Every API method has an
api_version_request object attached to the req object and that can be
used to modify behaviour based on its value::

    def index(self, req):
        <common code>

        req_version = req.api_version_request
        if req_version.matches("3.1", "3.5"):
            ....stuff....
        elif req_version.matches("3.6", "3.10"):
            ....other stuff....
        elif req_version > api_version_request.APIVersionRequest("3.10"):
            ....more stuff.....

        <common code>

The first argument to the matches method is the minimum acceptable version
and the second is maximum acceptable version. A specified version can be null::

    null_version = APIVersionRequest()

If the minimum version specified is null then there is no restriction on
the minimum version, and likewise if the maximum version is null there
is no restriction the maximum version. Alternatively an one sided comparison
can be used as in the example above.

Other necessary changes
-----------------------

If you are adding a patch which adds a new microversion, it is
necessary to add changes to other places which describe your change:

* Update ``REST_API_VERSION_HISTORY`` in
  ``cinder/api/openstack/api_version_request.py``

* Update ``_MAX_API_VERSION`` in
  ``cinder/api/openstack/api_version_request.py``

* Add a verbose description to
  ``cinder/api/openstack/rest_api_version_history.rst``.  There should
  be enough information that it could be used by the docs team for
  release notes.

* Constants should be used in the code to minimize errors on microversion
  merge conflicts. Define a constant for the new microversion in the
  ``cinder/api/microversions.py`` file and use that in the rest of the code.

* Update the expected versions in affected tests.

* API changes should almost always include a release note announcing the
  availability of the new API functionality. The description of the API change
  should indicate which microversion is required for the change, and it should
  refer to the numerical value of the microversion and not its constant name.

* Update the ``version`` parameter in api-ref responses here ``cinder/api-ref/
  source/v3/samples/versions/version-show-response.json`` and here
  ``cinder/api-ref/source/v3/samples/versions/versions-response.json``
  to the latest microversion to avoid functional test failure.

* If the API microversion has changed an endpoint accepted parameters or the
  values it returns, we need to create the appropriate API samples within
  the ``api-ref/source/v3/samples`` tree creating a new ``vX.Y`` directory
  with our request and/or response json.

* Update the functional API tests in the
  ``cinder/tests/functional/api_sample_tests`` tree to make requests and
  validate responses with the new microversion.  There are multiple convenience
  methods provided for testing, such as ``use_versions`` class decorator that
  allows us to run the same tests with different microversions (each will use
  their respective json and templates), the ``override_mv`` method decorator to
  change the microversion in a single test, and the ``common_api_sample``
  context manager to use the base sample instead of a microversion specific
  one.

* Update the documentation adding any new parameter to
  ``api-ref/source/v3/parameters.yaml`` (remember to add the ``min_version``)
  and then making appropriate changes to the ``.inc`` file in
  ``api-ref/source/v3/`` to reflect new possible return codes, new accepted
  parameters and their ``Request Example (vX.Y)`` title and include file, and
  returned values and their ``Response Example (vX.Y)`` title and include file.

The Cinder project's policy is that the sample requests and responses should
always reflect the *most recent* microversion.

Allocating a microversion
-------------------------

If you are adding a patch which adds a new microversion, it is
necessary to allocate the next microversion number. Except under
extremely unusual circumstances and this would have been mentioned in
the blueprint for the change, the minor number of ``_MAX_API_VERSION``
will be incremented. This will also be the new microversion number for
the API change.

It is possible that multiple microversion patches would be proposed in
parallel and the microversions would conflict between patches.  This
will cause a merge conflict. We don't reserve a microversion for each
patch in advance as we don't know the final merge order. Developers
may need over time to rebase their patch calculating a new version
number as above based on the updated value of ``_MAX_API_VERSION``.

Testing Microversioned API Methods
----------------------------------

Unit tests for microversions should be put in cinder/tests/unit/api/v3/ .
Since all existing functionality is tested in cinder/tests/unit/api/v2,
these unit tests are not replicated in .../v3, and only new functionality
needs to be place in the .../v3/directory.

Testing a microversioned API method is very similar to a normal controller
method test, you just need to add the ``OpenStack-API-Version``
header, for example::

    req = fakes.HTTPRequest.blank('/testable/url/endpoint')
    req.headers['OpenStack-API-Version'] = 'volume 3.6'
    req.api_version_request = api_version.APIVersionRequest('3.6')

    controller = controller.TestableController()

    res = controller.index(req)
    ... assertions about the response ...

REST API Version History
------------------------

Details for each existing microversion change can be found in the
:doc:`REST API Version History <api_microversion_history>` documentation.

.. toctree::
    :hidden:

    api_microversion_history