..  Copyright 2017 David Malcolm <dmalcolm@redhat.com>
    Copyright 2017 Red Hat, Inc.

    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
    License as published by the Free Software Foundation; either
    version 2.1 of the License, or (at your option) any later version.

    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Lesser General Public License for more details.

    You should have received a copy of the GNU Lesser General Public
    License along with this library; if not, write to the Free Software
    Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
    USA

Data model
==========

.. This could use an XML domain for Sphinx.

.. py:module:: firehose.model

.. py:class:: Analysis

   The :py:class:`Analysis` class represents one invocation of a
   code analysis tool.

   It corresponds to the ``<analysis>`` XML element, the top-level element
   of a Firehose XML document.

   .. py:attribute:: metadata

      :py:class:`Metadata`

   .. py:attribute:: results

      A list of :py:class:`Result` objects, representing the various
      issues, failures, and other information found during the analysis.

   .. py:attribute:: customfields

      :py:class:`CustomFields` or ``None``

   Here is the pertinent part of the XML schema:

   .. literalinclude:: ../firehose.rng
      :start-after: datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes">
      :end-before:   <define name="metadata-element">
      :language: xml

  .. py:method:: __init__(self, metadata, results, customfields=None):

     :param metadata:
     :type metadata: :py:class:`Metadata`
     :param results:
     :type results: list(:py:class:`Result`)
     :param customfields:
     :type customfields: :py:class:`CustomFields` or None

  ..
     TODO: talk about to_xml, from_xml, to_json, from_json

  .. py:classmethod:: from_xml(cls, fileobj)

     Parse XML from fileobj, and return an :py:class:`Analysis` instance
     representing the data seen there.

  .. py:method:: to_xml(self)

     Generate an :py:class:`ET.ElementTree()` representing the data
     within self.

  .. py:method:: to_xml_bytes(self)

     Generate a ``bytes`` instance containing an XML serialization of the
     data within self.

..
      def accept(self, visitor):

      def fixup_files(self, relativedir=None, hashalg=None):
          Record the absolute path of each file, and record the digest of the
          file content

      def set_custom_field(self, name, value):


Results
*******

.. py:class:: Result

   Result is a base class

   There are three subclasses:

   * an :py:class:`Issue` represents a report from the analyzer about a
     possible problem with the software under test.

   * an :py:class:`Info` represents additional kinds of information
     generated by an analyzer that isn't a problem per-se e.g.
     code metrics, licensing info, etc.

   * a :py:class:`Failure` represents a report about a failure of the
     analyzer itself (e.g. if the analyzer crashed).


.. py:class:: Issue(Result)

   An :py:class:`Issue` represents a report from the analyzer about a
   possible problem with the software under test.

   It corresponds to the ``<issue>`` XML element within a
   Firehose XML document.

   .. py:attribute:: cwe

      (``int`` or ``None``): The Common Weakness Enumeration ID
      (see http://cwe.mitre.org/index.html )
      e.g. "131" representing CWE-131
      aka "Incorrect Calculation of Buffer Size"
      http://cwe.mitre.org/data/definitions/131.html

   .. py:attribute:: testid

      (``str`` or ``None``): Each static analysis tool potentially has
      multiple tests, with its own IDs for its own tests.
      These can be captured here, as free-form strings.

   .. py:attribute:: location

      (:py:class:`Location`): Where is the problem?

   .. py:attribute:: message

      (:py:class:`Message`): A message summarizing the problem.

   .. py:attribute:: notes

      (:py:class:`Notes` or ``None``): Additional descriptive details.

   .. py:attribute:: trace

      (:py:class:`Trace` or ``None``): An optional list of events that
      describe the circumstances leading up to a problem.

   .. py:attribute:: severity

      (``str`` or ``None``): Each static analysis tool potentially can
      report a "severity", which may be of use for filtering.

      The precise strings are likely to vary from tool to tool.
      To avoid data-transfer issues, support storing it as an optional
      freeform string here.

      See:
      http://lists.fedoraproject.org/pipermail/firehose-devel/2013-February/000001.html

   .. py:attribute:: customfields

      (:py:class:`CustomFields` or ``None``): A given tool/testid may have
      additional key/value pairs that it may be useful to capture.


    ..
      def __init__(self,
                   cwe,
                   testid,
                   location,
                   message,
                   notes,
                   trace,
                   severity=None,
                   customfields=None):
          if cwe is not None:
              assert isinstance(cwe, int)
          if testid is not None:
              assert isinstance(testid, _string_type)
          assert isinstance(location, Location)
          assert isinstance(message, Message)
          if notes:
              assert isinstance(notes, Notes)
          if trace:
              assert isinstance(trace, Trace)
          if severity is not None:
              assert isinstance(severity, _string_type)
          if customfields is not None:
              assert isinstance(customfields, CustomFields)
          self.cwe = cwe
          self.testid = testid
          self.location = location
          self.message = message
          self.notes = notes
          self.trace = trace
          self.severity = severity
          self.customfields = customfields

   .. py:method:: write_as_gcc_output(self, out)

      Write the issue in the style of a GCC warning to the given
      file-like object.

        >>> issue.write_as_gcc_output(sys.stderr)
        examples/python-src-example.c:40:4: warning: ob_refcnt of '*item' is 1 too high [CWE-401]
        was expecting final item->ob_refcnt to be N + 1 (for some unknown N)
        due to object being referenced by: PyListObject.ob_item[0]
        but final item->ob_refcnt is N + 2
        examples/python-src-example.c:36:14: note: PyLongObject allocated at:         item = PyLong_FromLong(random());
        examples/python-src-example.c:37:8: note: when PyList_Append() succeeds

   .. py:method:: get_cwe_str(self)

      Get a string giving the CWE title, or None::

        >>> issue.get_cwe_str()
        'CWE-131'

   .. py:method:: get_cwe_url(self)

      Get a string containing the URL of the CWE id, or None::

        >>> issue.get_cwe_url()
        'http://cwe.mitre.org/data/definitions/131.html'


.. py:class:: Info(Result)

   An :py:class:`Info` represents additional kinds of information
   generated by an analyzer that isn't a problem per-se e.g.
   code metrics, licensing info, cross-referencing information, etc.

   It corresponds to the ``<info>`` XML element within a
   Firehose XML document.

   .. py:attribute:: infoid

      (``str`` or ``None``): an optional free-form string
      identifying the kind of information being reported.

   .. py:attribute:: location

      :py:class:`Location` or ``None``

   .. py:attribute:: message

      :py:class:`Message` or ``None``

   .. py:attribute:: customfields

      :py:class:`CustomFields` or ``None``

..
      def __init__(self, infoid, location, message, customfields):
          if infoid is not None:
              assert isinstance(infoid, _string_type)
          if location is not None:
              assert isinstance(location, Location)
          if message is not None:
              assert isinstance(message, Message)
          if customfields is not None:
              assert isinstance(customfields, CustomFields)
          self.infoid = infoid
          self.location = location
          self.message = message
          self.customfields = customfields


.. py:class:: Failure(Result)

   A :py:class:`Failure` represents a report about a failure of the
   analyzer itself (e.g. if the analyzer crashed).

   If any of these are present then we don't have full coverage.

   For some analyzers this is an all-or-nothing affair: we either
   get issues reported, or a failure happens (e.g. a segfault of the
   analysis tool).

   Other analyzers may be more fine-grained: able to report some issues,
   but choke on some subset of the code under analysis.  For example
   cpychecker runs once per function, and any unhandled Python exceptions
   only affect one function.

   It corresponds to the ``<failure>`` XML element within a
   Firehose XML document.

   .. py:attribute:: failureid

      (``str`` or ``None``): Each static analysis tool potentially can
      identify types of way that it can fail.

      Capture those that do here, as (optional) free-form strings.

   .. py:attribute:: location

      :py:class:`Location`: Some analysis tools may be able to annotate
      a failure report by providing the location *within the
      software-under-test* that broke them.

      For example, gcc-python-plugin has a ``gcc.set_location()`` method
      which can be used by a code analysis script to record what
      location is being analyzed, so that if unhandled Python exception
      happens, it is reported at that location.  This is invaluable
      when debugging analysis failures.

   .. py:attribute:: message

      :py:class:`Message`: A summary of the failure.

   .. py:attribute:: customfields

      :py:class:`CustomFields` or ``None``:
      Every type of failure seems to have its own kinds of data that
      are worth capturing:

           * stdout/stderr/returncode for a failed subprocess

           * traceback for an unhandled Python exception

           * verbose extra information about a cppcheck failure

      etc.
      Hence we allow a ``<failure>`` to optionally contain extra key/value
      pairs, based on the ``failureid``.

..
      def __init__(self, failureid, location, message, customfields):
          if failureid is not None:
              assert isinstance(failureid, _string_type)
          if location is not None:
              assert isinstance(location, Location)
          if message is not None:
              assert isinstance(message, Message)
          if customfields is not None:
              assert isinstance(customfields, CustomFields)
          self.failureid = failureid
          self.location = location
          self.message = message
          self.customfields = customfields


Metadata
********

.. py:class:: Metadata

   Holder for metadata about an analyzer invocation.

   It corresponds to the ``<metadata>`` XML element within a
   Firehose XML document.

   .. py:attribute:: generator

      :py:class:`Generator`

   .. py:attribute:: sut

      :py:class:`Sut` or ``None``

   .. py:attribute:: file_

      :py:class:`File` or ``None``

   .. py:attribute:: stats

      :py:class:`Stats` or ``None``

..
      def __init__(self, generator, sut, file_, stats):
          assert isinstance(generator, Generator)
          if sut is not None:
              assert isinstance(sut, Sut)
          if file_ is not None:
              assert isinstance(file_, File)
          if stats is not None:
              assert isinstance(stats, Stats)
          self.generator = generator
          self.sut = sut
          self.file_ = file_
          self.stats = stats

.. py:class:: Generator

   .. py:attribute:: name

      ``str``

   .. py:attribute:: version

      ``str`` or ``None``

.. py:class:: Stats

   :py:class:`Stats` is an optional field of :py:class:`Metadata` for
   capturing stats about an analysis run.

   .. py:attribute:: wallclocktime

      ``float``: how long (in seconds) the analyzer took to run


Describing the software under test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. warning:: this part of the schema may need more thought/work

.. py:class:: Sut

   Base class for describing the software-under-test.

.. py:class:: SourceRpm(Sut)

   It corresponds to the ``<source-rpm>`` XML element within
   a Firehose XML document.

   .. py:attribute:: name

      ``str``

   .. py:attribute:: version

      ``str``

   .. py:attribute:: release

      ``str``

   .. py:attribute:: buildarch

      ``str``

.. py:class:: DebianBinary(Sut)

   Internal Firehose representation of a Debian binary package. This Object
   is extremely similar to a SourceRpm.

   It corresponds to the ``<debian-binary>`` XML element within
   a Firehose XML document.

      .. py:attribute:: name

         ``str``: the *binary* package name.

      .. py:attribute:: version

         ``str``: should match Upstream's version number

      .. py:attribute:: release

         ``str`` or ``None``: should be the Debian package local version.
         This should only be omited if the package is a Debian Native package.

      .. py:attribute:: buildarch

         ``str``: valid entries include ``amd64`', ``kfreebsd-amd64``,
         ``armhf``, ``hurd-i386``, among others for Debian.

.. py:class:: DebianSource(Sut)

   Internal Firehose representation of a Debian source package. This Object
   is extremely similar to a SourceRpm, but does not include the `buildarch`
   attribute.

   It corresponds to the ``<debian-source>`` XML element within
   a Firehose XML document.

   .. py:attribute:: name

      ``str``: should be the *source* package name

   .. py:attribute:: version

      ``str``: should match Upstream's version number

   .. py:attribute:: release

      ``str`` or ``None``: if given, should be the Debian package local
      version. This should only be omited if the package is a
      Debian Native package.


.. py:class:: Message

   Summary text aimed at a developer.  This is required for an
   :py:class:`Issue`, but is also can (optionally) be provided by
   a :py:class:`Failure` or :py:class:`Info`.

   It corresponds to the ``<message>`` XML element within
   a Firehose XML document.

   .. py:attribute:: text

      ``str``


.. py:class:: Notes

   Additional optional descriptive details for a :py:class:`Result` or
   for a :py:class:`State`.

   It corresponds to the ``<notes>`` XML element within
   a Firehose XML document.

   .. py:attribute:: text

      ``str``

   .. This might support some simple markup at some point
      (as might :py:class:`Message`).

Describing source code
**********************

.. py:class:: Location

   A particular source code location.

   It corresponds to the ``<location>`` XML element within
   a Firehose XML document.

   .. py:attribute:: file

      :py:class:`File`

   .. py:attribute:: function

      :py:class:`Function` or ``None``.  The function (or method)
      containing the problem.

      This is optional.  Some problems occur in global scope, and
      unfortunately, some analyzers don't always report which function each
      problem was discovered in.  Given that function names are less
      likely to change than line numbers, this is something that we
      should patch in each upstream analyzer as we go.

   We can refer to either a location, or a range of locations
   within the file:

   .. py:attribute:: point

      :py:class:`Point` or ``None``

   .. py:attribute:: range_

      :py:class:`Range` or ``None``

..
      def __init__(self, file, function, point=None, range_=None):
          assert isinstance(file, File)
          if function is not None:
              assert isinstance(function, Function)
          if point is not None:
              assert isinstance(point, Point)
          if range_ is not None:
              assert isinstance(range_, Range)
          self.file = file
          self.function = function
          self.point = point
          self.range_ = range_
      @property
      def line(self):
          if self.point is not None:
              return self.point.line
          if self.range_ is not None:
              return self.range_.start.line
      @property
      def column(self):
          if self.point is not None:
              return self.point.column
          if self.range_ is not None:
              return self.range_.start.column

.. py:class:: File

   A description of a particular source file.

   It corresponds to the ``<file>`` XML element within
   a Firehose XML document.

     .. py:attribute:: givenpath

        ``str``: the filename given by the analyzer.

        This is typically the one supplied to it on the command line,
        which might be absolute or relative.

        Examples:

           * "foo.c"

           *  "./src/foo.c"

           *  "/home/david/libfoo-1.0/src/foo.c"

     .. py:attribute:: abspath

        (``str`` or ``None``): Optionally, a record of the absolute path
        of the file, to help deal with collating results from a build
        that changes working directory (e.g. recursive make).

     .. py:attribute:: hash_

        (:py:class:`Hash` or ``None``)

..
      def __init__(self, givenpath, abspath, hash_=None):
          assert isinstance(givenpath, ``str``)
          if abspath is not None:
              assert isinstance(abspath, ``str``)
          if hash_ is not None:
              assert isinstance(hash_, Hash)
          self.givenpath = givenpath
          self.abspath = abspath
          self.hash_ = hash_

.. py:class:: Hash

   An optional value within :py:class:`File`, allowing the report to
   specify a hash value for a particular file.

   This can be used for tracking different versions of files when
   collating different reports and e.g. for caching file content
   in a UI.

   It corresponds to the ``<hash>`` XML element within
   a Firehose XML document.

   .. py:attribute:: alg

      ``str``: the name of the hash algorithm.

      TODO: what naming convention?

   .. py:attribute:: hexdigest

      ``str``: the hexadecimal value of the digest (lower-case
      hexdigits, without any leading `0x`).

..
      def __init__(self, alg, hexdigest):
          assert isinstance(alg, ``str``)
          assert isinstance(hexdigest, ``str``)
          self.alg = alg
          self.hexdigest = hexdigest

.. py:class:: Function

   Identification of a particular function within source code.

   It corresponds to the ``<function>`` XML element within
   a Firehose XML document.

   .. py:attribute:: name

      ``str``: the name of the function or method.

..
      def __init__(self, name):
          self.name = name

.. py:class:: Point

   Identification of a particular line/column within a source file.

   It corresponds to the ``<point>`` XML element within
   a Firehose XML document.

   .. py:attribute:: line

      ``int``: the 1-based number of the line containing the point

   .. py:attribute:: column

      ``int``: 1-based number of the column

      .. note::

         GCC uses a 1-based convention for source columns,
         whereas Emacs's ``M-x column-number-mode`` uses a 0-based convention.

         For example, an error in the initial, left-hand
         column of source line 3 is reported by GCC as::

           some-file.c:3:1: error: ...etc...

         On navigating to the location of that error in Emacs
         (e.g. via ``next-error``), the locus is reported in the Mode Line
         (assuming ``M-x column-number-mode``) as::

           some-file.c   10%   (3, 0)

         i.e. ``3:1:`` in GCC corresponds to ``(3, 0)`` in Emacs.

.. py:class:: Range

   Identification of a range of text within a source file.

   It corresponds to the ``<range>`` XML element within
   a Firehose XML document.

   .. py:attribute:: start

      (:py:class:`Point`)

   .. py:attribute:: end

      (:py:class:`Point`)

Capturing the circumstances leading up to a problem
***************************************************

.. py:class:: Trace

   An optional list of events within an :py:class:`Issue` that describe
   the circumstances leading up to a problem.

   It corresponds to the ``<trace>`` XML element within a
   Firehose XML document.

   See :ref:`example of a trace <trace_example>`.

   .. py:attribute:: states

      list of :py:class:`State`

.. py:class:: State

   A state within a :py:class:`Trace`.

   .. py:attribute:: location

      :py:class:`Location`

   .. py:attribute:: notes

     :py:class:`Notes` or ``None``

..
      def __init__(self, location, notes):
          assert isinstance(location, Location)
          if notes is not None:
              assert isinstance(notes, Notes)
          self.location = location
          self.notes = notes

Other data
**********

.. py:class:: CustomFields(OrderedDict)

   A big escape-hatch in the data model: support for arbitrary, ordered
   key/value pairs for roundtripping data specific to a particular
   situation.  e.g. debugging attributes for a particular failure

   It corresponds to the ``<custom-fields>`` XML element within
   a Firehose XML document.


.. TODO:

   * .. py:class:: Visitor
   * main