.. _NEP26:

====================================================
NEP 26 — Summary of missing data NEPs and discussion
====================================================

:Author: Mark Wiebe <mwwiebe@gmail.com>, Nathaniel J. Smith <njs@pobox.com>
:Status: Deferred
:Type: Standards Track
:Created: 2012-04-22

*Context*: this NEP was written as summary of the large number of discussions
and proposals (:ref:`NEP12`, :ref:`NEP24`, :ref:`NEP25`), regarding missing data
functionality.

The debate about how NumPy should handle missing data, a subject with
many preexisting approaches, requirements, and conventions, has been long and
contentious. There has been more than one proposal for how to implement
support into NumPy, and there is a testable implementation which is
merged into NumPy's current main. The vast number of emails and differing
points of view has made it difficult for interested parties to understand
the issues and be comfortable with the direction NumPy is going.

Here is our (Mark and Nathaniel's) attempt to summarize the
problem, proposals, and points of agreement/disagreement in a single
place, to help the community move towards consensus.

The NumPy developers' problem
=============================

For this discussion, "missing data" means array elements
which can be indexed (e.g. A[3] in an array A with shape (5,)),
but have, in some sense, no value.

It does not refer to compressed or sparse storage techniques where
the value for A[3] is not actually stored in memory, but still has a
well-defined value like 0.

This is still vague, and to create an actual implementation,
it is necessary to answer such questions as:

* What values are computed when doing element-wise ufuncs.
* What values are computed when doing reductions.
* Whether the storage for an element gets overwritten when marking
  that value missing.
* Whether computations resulting in NaN automatically treat in the
  same way as a missing value.
* Whether one interacts with missing values using a placeholder object
  (e.g. called "NA" or "masked"), or through a separate boolean array.
* Whether there is such a thing as an array object that cannot hold
  missing array elements.
* How the (C and Python) API is expressed, in terms of dtypes,
  masks, and other constructs.
* If we decide to answer some of these questions in multiple ways,
  then that creates the question of whether that requires multiple
  systems, and if so how they should interact.

There's clearly a very large space of missing-data APIs that *could*
be implemented. There is likely at least one user, somewhere, who
would find any possible implementation to be just the thing they
need to solve some problem. On the other hand, much of NumPy's power
and clarity comes from having a small number of orthogonal concepts,
such as strided arrays, flexible indexing, broadcasting, and ufuncs,
and we'd like to preserve that simplicity.

There has been dissatisfaction among several major groups of NumPy users
about the existing status quo of missing data support. In particular,
neither the numpy.ma component nor use of floating-point NaNs as a
missing data signal fully satisfy the performance requirements and
ease of use for these users. The example of R, where missing data
is treated via an NA placeholder and is deeply integrated into all
computation, is where many of these users point to indicate what
functionality they would like. Doing a deep integration of missing
data like in R must be considered carefully, it must be clear it
is not being done in a way which sacrifices existing performance
or functionality.

Our problem is, how can we choose some incremental additions to
NumPy that will make a large class of users happy, be
reasonably elegant, complement the existing design, and that we're
comfortable we won't regret being stuck with in the long term.

Prior art
=========

So a major (maybe *the* major) problem is figuring out how ambitious
the project to add missing data support to NumPy should be, and which
kinds of problems are in scope. Let's start with the
best understood situation where "missing data" comes into play:

"Statistical missing data"
--------------------------

In statistics, social science, etc., "missing data" is a term of art
referring to a specific (but extremely common and important)
situation: we have tried to gather some measurements according to some
scheme, but some of these measurements are missing. For example, if we
have a table listing the height, age, and income of a number of
individuals, but one person did not provide their income, then we need
some way to represent this::

  Person | Height | Age | Income
  ------------------------------
     1   |   63   | 25  | 15000
     2   |   58   | 32  | <missing>
     3   |   71   | 45  | 30000

The traditional way is to record that income as, say, "-99", and
document this in the README along with the data set. Then, you have to
remember to check for and handle such incomes specially; if you
forget, you'll get superficially reasonable but completely incorrect
results, like calculating the average income on this data set as
14967. If you're in one of these fields, then such missing-ness is
routine and inescapable, and if you use the "-99" approach then it's a
pitfall you have to remember to check for explicitly on literally
*every* calculation you ever do. This is, obviously, an unpleasant way
to live.

Let's call this situation the "statistical missing data" situation,
just to have a convenient handle for it. (As mentioned, practitioners
just call this "missing data", and what to do about it is literally an
entire sub-field of statistics; if you google "missing data" then
every reference is on how to handle it.) NumPy isn't going to do
automatic imputation or anything like that, but it could help a great
deal by providing some standard way to at least represent data which
is missing in this sense.

The main prior art for how this could be done comes from the S/S+/R
family of languages. Their strategy is, for each type they support,
to define a special value called "NA". (For ints this is INT_MAX,
for floats it's a special NaN value that's distinguishable from
other NaNs, ...) Then, they arrange that in computations, this
value has a special semantics that we will call "NA semantics".

NA semantics
------------

The idea of NA semantics is that any computations involving NA
values should be consistent with what would have happened if we
had known the correct value.

For example, let's say we want to compute the mean income, how might
we do this? One way would be to just ignore the missing entry, and
compute the mean of the remaining entries. This gives us (15000 +
30000)/2, or 22500.

Is this result consistent with discovering the income of person 2?
Let's say we find out that person 2's income is 50000. This means
the correct answer is (15000 + 50000 + 30000)/3, or 31666.67,
indicating clearly that it is not consistent. Therefore, the mean
income is NA, i.e. a specific number whose value we are unable
to compute.

This motivates the following rules, which are how R implements NA:

Assignment:
  NA values are understood to represent specific
  unknown values, and thus should have value-like semantics with
  respect to assignment and other basic data manipulation
  operations. Code which does not actually look at the values involved
  should work the same regardless of whether some of them are
  missing. For example, one might write::

    income[:] = income[np.argsort(height)]

  to perform an in-place sort of the ``income`` array, and know that
  the shortest person's income would end up being first. It turns out
  that the shortest person's income is not known, so the array should
  end up being ``[NA, 15000, 30000]``, but there's nothing
  special about NAness here.

Propagation:
  In the example above, we concluded that an operation like ``mean``
  should produce NA when one of its data values was NA.
  If you ask me, "what is 3 plus x?", then my only possible answer is
  "I don't know what x is, so I don't know what 3 + x is either". NA
  means "I don't know", so 3 + NA is NA.

  This is important for safety when analyzing data: missing data often
  requires special handling for correctness -- the fact that you are
  missing information might mean that something you wanted to compute
  cannot actually be computed, and there are whole books written on
  how to compensate in various situations. Plus, it's easy to not
  realize that you have missing data, and write code that assumes you
  have all the data. Such code should not silently produce the wrong
  answer.

  There is an important exception to characterizing this as propagation,
  in the case of boolean values. Consider the calculation::

    v = np.any([False, False, NA, True])

  If we strictly propagate, ``v`` will become NA. However, no
  matter whether we place True or False into the third array position,
  ``v`` will then get the value True. The answer to the question
  "Is the result True consistent with later discovering the value
  that was missing?" is yes, so it is reasonable to not propagate here,
  and instead return the value True. This is what R does::

    > any(c(F, F, NA, T))
    [1] TRUE
    > any(c(F, F, NA, F))
    [1] NA

Other:
  NaN and NA are conceptually distinct. 0.0/0.0 is not a mysterious,
  unknown value -- it's defined to be NaN by IEEE floating point, Not
  a Number. NAs are numbers (or strings, or whatever), just unknown
  ones. Another small but important difference is that in Python, ``if
  NaN: ...`` treats NaN as True (NaN is "truthy"); but ``if NA: ...``
  would be an error.

  In R, all reduction operations implement an alternative semantics,
  activated by passing a special argument (``na.rm=TRUE`` in R).
  ``sum(a)`` means "give me the sum of all the
  values" (which is NA if some of the values are NA);
  ``sum(a, na.rm=True)`` means "give me the sum of all the non-NA
  values".

Other prior art
---------------

Once we move beyond the "statistical missing data" case, the correct
behavior for missing data becomes less clearly defined. There are many
cases where specific elements are singled out to be treated specially
or excluded from computations, and these could often be conceptualized
as involving 'missing data' in some sense.

In image processing, it's common to use a single image together with
one or more boolean masks to e.g. composite subsets of an image. As
Joe Harrington pointed out on the list, in the context of processing
astronomical images, it's also common to generalize to a
floating-point valued mask, or alpha channel, to indicate degrees of
"missingness". We think this is out of scope for the present design,
but it is an important use case, and ideally NumPy should support
natural ways of manipulating such data.

After R, numpy.ma is probably the most mature source of
experience on missing-data-related APIs. Its design is quite different
from R; it uses different semantics -- reductions skip masked values
by default and NaNs convert to masked -- and it uses a different
storage strategy via a separate mask. While it seems to be generally
considered sub-optimal for general use, it's hard to pin down whether
this is because the API is immature but basically good, or the API
is fundamentally broken, or the API is great but the code should be
faster, or what. We looked at some of those users to try and get a
better idea.

Matplotlib is perhaps the best known package to rely on numpy.ma. It
seems to use it in two ways. One is as a way for users to indicate
what data is missing when passing it to be graphed. (Other ways are
also supported, e.g., passing in NaN values gives the same result.) In
this regard, matplotlib treats np.ma.masked and NaN values in the same way
that R's plotting routines handle NA and NaN values. For these purposes,
matplotlib doesn't really care what semantics or storage strategy is
used for missing data.

Internally, matplotlib uses numpy.ma arrays to store and pass around
separately computed boolean masks containing 'validity' information
for each input array in a cheap and non-destructive fashion. Mark's
impression from some shallow code review is that mostly it works
directly with the data and mask attributes of the masked arrays,
not extensively using the particular computational semantics of
numpy.ma. So, for this usage they do rely on the non-destructive
mask-based storage, but this doesn't say much about what semantics
are needed.

Paul Hobson `posted some code`__ on the list that uses numpy.ma for
storing arrays of contaminant concentration measurements. Here the
mask indicates whether the corresponding number represents an actual
measurement, or just the estimated detection limit for a concentration
which was too small to detect. Nathaniel's impression from reading
through this code is that it also mostly uses the .data and .mask
attributes in preference to performing operations on the MaskedArray
directly.

__ https://mail.scipy.org/pipermail/numpy-discussion/2012-April/061743.html

So, these examples make it clear that there is demand for a convenient
way to keep a data array and a mask array (or even a floating point
array) bundled up together and "aligned". But they don't tell us much
about what semantics the resulting object should have with respect to
ufuncs and friends.

Semantics, storage, API, oh my!
===============================

We think it's useful to draw a clear line between use cases,
semantics, and storage. Use cases are situations that users encounter,
regardless of what NumPy does; they're the focus of the previous
section. When we say *semantics*, we mean the result of different
operations as viewed from the Python level without regard to the
underlying implementation.

*NA semantics* are the ones described above and used by R::

  1 + NA = NA
  sum([1, 2, NA]) = NA
  NA | False = NA
  NA | True = True

With ``na.rm=TRUE`` or ``skipNA=True``, this switches to::

  1 + NA = illegal # in R, only reductions take na.rm argument
  sum([1, 2, NA], skipNA=True) = 3

There's also been discussion of what we'll call *ignore
semantics*. These are somewhat underdefined::

  sum([1, 2, IGNORED]) = 3
  # Several options here:
  1 + IGNORED = 1
  #  or
  1 + IGNORED = <leaves output array untouched>
  #  or
  1 + IGNORED = IGNORED

The numpy.ma semantics are::

  sum([1, 2, masked]) = 3
  1 + masked = masked

If either NA or ignore semantics are implemented with masks, then there
is a choice of what should be done to the value in the storage
for an array element which gets assigned a missing value. Three
possibilities are:

* Leave that memory untouched (the choice made in the NEP).
* Do the calculation with the values independently of the mask
  (perhaps the most useful option for Paul Hobson's use-case above).
* Copy whatever value is stored behind the input missing value into
  the output (this is what numpy.ma does. Even that is ambiguous in
  the case of ``masked + masked`` -- in this case numpy.ma copies the
  value stored behind the leftmost masked value).

When we talk about *storage*, we mean the debate about whether missing
values should be represented by designating a particular value of the
underlying data-type (the *bitpattern dtype* option, as used in R), or
by using a separate *mask* stored alongside the data itself.

For mask-based storage, there is also an important question about what
the API looks like for accessing the mask, modifying the mask, and
"peeking behind" the mask.

Designs that have been proposed
===============================

One option is to just copy R, by implementing a mechanism whereby
dtypes can arrange for certain bitpatterns to be given NA semantics.

One option is to copy numpy.ma closely, but with a more optimized
implementation. (Or to simply optimize the existing implementation.)

One option is that described in `NEP12`, for which an implementation
of mask-based missing data exists. This system is roughly:

* There is both bitpattern and mask-based missing data, and both
  have identical interoperable NA semantics.
* Masks are modified by assigning np.NA or values to array elements.
  The way to peek behind the mask or to unmask values is to keep a
  view of the array that shares the data pointer but not the mask pointer.
* Mark would like to add a way to access and manipulate the mask more
  directly, to be used in addition to this view-based API.
* If an array has both a bitpattern dtype and a mask, then assigning
  np.NA writes to the mask, rather than to the array itself. Writing
  a bitpattern NA to an array which supports both requires accessing
  the data by "peeking under the mask".

Another option is that described in `NEP24`, which is to implement
bitpattern dtypes with NA semantics for the "statistical missing data"
use case, and to also implement a totally independent API for masked
arrays with ignore semantics and all mask manipulation done explicitly
through a .mask attribute.

Another option would be to define a minimalist aligned array container
that holds multiple arrays and that can be used to pass them around
together. It would support indexing (to help with the common problem
of wanting to subset several arrays together without their becoming
unaligned), but all arithmetic etc. would be done by accessing the
underlying arrays directly via attributes. The "prior art" discussion
above suggests that something like this holding a .data and a .mask
array might actually be solve a number of people's problems without
requiring any major architectural changes to NumPy. This is similar to
a structured array, but with each field in a separately stored array
instead of packed together.

Several people have suggested that there should be a single system
that has multiple missing values that each have different semantics,
e.g., a MISSING value that has NA semantics, and a separate IGNORED
value that has ignored semantics.

None of these options are necessarily exclusive.

The debate
==========

We both are dubious of using ignored semantics as a default missing
data behavior. **Nathaniel** likes NA semantics because he is most
interested in the "statistical missing data" use case, and NA semantics
are exactly right for that. **Mark** isn't as interested in that use
case in particular, but he likes the NA computational abstraction
because it is unambiguous and well-defined in all cases, and has a
lot of existing experience to draw from.

What **Nathaniel** thinks, overall:

* The "statistical missing data" use case is clear and compelling; the
  other use cases certainly deserve our attention, but it's hard to say what
  they *are* exactly yet, or even if the best way to support them is
  by extending the ndarray object.
* The "statistical missing data" use case is best served by an R-style
  system that uses bitpattern storage to implement NA semantics. The
  main advantage of bitpattern storage for this use case is that it
  avoids the extra memory and speed overhead of storing and checking a
  mask (especially for the common case of floating point data, where
  some tricks with NaNs allow us to effectively hardware-accelerate
  most NA operations). These concerns alone appears to make a
  mask-based implementation unacceptable to many NA users,
  particularly in areas like neuroscience (where memory is tight) or
  financial modeling (where milliseconds are critical). In addition,
  the bit-pattern approach is less confusing conceptually (e.g.,
  assignment really is just assignment, no magic going on behind the
  curtain), and it's possible to have in-memory compatibility with R
  for inter-language calls via rpy2.  The main disadvantage of the
  bitpattern approach is the need to give up a value to represent NA,
  but this is not an issue for the most important data types (float,
  bool, strings, enums, objects); really, only integers are
  affected. And even for integers, giving up a value doesn't really
  matter for statistical problems. (Occupy Wall Street
  notwithstanding, no-one's income is 2**63 - 1. And if it were, we'd
  be switching to floats anyway to avoid overflow.)
* Adding new dtypes requires some cooperation with the ufunc and
  casting machinery, but doesn't require any architectural changes or
  violations of NumPy's current orthogonality.
* His impression from the mailing list discussion, esp. the `"what can
  we agree on?" thread`__, is that many numpy.ma users specifically
  like the combination of masked storage, the mask being easily
  accessible through the API, and ignored semantics. He could be
  wrong, of course. But he cannot remember seeing anybody besides Mark
  advocate for the specific combination of masked storage and NA
  semantics, which makes him nervous.

  __ http://thread.gmane.org/gmane.comp.python.numeric.general/46704
* Also, he personally is not very happy with the idea of having two
  storage implementations that are almost-but-not-quite identical at
  the Python level. While there likely are people who would like to
  temporarily pretend that certain data is "statistically missing
  data" without making a copy of their array, it's not at all clear
  that they outnumber the people who would like to use bitpatterns and
  masks simultaneously for distinct purposes. And honestly he'd like
  to be able to just ignore masks if he wants and stick to
  bitpatterns, which isn't possible if they're coupled together
  tightly in the API.  So he would say the jury is still very much out
  on whether this aspect of the NEP design is an advantage or a
  disadvantage. (Certainly he's never heard of any R users complaining
  that they really wish they had an option of making a different
  trade-off here.)
* R's NA support is a `headline feature`__ and its target audience
  consider it a compelling advantage over other platforms like Matlab
  or Python. Working with statistical missing data is very painful
  without platform support.

  __ http://www.sr.bham.ac.uk/~ajrs/R/why_R.html
* By comparison, we clearly have much more uncertainty about the use
  cases that require a mask-based implementation, and it doesn't seem
  like people will suffer too badly if they are forced for now to
  settle for using NumPy's excellent mask-based indexing, the new
  where= support, and even numpy.ma.
* Therefore, bitpatterns with NA semantics seem to meet the criteria
  of making a large class of users happy, in an elegant way, that fits
  into the original design, and where we can have reasonable certainty
  that we understand the problem and use cases well enough that we'll
  be happy with them in the long run. But no mask-based storage
  proposal does, yet.

What **Mark** thinks, overall:

* The idea of using NA semantics by default for missing data, inspired
  by the "statistical missing data" problem, is better than all the
  other default behaviors which were considered. This applies equally
  to the bitpattern and the masked approach.

* For NA-style functionality to get proper support by all NumPy
  features and eventually all third-party libraries, it needs to be
  in the core. How to correctly and efficiently handle missing data
  differs by algorithm, and if thinking about it is required to fully
  support NumPy, NA support will be broader and higher quality.

* At the same time, providing two different missing data interfaces,
  one for masks and one for bitpatterns, requires NumPy developers
  and third-party NumPy plugin developers to separately consider the
  question of what to do in either case, and do two additional
  implementations of their code. This complicates their job,
  and could lead to inconsistent support for missing data.

* Providing the ability to work with both masks and bitpatterns through
  the same C and Python programming interface makes missing data support
  cleanly orthogonal with all other NumPy features.

* There are many trade-offs of memory usage, performance, correctness, and
  flexibility between masks and bitpatterns. Providing support for both
  approaches allows users of NumPy to choose the approach which is
  most compatible with their way of thinking, or has characteristics
  which best match their use-case. Providing them through the same
  interface further allows them to try both with minimal effort, and
  choose the one which performs better or uses the least memory for
  their programs.

* Memory Usage

  * With bitpatterns, less memory is used for storing a single array
    containing some NAs.

  * With masks, less memory is used for storing multiple arrays that
    are identical except for the location of their NAs. (In this case a
    single data array can be re-used with multiple mask arrays;
    bitpattern NAs would need to copy the whole data array.)

* Performance

  * With bitpatterns, the floating point type can use native hardware
    operations, with nearly correct behavior. For fully correct floating
    point behavior and with other types, code must be written which
    specially tests for equality with the missing-data bitpattern.

  * With masks, there is always the overhead of accessing mask memory
    and testing its truth value. The implementation that currently exists
    has no performance tuning, so it is only good to judge a minimum
    performance level. Optimal mask-based code is in general going to
    be slower than optimal bitpattern-based code.

* Correctness

  * Bitpattern integer types must sacrifice a valid value to represent NA.
    For larger integer types, there are arguments that this is ok, but for
    8-bit types there is no reasonable choice. In the floating point case,
    if the performance of native floating point operations is chosen,
    there is a small inconsistency that NaN+NA and NA+NaN are different.
  * With masks, it works correctly in all cases.

* Generality

  * The bitpattern approach can work in a fully general way only when
    there is a specific value which can be given up from the
    data type. For IEEE floating point, a NaN is an obvious choice,
    and for booleans represented as a byte, there are plenty of choices.
    For integers, a valid value must be sacrificed to use this approach.
    Third-party dtypes which plug into NumPy will also have to
    make a bitpattern choice to support this system, something which
    may not always be possible.

  * The mask approach works universally with all data types.

Recommendations for moving forward
==================================

**Nathaniel** thinks we should:

* Go ahead and implement bitpattern NAs.
* *Don't* implement masked arrays in the core -- or at least, not
  yet. Instead, we should focus on figuring out how to implement them
  out-of-core, so that people can try out different approaches without
  us committing to any one approach. And so new prototypes can be
  released more quickly than the NumPy release cycle. And anyway,
  we're going to have to figure out how to experiment with such
  changes out-of-core if NumPy is to continue to evolve without
  forking -- might as well do it now. The existing code can live in
  the main branch, be disabled, or live its own branch -- it'll still be there
  once we know what we're doing.

**Mark** thinks we should:

* The existing code should remain as is, with a global run-time experimental
  flag added which disables NA support by default.

A more detailed rationale for this recommendation is:

* A solid preliminary NA-mask implementation is currently in NumPy
  main. This implementation has been extensively tested
  against scipy and other third-party packages, and has been in main
  in a stable state for a significant amount of time.
* This implementation integrates deeply with the core, providing an
  interface which is usable in the same way R's NA support is. It
  provides a compelling, user-friendly answer to R's NA support.
* The missing data NEP provides a plan for adding bitpattern-based
  dtype support of NAs, which will operate through the same interface
  but allow for the same performance/correctness tradeoffs that R has made.
* Making it very easy for users to try out this implementation, which
  has reasonable feature coverage and performance characteristics, is
  the best way to get more concrete feedback about how NumPy's missing
  data support should look.

Because of its preliminary state, the existing implementation is marked
as experimental in the NumPy documentation. It would be good for this
to remain marked as experimental until it is more fleshed out, for
example supporting struct and array dtypes and with a fuller set of
NumPy operations.

I think the code should stay as it is, except to add a run-time global
NumPy flag, perhaps numpy.experimental.maskna, which defaults to
False and can be toggled to True. In its default state, any NA feature
usage would raise an "ExperimentalError" exception, a measure which
would prevent it from being accidentally used and communicate its
experimental status very clearly.

The `ABI issues`__ seem very tricky to deal with effectively in the 1.x
series of releases, but I believe that with proper implementation-hiding
in a 2.0 release, evolving the software to support various other
ABI ideas that have been discussed is feasible. This is the approach
I like best.

__ http://thread.gmane.org/gmane.comp.python.numeric.general/49485>

**Nathaniel** notes in response that he doesn't really have any
objection to shipping experimental APIs in the main numpy distribution
*if* we're careful to make sure that they don't "leak out" in a way
that leaves us stuck with them. And in principle some sort of "this
violates your warranty" global flag could be a way to do that. (In
fact, this might also be a useful strategy for the kinds of changes
that he favors, of adding minimal hooks to enable us to build
prototypes more easily -- we could have some "rapid prototyping only"
hooks that let prototype hacks get deeper access to NumPy's internals
than we were otherwise ready to support.)

But, he wants to point out two things. First, it seems like we still
have fundamental questions to answer about the NEP design, like
whether masks should have NA semantics or ignore semantics, and there
are already plans to majorly change how NEP masks are exposed and
accessed. So he isn't sure what we'll learn by asking for feedback on
the NEP code in its current state.

And second, given the concerns about their causing (minor) ABI issues,
it's not clear that we could really prevent them from leaking out. (He
looks forward to 2.0 too, but we're not there yet.) So maybe it would
be better if they weren't present in the C API at all, and the hoops
required for testers were instead something like, 'we have included a
hacky pure-Python prototype accessible by typing "import
numpy.experimental.donttrythisathome.NEP" and would welcome feedback'?

If so, then he should mention that he did implement a horribly klugy,
pure Python implementation of the NEP API that works with NumPy
1.6.1. This was mostly as an experiment to see how possible such
prototyping was and to test out a possible ufunc override mechanism,
but if there's interest, the module is available here:
https://github.com/njsmith/numpyNEP

It passes the maskna test-suite, with some minor issues described
in a big comment at the top.

**Mark** responds:

I agree that it's important to be careful when adding new
features to NumPy, but I also believe it is essential that the project
have forward development momentum. A project like NumPy requires
developers to write code for advancement to occur, and obstacles
that impede the writing of code discourage existing developers
from contributing more, and potentially scare away developers
who are thinking about joining in.

All software projects, both open source and closed source, must
balance between short-term practicality and long-term planning.
In the case of the missing data development, there was a short-term
resource commitment to tackle this problem, which is quite immense
in scope. If there isn't a high likelihood of getting a contribution
into NumPy that concretely advances towards a solution, I expect
that individuals and companies interested in doing such work will
have a much harder time justifying a commitment of their resources.
For a project which is core to so many other libraries, only
relying on the good will of selfless volunteers would mean that
NumPy could more easily be overtaken by another project.

In the case of the existing NA contribution at issue, how we resolve
this disagreement represents a decision about how NumPy's
developers, contributors, and users should interact. If we create
a document describing a dispute resolution process, how do we
design it so that it doesn't introduce a large burden and excessive
uncertainty on developers that could prevent them from productively
contributing code?

If we go this route of writing up a decision process which includes
such a dispute resolution mechanism, I think the meat of it should
be a roadmap that potential contributors and developers can follow
to gain influence over NumPy. NumPy development needs broad support
beyond code contributions, and tying influence in the project to
contributions seems to me like it would be a good way to encourage
people to take on tasks like bug triaging/management, continuous
integration/build server administration, and the myriad other
tasks that help satisfy the project's needs. No specific meritocratic,
democratic, consensus-striving system will satisfy everyone, but the
vigour of the discussions around governance and process indicate that
something at least a little bit more formal than the current status
quo is necessary.

In conclusion, I would like the NumPy project to prioritize movement
towards a more flexible and modular ABI/API, balanced with strong
backwards-compatibility constraints and feature additions that
individuals, universities, and companies want to contribute.
I do not believe keeping the NA code in 1.7 as it is, with the small
additional measure of requiring it to be enabled by an experimental
flag, poses a risk of long-term ABI troubles. The greater risk I see
is a continuing lack of developers contributing to the project,
and I believe backing out this code because these worries would create a
risk of reducing developer contribution.


References and footnotes
------------------------

:ref:`NEP12` describes Mark's NA-semantics/mask implementation/view based mask
handling API.

:ref:`NEP24` ("the alterNEP") was Nathaniel's initial attempt at separating MISSING
and IGNORED handling into bit-patterns versus masks, though there's a bunch
he would change about the proposal at this point.

:ref:`NEP25` ("miniNEP 2") was a later attempt by Nathaniel to sketch out an
implementation strategy for NA dtypes.

A further discussion overview page can be found at:
https://github.com/njsmith/numpy/wiki/NA-discussion-status


Copyright
---------

This document has been placed in the public domain.