1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
|
.. _roadmap:
=======
Roadmap
=======
This page provides an overview of the major themes in pandas' development. Each of
these items requires a relatively large amount of effort to implement. These may
be achieved more quickly with dedicated funding or interest from contributors.
An item being on the roadmap does not mean that it will *necessarily* happen, even
with unlimited funding. During the implementation period we may discover issues
preventing the adoption of the feature.
Additionally, an item *not* being on the roadmap does not exclude it from inclusion
in pandas. The roadmap is intended for larger, fundamental changes to the project that
are likely to take months or years of developer time. Smaller-scoped items will continue
to be tracked on our `issue tracker <https://github.com/pandas-dev/pandas/issues>`__.
See :ref:`roadmap.evolution` for proposing changes to this document.
Extensibility
-------------
pandas :ref:`extending.extension-types` allow for extending NumPy types with custom
data types and array storage. pandas uses extension types internally, and provides
an interface for 3rd-party libraries to define their own custom data types.
Many parts of pandas still unintentionally convert data to a NumPy array.
These problems are especially pronounced for nested data.
We'd like to improve the handling of extension arrays throughout the library,
making their behavior more consistent with the handling of NumPy arrays. We'll do this
by cleaning up pandas' internals and adding new methods to the extension array interface.
String data type
----------------
Currently, pandas stores text data in an ``object`` -dtype NumPy array.
The current implementation has two primary drawbacks: First, ``object`` -dtype
is not specific to strings: any Python object can be stored in an ``object`` -dtype
array, not just strings. Second: this is not efficient. The NumPy memory model
isn't especially well-suited to variable width text data.
To solve the first issue, we propose a new extension type for string data. This
will initially be opt-in, with users explicitly requesting ``dtype="string"``.
The array backing this string dtype may initially be the current implementation:
an ``object`` -dtype NumPy array of Python strings.
To solve the second issue (performance), we'll explore alternative in-memory
array libraries (for example, Apache Arrow). As part of the work, we may
need to implement certain operations expected by pandas users (for example
the algorithm used in, ``Series.str.upper``). That work may be done outside of
pandas.
Apache Arrow interoperability
-----------------------------
`Apache Arrow <https://arrow.apache.org>`__ is a cross-language development
platform for in-memory data. The Arrow logical types are closely aligned with
typical pandas use cases.
We'd like to provide better-integrated support for Arrow memory and data types
within pandas. This will let us take advantage of its I/O capabilities and
provide for better interoperability with other languages and libraries
using Arrow.
Block manager rewrite
---------------------
We'd like to replace pandas current internal data structures (a collection of
1 or 2-D arrays) with a simpler collection of 1-D arrays.
pandas internal data model is quite complex. A DataFrame is made up of
one or more 2-dimensional "blocks", with one or more blocks per dtype. This
collection of 2-D arrays is managed by the BlockManager.
The primary benefit of the BlockManager is improved performance on certain
operations (construction from a 2D array, binary operations, reductions across the columns),
especially for wide DataFrames. However, the BlockManager substantially increases the
complexity and maintenance burden of pandas.
By replacing the BlockManager we hope to achieve
* Substantially simpler code
* Easier extensibility with new logical types
* Better user control over memory use and layout
* Improved micro-performance
* Option to provide a C / Cython API to pandas' internals
See `these design documents <https://dev.pandas.io/pandas2/internal-architecture.html#removal-of-blockmanager-new-dataframe-internals>`__
for more.
Decoupling of indexing and internals
------------------------------------
The code for getting and setting values in pandas' data structures needs refactoring.
In particular, we must clearly separate code that converts keys (e.g., the argument
to ``DataFrame.loc``) to positions from code that uses these positions to get
or set values. This is related to the proposed BlockManager rewrite. Currently, the
BlockManager sometimes uses label-based, rather than position-based, indexing.
We propose that it should only work with positional indexing, and the translation of keys
to positions should be entirely done at a higher level.
Indexing is a complicated API with many subtleties. This refactor will require care
and attention. More details are discussed at
https://github.com/pandas-dev/pandas/wiki/(Tentative)-rules-for-restructuring-indexing-code
Numba-accelerated operations
----------------------------
`Numba <https://numba.pydata.org>`__ is a JIT compiler for Python code. We'd like to provide
ways for users to apply their own Numba-jitted functions where pandas accepts user-defined functions
(for example, :meth:`Series.apply`, :meth:`DataFrame.apply`, :meth:`DataFrame.applymap`,
and in groupby and window contexts). This will improve the performance of
user-defined-functions in these operations by staying within compiled code.
Documentation improvements
--------------------------
We'd like to improve the content, structure, and presentation of the pandas documentation.
Some specific goals include
* Overhaul the HTML theme with a modern, responsive design (:issue:`15556`)
* Improve the "Getting Started" documentation, designing and writing learning paths
for users different backgrounds (e.g. brand new to programming, familiar with
other languages like R, already familiar with Python).
* Improve the overall organization of the documentation and specific subsections
of the documentation to make navigation and finding content easier.
Performance monitoring
----------------------
pandas uses `airspeed velocity <https://asv.readthedocs.io/en/stable/>`__ to
monitor for performance regressions. ASV itself is a fabulous tool, but requires
some additional work to be integrated into an open source project's workflow.
The `asv-runner <https://github.com/asv-runner>`__ organization, currently made up
of pandas maintainers, provides tools built on top of ASV. We have a physical
machine for running a number of project's benchmarks, and tools managing the
benchmark runs and reporting on results.
We'd like to fund improvements and maintenance of these tools to
* Be more stable. Currently, they're maintained on the nights and weekends when
a maintainer has free time.
* Tune the system for benchmarks to improve stability, following
https://pyperf.readthedocs.io/en/latest/system.html
* Build a GitHub bot to request ASV runs *before* a PR is merged. Currently, the
benchmarks are only run nightly.
.. _roadmap.evolution:
Roadmap evolution
-----------------
pandas continues to evolve. The direction is primarily determined by community
interest. Everyone is welcome to review existing items on the roadmap and
to propose a new item.
Each item on the roadmap should be a short summary of a larger design proposal.
The proposal should include
1. Short summary of the changes, which would be appropriate for inclusion in
the roadmap if accepted.
2. Motivation for the changes.
3. An explanation of why the change is in scope for pandas.
4. Detailed design: Preferably with example-usage (even if not implemented yet)
and API documentation
5. API Change: Any API changes that may result from the proposal.
That proposal may then be submitted as a GitHub issue, where the pandas maintainers
can review and comment on the design. The `pandas mailing list <https://mail.python.org/mailman/listinfo/pandas-dev>`__
should be notified of the proposal.
When there's agreement that an implementation
would be welcome, the roadmap should be updated to include the summary and a
link to the discussion issue.
|
