1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
|
.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied. See the License for the
.. specific language governing permissions and limitations
.. under the License.
Installing PyArrow
==================
System Compatibility
--------------------
PyArrow is regularly built and tested on Windows, macOS and various
Linux distributions. We strongly recommend using a 64-bit system.
Python Compatibility
--------------------
PyArrow is currently compatible with Python 3.10, 3.11, 3.12, 3.13 and 3.14.
Using Conda
-----------
Install the latest version of PyArrow from
`conda-forge <https://conda-forge.org/>`_ using `Conda <https://conda.io>`_:
.. code-block:: bash
conda install -c conda-forge pyarrow
.. note::
While the ``pyarrow`` `conda-forge <https://conda-forge.org/>`_ package is
the right choice for most users, both a minimal and maximal variant of the
package exist, either of which may be better for your use case. See
:ref:`python-conda-differences`.
Using Pip
---------
Install the latest version from `PyPI <https://pypi.org/>`_ (Windows, Linux,
and macOS):
.. code-block:: bash
pip install pyarrow
If you encounter any importing issues of the pip wheels on Windows, you may
need to install the `latest Visual C++ Redistributable for Visual Studio
<https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170#latest-microsoft-visual-c-redistributable-version>`_.
.. warning::
On Linux, you will need pip >= 19.0 to detect the prebuilt binary packages.
Installing nightly packages or from source
------------------------------------------
See :ref:`python-development`.
Dependencies
------------
Optional dependencies
* **NumPy 1.21.2** or higher.
* **pandas 1.3.4** or higher,
* **cffi**.
Additional packages PyArrow is compatible with are :ref:`fsspec <filesystem-fsspec>`
and **pytz**, **dateutil** or **tzdata** package for timezones.
tzdata on Windows
^^^^^^^^^^^^^^^^^
While Arrow uses the OS-provided timezone database on Linux and macOS, it requires a
user-provided database on Windows. To download and extract the text version of
the IANA timezone database follow the instructions in the C++
:ref:`download-timezone-database` or use pyarrow utility function
``pyarrow.util.download_tzdata_on_windows()`` that does the same.
By default, the timezone database will be detected at ``%USERPROFILE%\Downloads\tzdata``.
If the database has been downloaded in a different location, you will need to set
a custom path to the database from Python:
.. code-block:: python
>>> import pyarrow as pa
>>> pa.set_timezone_db_path("custom_path")
You may encounter problems writing datetime data to an ORC file if you install
pyarrow with pip. One possible solution to fix this problem:
1. Install tzdata with ``pip install tzdata``
2. Set the environment variable ``TZDIR = path\to\.venv\Lib\site-packages\tzdata\``
You can find where ``tzdata`` is installed with the following python
command:
.. code-block:: python
>>> import tzdata
>>> print(tzdata.__file__)
path\to\.venv\Lib\site-packages\tzdata\__init__.py
.. _python-conda-differences:
Differences between conda-forge packages
----------------------------------------
On `conda-forge <https://conda-forge.org/>`_, PyArrow is published as three
separate packages, each providing varying levels of functionality. This is in
contrast to PyPi, where only a single PyArrow package is provided.
The purpose of this split is to minimize the size of the installed package for
most users (``pyarrow``), provide a smaller, minimal package for specialized use
cases (``pyarrow-core``), while still providing a complete package for users who
require it (``pyarrow-all``). What was historically ``pyarrow`` on
`conda-forge <https://conda-forge.org/>`_ is now ``pyarrow-all``, though most
users can continue using ``pyarrow``.
The ``pyarrow-core`` package includes the following functionality:
- :ref:`data`
- :ref:`compute` (i.e., ``pyarrow.compute``)
- :ref:`io`
- :ref:`ipc` (i.e., ``pyarrow.ipc``)
- :ref:`filesystem` (i.e., ``pyarrow.fs``. Note: It's planned to move cloud fileystems (i.e., :ref:`S3<filesystem-s3>`, :ref:`GCS<filesystem-gcs>`, etc) into ``pyarrow`` in a future release though :ref:`filesystem-localfs` will remain in ``pyarrow-core``.)
- File formats: :ref:`Arrow/Feather<feather>`, :ref:`JSON<json>`, :ref:`CSV<py-csv>`, :ref:`ORC<orc>` (but not Parquet)
The ``pyarrow`` package adds the following:
- Acero (i.e., ``pyarrow.acero``)
- :ref:`dataset` (i.e., ``pyarrow.dataset``)
- :ref:`Parquet<parquet>` (i.e., ``pyarrow.parquet``)
- Substrait (i.e., ``pyarrow.substrait``)
Finally, ``pyarrow-all`` adds:
- :ref:`flight` and Flight SQL (i.e., ``pyarrow.flight``)
- Gandiva (i.e., ``pyarrow.gandiva``)
The following table lists the functionality provided by each package and may be
useful when deciding to use one package over another or when
:ref:`python-conda-custom-selection`.
+------------+---------------------+--------------+---------+-------------+
| Component | Package | pyarrow-core | pyarrow | pyarrow-all |
+------------+---------------------+--------------+---------+-------------+
| Core | pyarrow-core | ✓ | ✓ | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Parquet | libparquet | | ✓ | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Dataset | libarrow-dataset | | ✓ | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Acero | libarrow-acero | | ✓ | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Substrait | libarrow-substrait | | ✓ | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Flight | libarrow-flight | | | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Flight SQL | libarrow-flight-sql | | | ✓ |
+------------+---------------------+--------------+---------+-------------+
| Gandiva | libarrow-gandiva | | | ✓ |
+------------+---------------------+--------------+---------+-------------+
.. _python-conda-custom-selection:
Creating A Custom Selection
^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you know which components you need and want to control what's installed, you
can create a custom selection of packages to include only the extra features you
need. For example, to install ``pyarrow-core`` and add support for reading and
writing Parquet, install ``libparquet`` alongside ``pyarrow-core``:
.. code-block:: shell
conda install -c conda-forge pyarrow-core libparquet
Or if you wish to use ``pyarrow`` but need support for Flight RPC:
.. code-block:: shell
conda install -c conda-forge pyarrow libarrow-flight
|
