1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
<style>
.md-content .md-typeset h1 {
display: none;
}
#upath-logo {
margin-top: -2em;
}
</style>
{: #upath-logo }
[](https://pypi.org/project/universal_pathlib/)
[](https://github.com/fsspec/universal_pathlib/blob/main/LICENSE)
[](https://pypi.org/project/universal_pathlib/)
[](https://anaconda.org/conda-forge/universal_pathlib)
[](https://universal-pathlib.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/fsspec/universal_pathlib/actions/workflows/tests.yml)
[](https://github.com/fsspec/universal_pathlib/issues)
[](https://github.com/psf/black)
[](./changelog.md)
---
**Universal Pathlib** is a Python library that extends the [`pathlib_abc.JoinablePath`][pathlib_abc], [`pathlib_abc.Readable`][pathlib_abc], and [`pathlib_abc.Writable`][pathlib_abc] API to give you a unified, Pythonic interface for working with files, whether they're on your local machine, in S3, on GitHub, or anywhere else. Built on top of [`filesystem_spec`][fsspec], it brings the convenienve of a [`pathlib.Path`][pathlib]-like interface to cloud storage, remote filesystems, and more! :sparkles:
[pathlib_abc]: https://github.com/barneygale/pathlib-abc
[pathlib]: https://docs.python.org/3/library/pathlib.html
[fsspec]: https://filesystem-spec.readthedocs.io/en/latest/intro.html
---
If you enjoy working with Python's [pathlib][pathlib] objects to operate on local file system paths,
universal pathlib provides the same interface for many supported [ filesystem_spec ][fsspec]
implementations, from cloud-native object storage like `Amazon's S3 Storage`, `Google Cloud Storage`,
`Azure Blob Storage`, to `http`, `sftp`, `memory` stores, and many more...
If you're familiar with [ filesystem_spec ][fsspec], then universal pathlib provides a convenient
way to handle the path, protocol and storage options of a object stored on a fsspec filesystem in a
single container (`upath.UPath`). And it further provides a pathlib interface to do path operations on the
fsspec urlpath.
The great part is, if you're familiar with the [pathlib.Path][pathlib] API, you can immediately
switch from working with local paths to working on remote and virtual filesystem by simply using
the `UPath` class:
=== "The Problem"
```python
# Local files: use pathlib
from pathlib import Path
local_file = Path("data/file.txt")
content = local_file.read_text()
# S3 files: use boto3/s3fs
import boto3
s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='data/file.txt')
content = obj['Body'].read().decode('utf-8')
# Different APIs, different patterns 😫
```
=== "The Solution"
```python
# All files: use UPath! ✨
from upath import UPath
local_file = UPath("data/file.txt")
s3_file = UPath("s3://bucket/data/file.txt")
# Same API everywhere! 🎉
content = local_file.read_text()
content = s3_file.read_text()
```
[Learn more about why you should use Universal Pathlib →](why.md){ .md-button }
---
## Quick Start :rocket:
### Installation
```bash
pip install universal-pathlib
```
!!! tip "Installing for specific filesystems"
To use cloud storage or other remote filesystems, install the necessary fsspec extras:
```bash
pip install "universal-pathlib" "fsspec[s3,gcs,azure]"
```
See the [Installation Guide](install.md) for more details.
### TL;DR Examples
```python
from upath import UPath
# Works with local paths
local_path = UPath("documents/notes.txt")
local_path.write_text("Hello, World!")
print(local_path.read_text()) # "Hello, World!"
# Works with S3
s3_path = UPath("s3://my-bucket/data/processed/results.csv")
if s3_path.exists():
data = s3_path.read_text()
# Works with HTTP
http_path = UPath("https://example.com/data/file.json")
if http_path.exists():
content = http_path.read_bytes()
# Works with many more! 🌟
```
---
## Currently supported filesystems
- :fontawesome-solid-folder: `file:` and `local:` Local filesystem
- :fontawesome-solid-memory: `memory:` Ephemeral filesystem in RAM
- :fontawesome-brands-microsoft: `az:`, `adl:`, `abfs:` and `abfss:` Azure Storage _(requires `adlfs`)_
- :fontawesome-solid-database: `data:` RFC 2397 style data URLs _(requires `fsspec>=2023.12.2`)_
- :fontawesome-solid-network-wired: `ftp:` FTP filesystem
- :fontawesome-brands-github: `github:` GitHub repository filesystem
- :fontawesome-solid-globe: `http:` and `https:` HTTP(S)-based filesystem
- :fontawesome-solid-server: `hdfs:` Hadoop distributed filesystem
- :fontawesome-brands-google: `gs:` and `gcs:` Google Cloud Storage _(requires `gcsfs`)_
- :simple-huggingface: `hf:` Hugging Face Hub _(requires `huggingface_hub`)_
- :fontawesome-brands-aws: `s3:` and `s3a:` AWS S3 _(requires `s3fs`)_
- :fontawesome-solid-network-wired: `sftp:` and `ssh:` SFTP and SSH filesystems _(requires `paramiko`)_
- :fontawesome-solid-share-nodes: `smb:` SMB filesystems _(requires `smbprotocol`)_
- :fontawesome-solid-cloud: `webdav:`, `webdav+http:` and `webdav+https:` WebDAV _(requires `webdav4[fsspec]`)_
!!! info "Untested Filesystems"
Other fsspec-compatible filesystems likely work through the default implementation. If you encounter issues, please [report it our issue tracker](https://github.com/fsspec/universal_pathlib/issues)! We're happy to add official support!
---
## Getting Help :question:
Need help? We're here for you!
- :fontawesome-brands-github: [GitHub Issues](https://github.com/fsspec/universal_pathlib/issues) - Report bugs or request features
- :material-book-open-variant: [Documentation](https://universal-pathlib.readthedocs.io/) - You're reading it!
!!! tip "Before Opening an Issue"
Please check if your question has already been answered in the documentation or existing issues.
---
## License :page_with_curl:
Universal Pathlib is distributed under the [MIT license](https://github.com/fsspec/universal_pathlib/blob/main/LICENSE), making it free and open source software. Use it freely in your projects!
---
<div align="center" markdown>
**Ready to get started?**
[Install Now](install.md){ .md-button .md-button--primary }
</div>
|
