API Reference
=============

.. currentmodule:: fsspec

User Functions
--------------

.. autosummary::
   available_compressions
   available_protocols
   filesystem
   fuse.run
   generic.rsync
   get_filesystem_class
   get_mapper
   gui.FileSelector
   open
   open_files
   open_local

.. autofunction:: fsspec.available_compressions
.. autofunction:: fsspec.available_protocols
.. autofunction:: fsspec.filesystem
.. autofunction:: fsspec.fuse.run
.. autofunction:: fsspec.generic.rsync
.. autofunction:: fsspec.get_filesystem_class
.. autofunction:: fsspec.get_mapper
.. autoclass:: fsspec.gui.FileSelector
   :members:
.. autofunction:: fsspec.open
.. autofunction:: fsspec.open_files
.. autofunction:: fsspec.open_local

Base Classes
------------

.. autosummary::
   archive.AbstractArchiveFileSystem
   asyn.AsyncFileSystem
   callbacks.Callback
   callbacks.DotPrinterCallback
   callbacks.NoOpCallback
   callbacks.TqdmCallback
   core.BaseCache
   core.OpenFile
   core.OpenFiles
   core.get_fs_token_paths
   core.url_to_fs
   dircache.DirCache
   FSMap
   generic.GenericFileSystem
   registry.register_implementation
   spec.AbstractBufferedFile
   spec.AbstractFileSystem
   spec.Transaction

.. autoclass:: fsspec.archive.AbstractArchiveFileSystem
   :members:

.. autoclass:: fsspec.callbacks.Callback
   :members:

.. autoclass:: fsspec.callbacks.DotPrinterCallback
   :members:

.. autoclass:: fsspec.callbacks.NoOpCallback
   :members:

.. autoclass:: fsspec.callbacks.TqdmCallback
   :members:

.. autoclass:: fsspec.core.BaseCache
   :members:

.. autoclass:: fsspec.core.OpenFile
   :members:

.. autoclass:: fsspec.core.OpenFiles

.. autofunction:: fsspec.core.get_fs_token_paths

.. autofunction:: fsspec.core.url_to_fs

.. autoclass:: fsspec.dircache.DirCache
   :members: __init__

.. autoclass:: fsspec.FSMap
   :members:

.. autoclass:: fsspec.generic.GenericFileSystem

.. autofunction:: fsspec.registry.register_implementation

.. autoclass:: fsspec.spec.AbstractBufferedFile
   :members:

.. autoclass:: fsspec.spec.AbstractFileSystem
   :members:

.. autoclass:: fsspec.spec.Transaction
   :members:

.. _implementations:

Built-in Implementations
------------------------

.. autosummary::
   implementations.arrow.ArrowFSWrapper
   implementations.arrow.HadoopFileSystem
   implementations.cached.CachingFileSystem
   implementations.cached.SimpleCacheFileSystem
   implementations.cached.WholeFileCacheFileSystem
   implementations.dask.DaskWorkerFileSystem
   implementations.data.DataFileSystem
   implementations.dbfs.DatabricksFileSystem
   implementations.dirfs.DirFileSystem
   implementations.ftp.FTPFileSystem
   implementations.gist.GistFileSystem
   implementations.git.GitFileSystem
   implementations.github.GithubFileSystem
   implementations.http.HTTPFileSystem
   implementations.jupyter.JupyterFileSystem
   implementations.libarchive.LibArchiveFileSystem
   implementations.local.LocalFileSystem
   implementations.memory.MemoryFileSystem
   implementations.reference.ReferenceFileSystem
   implementations.reference.LazyReferenceMapper
   implementations.sftp.SFTPFileSystem
   implementations.smb.SMBFileSystem
   implementations.tar.TarFileSystem
   implementations.webhdfs.WebHDFS
   implementations.zip.ZipFileSystem

.. autoclass:: fsspec.implementations.arrow.ArrowFSWrapper
   :members: __init__

.. autoclass:: fsspec.implementations.arrow.HadoopFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.CachingFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.SimpleCacheFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.cached.WholeFileCacheFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dask.DaskWorkerFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.data.DataFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dbfs.DatabricksFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.dirfs.DirFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.ftp.FTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.gist.GistFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.git.GitFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.github.GithubFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.http.HTTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.jupyter.JupyterFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.libarchive.LibArchiveFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.local.LocalFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.memory.MemoryFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.reference.ReferenceFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.reference.LazyReferenceMapper
   :members: __init__

.. autoclass:: fsspec.implementations.sftp.SFTPFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.smb.SMBFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.tar.TarFileSystem
   :members: __init__

.. autoclass:: fsspec.implementations.webhdfs.WebHDFS
   :members: __init__

.. autoclass:: fsspec.implementations.zip.ZipFileSystem
   :members: __init__

.. _external_implementations:

Other Known Implementations
---------------------------


Note that most of these projects are hosted outside of the `fsspec` organisation. Please read their
documentation carefully before using any particular package.

- `abfs`_ for Azure Blob service, with protocol "abfs://"
- `adl`_ for Azure DataLake storage, with protocol "adl://"
- `alluxiofs`_ to access fsspec implemented filesystem with Alluxio distributed cache
- `boxfs`_ for access to Box file storage, with protocol "box://"
- `csvbase`_ for access to csvbase.com hosted CSV files, with protocol "csvbase://"
- `dropbox`_ for access to dropbox shares, with protocol "dropbox://"
- `dvc`_ to access DVC/Git repository as a filesystem
- `fsspec-encrypted`_ for transparent encryption on top of other fsspec filesystems.
- `gcsfs`_ for Google Cloud Storage, with protocol "gs://" or "gcs://"
- `gdrive`_ to access Google Drive and shares (experimental)
- `git`_ to access Git repositories
- `huggingface_hub`_ to access the Hugging Face Hub filesystem, with protocol "hf://"
- `hdfs-native`_ to access Hadoop filesystem, with protocol "hdfs://"
- `httpfs-sync`_ to access HTTP(s) files in a synchronous manner to offer an alternative to the aiohttp-based implementation.
- `ipfsspec`_ for the InterPlanetary File System (IPFS), with protocol "ipfs://"
- `irods`_ for access to iRODS servers, with protocol "irods://"
- `lakefs`_ for lakeFS data lakes, with protocol "lakefs://"
- `morefs`_ for `OverlayFileSystem`, `DictFileSystem`, and others
- `obstore`_: zero-dependency access to Amazon S3, Google Cloud Storage, and Azure Blob Storage using the underlying Rust `object_store`_ library, with protocols "s3://", "gs://", and "abfs://".
- `ocifs`_ for access to Oracle Cloud Object Storage, with protocol "oci://"
- `ocilake`_ for OCI Data Lake storage
- `ossfs`_ for Alibaba Cloud (Aliyun) Object Storage System (OSS)
- `p9fs`_ for 9P (Plan 9 Filesystem Protocol) servers
- `PyAthena`_ for S3 access to Amazon Athena, with protocol "s3://" or "s3a://"
- `PyDrive2`_ for Google Drive access
- `fsspec-proxy`_ for "pyscript:" URLs via a proxy server
- `s3fs`_ for Amazon S3 and other compatible stores, with protocol "s3://"
- `sshfs`_ for access to SSH servers, with protocol "ssh://" or "sftp://"
- `swiftspec`_ for OpenStack SWIFT, with protocol "swift://"
- `tosfs`_ for ByteDance volcano engine Tinder Object Storage (TOS)
- `wandbfs`_ to access Wandb run data (experimental)
- `wandbfsspec`_ to access Weights & Biases (experimental)
- `webdav4`_ for WebDAV, with protocol "webdav://" or "dav://"
- `xrootd`_ for xrootd, with protocol "root://"
- `msgraphfs`_ for Microsoft storage (ie Sharepoint) using the drive API through Microsoft Graph, with protocol "msgd://"

.. _abfs: https://github.com/dask/adlfs
.. _adl: https://github.com/dask/adlfs
.. _alluxiofs: https://github.com/fsspec/alluxiofs
.. _boxfs: https://github.com/IBM/boxfs
.. _csvbase: https://github.com/calpaterson/csvbase-client
.. _dropbox: https://github.com/fsspec/dropboxdrivefs
.. _dvc: https://github.com/iterative/dvc
.. _fsspec-encrypted: https://github.com/thevgergroup/fsspec-encrypted
.. _fsspec-proxy: https://github.com/fsspec/fsspec-proxy
.. _gcsfs: https://gcsfs.readthedocs.io/en/latest/
.. _gdrive: https://github.com/fsspec/gdrivefs
.. _git: https://github.com/iterative/scmrepo
.. _hdfs-native: https://github.com/Kimahriman/hdfs-native/blob/master/python/hdfs_native/fsspec.py
.. _httpfs-sync: https://github.com/moradology/httpfs-sync
.. _huggingface_hub: https://huggingface.co/docs/huggingface_hub/main/en/guides/hf_file_system
.. _ipfsspec: https://github.com/fsspec/ipfsspec
.. _irods: https://github.com/xwcl/irods_fsspec
.. _lakefs: https://github.com/aai-institute/lakefs-spec
.. _morefs: https://github.com/iterative/morefs
.. _object_store: https://docs.rs/object_store/latest/object_store/
.. _obstore: https://developmentseed.org/obstore/latest/
.. _ocifs: https://ocifs.readthedocs.io/en/latest/
.. _ocilake: https://github.com/oracle/ocifs
.. _ossfs: https://github.com/fsspec/ossfs
.. _p9fs: https://github.com/pbchekin/p9fs-py
.. _PyAthena: https://github.com/laughingman7743/PyAthena
.. _PyDrive2: https://github.com/iterative/PyDrive2
.. _s3fs: https://s3fs.readthedocs.io/en/latest/
.. _sshfs: https://github.com/fsspec/sshfs
.. _swiftspec: https://github.com/fsspec/swiftspec
.. _tosfs: https://tosfs.readthedocs.io/en/latest/
.. _wandbfs: https://github.com/jkulhanek/wandbfs
.. _wandbfsspec: https://github.com/alvarobartt/wandbfsspec
.. _webdav4: https://github.com/skshetry/webdav4
.. _xrootd: https://github.com/CoffeaTeam/fsspec-xrootd
.. _msgraphfs: https://github.com/acsone/msgraphfs

.. _readbuffering:

Read Buffering
--------------

.. autosummary::
   caching.BlockCache
   caching.BytesCache
   caching.MMapCache
   caching.ReadAheadCache
   caching.FirstChunkCache
   caching.BackgroundBlockCache

.. autoclass:: fsspec.caching.BlockCache
   :members:

.. autoclass:: fsspec.caching.BytesCache
   :members:

.. autoclass:: fsspec.caching.MMapCache
   :members:

.. autoclass:: fsspec.caching.ReadAheadCache
   :members:

.. autoclass:: fsspec.caching.FirstChunkCache
   :members:

.. autoclass:: fsspec.caching.BackgroundBlockCache
   :members:

Utilities
---------

.. autosummary::

   utils.read_block

.. autofunction:: fsspec.utils.read_block