<NME> features.rst
<BEF> Features of fsspec
==================

Here follows a brief description of some features of note of ``fsspec`` that provides to make
it an interesting project beyond some other file-system abstractions.

Serialisability
---------------

Coming out of the Dask stable, it was an important design decision that file-system instances
be serialisable, so that they could be created in one process (e.g., the client) and used in
other processes (typically the workers). These other processes may even be on other machines,
so in many cases they would need to be able to re-establish credentials, ideally without passing
sensitive tokens in the pickled binary data.

``fsspec`` instances, generally speaking, abide by these rules, do not include locks, files and other
thread-local material, and where possible, use local credentials (such as a token file)
for re-establishing sessions upon de-serialisation. (While making use of cached instances, where
they exist, see below).

``OpenFile`` instances
----------------------

The :func:`fsspec.core.OpenFile` class provides a convenient way to prescribe the manner to
open some file (local,
remote, in a compressed store, etc.) which is portable, and can also apply any compression and
text-mode to the file. These instances are also serialisable, because they do not contain any open
files.

The way to work with ``OpenFile`` s is to isolate interaction with in a ``with`` context. It is
the initiation of the context which actually does the work of creating file-like instances.

.. code-block:: python

    of = fsspec.open(url, ...)
    # of is just a place-holder
    with of as f:
        # f is now a real file-like object holding resources
        f.read(...)

File Buffering and random access
--------------------------------

Most implementations create file objects which derive from ``fsspec.spec.AbstractBufferedFile``, and
have many behaviours in common. A subclass of ``AbstractBufferedFile`` provides
random access for the underlying file-like data (without downloading the whole thing).
This is a critical feature in the big-data access model, where each sub-task of an operation
may need on a small part of a file, and does not, therefore want to be forced into downloading the
whole thing.

These files offer buffering of both read and write operations, so that
communication with the remote resource is limited. The size of the buffer is generally configured
with the ``blocksize=`` kwarg at open time, although the implementation may have some minimum or
maximum sizes that need to be respected.

For reading, a number of buffering schemes are available, listed in ``fsspec.caching.caches``
(see :ref:`readbuffering`), or "none" for no buffering at all, e.g., for a simple read-ahead
buffer, you can do

.. code-block:: python

   fs = fsspec.filesystem(...)
   with fs.open(path, mode='rb', cache_type='readahead') as f:
       use_for_something(f)

Transparent text-mode and compression
-------------------------------------

As mentioned above, the ``OpenFile`` class allows for the opening of files on a binary store,
which appear to be in text mode and/or allow for a compression/decompression layer between the
caller and the back-end storage system. The list of ``fsspec`` supported codec
can be retrieved using :func:`fsspec.available_compressions`.
From the user's point of view, this is achieved simply by passing arguments to
the :func:`fsspec.open_files` or :func:`fsspec.open` functions, and
thereafter happens transparently.

Key-value stores
----------------

File-systems are naturally like dict-like key-value mappings: each (string) path corresponds to some
binary data on the storage back-end. For some use-cases, it is very convenient to be able to
view some path within the file-system as a dict-like store, and the function :func:`fsspec.get_mapper`
gives a one-stop way to return such an object. This has become useful, for example, in the
context of the `zarr`_ project, which stores it array chunks in keys in any arbitrary mapping-like
object.

.. code-block:: python

    mapper = fsspec.get_mapper('protocol://server/path', args)
    list(mapper)
    mapper[k] = b'some data'

.. _zarr: https://zarr.readthedocs.io/en/stable/

PyArrow integration
-------------------

`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.fs.FileSystem``),
and some functions, particularly the loading of parquet, require that the target be compatible.
As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with ``fsspec``
(this is not by accident).

At import time, ``fsspec`` checks for the existence of ``pyarrow``, and, if ``pyarrow < 2.0`` is
found, adds its base filesystem to the superclasses of the spec base-class.
For ``pyarrow >= 2.0``, ``fsspec`` file systems can simply be passed to ``pyarrow`` functions
that expect ``pyarrow`` filesystems, and ``pyarrow`` `will automatically wrap them
<https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems>`_.

In this manner, all ``fsspec``-derived file-systems are also ``pyarrow`` file-systems, and can be used
by ``pyarrow`` functions.


.. _pyarrow: https://arrow.apache.org/docs/python/

Transactions
------------

``fsspec`` supports *transactions*, during which writing to files on a remote store are deferred
(typically put into a temporary location) until the transaction is over, whereupon the whole
transaction is finalised in a semi-atomic way, and all the files are moved/committed to their
final destination. The implementation of the details is file-system specific (and not all
support it yet), but the idea is,
that all files should get written or none, to mitigate against data corruption. The feature
can be used like

.. code-block:: python

    fs = fsspec.filesystem(...)
    with fs.transaction:
        with fs.open('file1', 'wb') as f:
            f.write(b'some data')
        with fs.open('file2', 'wb') as f:
            f.write(b'more data')

Here, files 1 and 2 do not get moved to the target location until the transaction context finishes.
If the context finishes due to an (uncaught) exception, then the files are discarded and the
file target locations untouched.

The class :func:`fsspec.spec.Transaction` allows for fine-tuning of the operation, and every
``fsspec`` instance has an instance of this as an attribute ``.transaction`` to give access.

Note that synchronising transactions across multiple instances, perhaps across a cluster,
is a harder problem to solve, and the implementation described here is only part of the solution.

Mount anything with FUSE
------------------------

Any path of any file-system can be mapped to a local directory using `fusepy <https://pypi.org/project/fusepy/>`_ and
:func:`fsspec.fuse.run`. This feature is experimental, but basic file listing with
details, and read/write should generally be available to the extent that the
remote file-system provides enough information. Naturally, if a file-system is read-only,
then write operations will fail - but they will tend to fail late and with obscure
error messages such as "bad address".

Some specific quirks of some file-systems may cause confusion for FUSE. For example,
it is possible for a given path on s3 to be both a valid key (i.e., containing binary
data, like a file) and a valid prefix (i.e., can be listed to find subkeys, like a
directory). Since this breaks the assumptions of a normal file-system, it may not
be possible to reach all paths on the remote.

Instance Caching
----------------

In a file-system implementation class is marked as *cachable* (attribute ``.cachable``),
then its instances will
get stored in a class attribute, to enable quick look-up instead of needing to regenerate
potentially expensive connections and sessions. They key in the cache is a tokenisation of
the arguments to create the instance. The cache itself (attribute ``._cache``)
is currently a simple dict, but could in the future be LRU, or something more complicated,
to fine-tune instance lifetimes.

Since files can hold on to write caches and read buffers,
the instance cache may cause excessive memory usage in some situations; but normally, files'
``close`` methods will be called, discarding the data. Only when there is also an unfinalised transaction or
captured traceback might this be anticipated becoming a problem.

To disable instance caching, i.e., get a fresh instance which is not in the cache
even for a cachable class, pass ``skip_instance_cache=True``.

Listings Caching
----------------

For some implementations, getting file listings (i.e., ``ls`` and anything that
depends on it) is expensive. These implementations use dict-like instances of
:class:`fsspec.dircache.DirCache` to manage the listings.

The cache allows for time-based expiry of entries with the ``listings_expiry_time``
parameter, or LRU expiry with the ``max_paths`` parameter. These can be
set on any implementation instance that uses listings caching; or to skip the
caching altogether, use ``use_listings_cache=False``. That would be appropriate
when the target location is known to be volatile because it is being written
to from other sources.

When the ``fsspec`` instance writes to the backend, the method ``invalidate_cache``
is called, so that subsequent listing of the given paths will force a refresh. In
addition, some methods like ``ls`` have a ``refresh`` parameter to force fetching
the listing again.

URL chaining
------------

Some implementations proxy or otherwise make use of another filesystem implementation, such
as locally caching remote files, i.e., finding out what files exist using the remote implementation,
but actually opening the local copies upon access. Other examples include reading from a Dask worker
which can see file-systems the client cannot, and accessing a zip file which is being read from
another backend.
   with fs.open(path, mode='rb', cache_type='readahead') as f:
       use_for_something(f)

Caching Files Locally
---------------------

   fs = fsspec.filesystem("filecache", target_protocol='s3', target_options={'anon': True},
                          cache_storage='/tmp/files/')

Each time you open a remote file on S3, it will first copy it to
a local temporary directory, and then all further access will use the local file. Since we specify
a particular local location, the files will persist and can be reused from future sessions, although
you can also set policies to have cached files expire after some time, or to check the remote file system
on each open, to see if the target file has changed since it was copied.

With the top-level functions ``open``, ``open_local`` and ``open_files``, you can use the
same set of kwargs as the example above, or you can chain the URL - the following would
be the equivalent

.. code-block:: python

    of = fsspec.open("filecache::s3://bucket/key",
                     s3={'anon': True}, filecache={'cache_storage':'/tmp/files'})
you can also set policies to have cached files expire after some time, or to check the remote file system
on each open, to see if the target file has changed since it was copied.

With the "blockcache" variant, data is downloaded block-wise: only the specific parts of the remote file
which are accessed. This means that the local copy of the file might end up being much smaller than the
remote one, if only certain parts of it are required.
--------------------

libraries to use, "blockcache" has restrictions: that you have a storage/OS combination which supports
sparse files, that the backend implementation uses files which derive ``from AbstractBufferedFile``,
and that the library you pass the resultant object to accepts generic python file-like objects. You
should not mix block- and file-caches in the same directory.
       f.write(b"some data")

This will open a local file for writing, and when this file is closed, it will be uploaded
to the target URL, in this case on S3. The file-like object ``f`` can be passed to any
library expecting to write to a file. Note that we pass parameters to ``S3FileSystem`` using
the key ``"s3"``, the same as the name of the protocol.

File Selector (GUI)
-------------------

The module ``fsspec.gui`` contains a graphical file selector interface. It is built
using `panel`_, which must be installed in order to use the GUI. Upon instantiation,
you can provide the initial URL location (which can be returned to with the "🏠" button),
arguments and filters.

.. _panel: https://panel.holoviz.org/

.. image:: img/gui.png

Clicking on a directory will descend into it, and selecting a file will mark it as
the output of the interface. You can select any of the known protocols, but should
provide any required arguments in the "kwargs" box (as a dictionary) and any
absolute URL location before clicking "⇨" to go to that location. If using file filters,
they will appear as a list of checkboxes; only those file-endings selected will be
shown (or if none are selected, all files are shown).

The interface provides the following outputs:

#. ``.urlpath``: the currently selected item (if any)
#. ``.storage_options``: the value of the kwargs box
#. ``.fs``: the current filesystem instance
#. ``.open_file()``: produces an ``OpenFile`` instance for the current selection

Configuration
-------------

You can set default keyword arguments to pass to any fsspec backend by editing
config files, providing environment variables, or editing the contents of
the dictionary ``fsspec.config.conf``.

Files are stored in the directory pointed to by ``FSSPEC_CONFIG_DIR``,
``"~/.config/fsspec/`` by default. All \*.ini and \*.json files will be
loaded and parsed from their respective formats and fed into the config dict
at import time. For example, if there is a file "~/.config/fsspec/conf.json"
containing

.. code-block:: json

   {"file": {"auto_mkdir": true}}

then any instance of the file system whose protocol is "file" (i.e.,
``LocalFileSystem``) with be passed the kwargs ``auto_mkdir=True``
**unless** the user supplies the kwarg themselves.

For instance:

.. code-block:: python

    import fsspec
    fs = fsspec.filesystem("file")
    assert fs.auto_mkdir == True
    fs = fsspec.filesystem("file", auto_mkdir=False)
    assert fs.auto_mkdir == False

Obviously, you should only define default values that are appropriate for
a given file system implementation. INI files only support string values.

Alternatively, you can provide overrides with environment variables of
the style ``FSSPEC_{protocol}_{kwargname}=value``.

Configuration is determined in the following order, with later items winning:

#. the contents of ini files, and json files in the config directory, sorted
   alphabetically
#. environment variables
#. the contents of ``fsspec.config.conf``, which can be edited at runtime
#. kwargs explicitly passed, whether with ``fsspec.open``, ``fsspec.filesystem``
   or directly instantiating the implementation class.


Asynchronous
------------

Some implementations, those deriving from ``fsspec.asyn.AsyncFileSystem``, have
async/coroutine implementations of some file operations. The async methods have
names beginning with ``_``, and listed in the ``asyn`` module; synchronous or
blocking functions are automatically generated, which will operate via an
event loop in another thread, by default.

See :doc:`async` for modes of operation and how to implement such file systems.


Callbacks
---------

Some methods support a ``callback=`` argument, which is the entry point to
providing feedback on transfers to the user or any other logging service. This
feature is new and experimental and supported by varying amounts in the
backends.

See the docstrings in the callbacks module for further details.
``fsspec.callbacks.TqdmCallback`` can be used to display a progress bar using
tqdm.

<MSG> Merge pull request #258 from martindurant/chain-docs

Update docs after chains
<DFF> @@ -207,6 +207,53 @@ buffer, you can do
    with fs.open(path, mode='rb', cache_type='readahead') as f:
        use_for_something(f)
 
+URL chaining
+------------
+
+Some implementations proxy or otherwise make use of another filesystem implementation, such
+as locally caching remote files, i.e., finding out what files exist using the remote implementation,
+but actually opening the local copies upon access. Other examples include reading from a Dask worker
+which can see file-systems the client cannot, and accessing a zip file which is being read from
+another backend.
+
+In such cases, you can specify the parameters exactly as specified in the implementation docstrings,
+for the dask case something like
+
+.. code-block:: python
+
+    of = fsspec.open('dask://bucket/key', target_protocol='s3', target_options={'anon': True})
+
+As a shorthand, particularly useful where you have multiple hops, is to "chain" the URLs with
+the special separator ``"::"``. The arguments to be passed on to each of the implementations referenced
+are keywed by the protocol names included in the URL. Here is the equivalent to the line above:
+
+.. code-block:: python
+
+   of = fsspec.open('dask::s3://bucket/key', s3={'anon': True})
+
+A couple of more complicates cases:
+
+.. code-block:: python
+
+  of = fsspec.open_files('zip://*.csv::simplecache::gcs://bucket/afile.zip',
+                         simplecache={'cache_storage': '/stored/zip/files'},
+                         gcs={'project': 'my-project'})
+
+reads a zip-file from google, stores it locally, and gives access to the contained CSV files. Conversely,
+
+.. code-block:: python
+
+  of = fsspec.open_files('simplecache::zip://*.csv::gcs://bucket/afile.zip',
+                         simplecache={'cache_storage': '/stored/csv/files'},
+                         gcs={'project': 'my-project'})
+
+reads the same zip-file, but extracts the CSV files and stores them locally in the cache.
+
+**For developers**: this "chaining" methods works by formatting the arguments passed to ``open_*``
+into ``target_protocol`` (a simple string) and ``target_options`` (a dict) and also optionally
+``fo`` (target path, if a specific file is required). In order for an implementation to chain
+successfully like this, it must look for exactly those named arguments.
+
 Caching Files Locally
 ---------------------
 
@@ -230,6 +277,15 @@ a particular local location, the files will persist and can be reused from futur
 you can also set policies to have cached files expire after some time, or to check the remote file system
 on each open, to see if the target file has changed since it was copied.
 
+With the top-level functions ``open``, ``open_local`` and ``open_files``, you can use the
+same set of kwargs as the example above, or you can chain the URL - the following would
+be the equivalent
+
+.. code-block:: python
+
+    of = fsspec.open("filecache::s3://bucket/key",
+                     s3={'anon': True}, cache_storage='/tmp/files')
+
 With the "blockcache" variant, data is downloaded block-wise: only the specific parts of the remote file
 which are accessed. This means that the local copy of the file might end up being much smaller than the
 remote one, if only certain parts of it are required.
@@ -238,4 +294,7 @@ Whereas "filecache" works for all file system implementations, and provides a re
 libraries to use, "blockcache" has restrictions: that you have a storage/OS combination which supports
 sparse files, that the backend implementation uses files which derive ``from AbstractBufferedFile``,
 and that the library you pass the resultant object to accepts generic python file-like objects. You
-should not mix block- and file-caches in the same directory.
+should not mix block- and file-caches in the same directory. "simplecache" is the same as "filecache",
+except without the options for cache expiry and to check the original source - it can be used where the
+target can be considered static, and particularly where a large number of target files are expected
+(because no metadata is written to disc). Only "simplecache" is guaranteed thread/process-safe.

{'commit': 'diff --git a/docs/source/features.rst b/docs/source/features.rst\nindex 7787b51..1646b48 100644\n--- a/docs/source/features.rst\n+++ b/docs/source/features.rst\n@@ -207,6 +207,53 @@ buffer, you can do\n    with fs.open(path, mode=\'rb\', cache_type=\'readahead\') as f:\n        use_for_something(f)\n \n+URL chaining\n+------------\n+\n+Some implementations proxy or otherwise make use of another filesystem implementation, such\n+as locally caching remote files, i.e., finding out what files exist using the remote implementation,\n+but actually opening the local copies upon access. Other examples include reading from a Dask worker\n+which can see file-systems the client cannot, and accessing a zip file which is being read from\n+another backend.\n+\n+In such cases, you can specify the parameters exactly as specified in the implementation docstrings,\n+for the dask case something like\n+\n+.. code-block:: python\n+\n+    of = fsspec.open(\'dask://bucket/key\', target_protocol=\'s3\', target_options={\'anon\': True})\n+\n+As a shorthand, particularly useful where you have multiple hops, is to "chain" the URLs with\n+the special separator ``"::"``. The arguments to be passed on to each of the implementations referenced\n+are keywed by the protocol names included in the URL. Here is the equivalent to the line above:\n+\n+.. code-block:: python\n+\n+   of = fsspec.open(\'dask::s3://bucket/key\', s3={\'anon\': True})\n+\n+A couple of more complicates cases:\n+\n+.. code-block:: python\n+\n+  of = fsspec.open_files(\'zip://*.csv::simplecache::gcs://bucket/afile.zip\',\n+                         simplecache={\'cache_storage\': \'/stored/zip/files\'},\n+                         gcs={\'project\': \'my-project\'})\n+\n+reads a zip-file from google, stores it locally, and gives access to the contained CSV files. Conversely,\n+\n+.. code-block:: python\n+\n+  of = fsspec.open_files(\'simplecache::zip://*.csv::gcs://bucket/afile.zip\',\n+                         simplecache={\'cache_storage\': \'/stored/csv/files\'},\n+                         gcs={\'project\': \'my-project\'})\n+\n+reads the same zip-file, but extracts the CSV files and stores them locally in the cache.\n+\n+**For developers**: this "chaining" methods works by formatting the arguments passed to ``open_*``\n+into ``target_protocol`` (a simple string) and ``target_options`` (a dict) and also optionally\n+``fo`` (target path, if a specific file is required). In order for an implementation to chain\n+successfully like this, it must look for exactly those named arguments.\n+\n Caching Files Locally\n ---------------------\n \n@@ -230,6 +277,15 @@ a particular local location, the files will persist and can be reused from futur\n you can also set policies to have cached files expire after some time, or to check the remote file system\n on each open, to see if the target file has changed since it was copied.\n \n+With the top-level functions ``open``, ``open_local`` and ``open_files``, you can use the\n+same set of kwargs as the example above, or you can chain the URL - the following would\n+be the equivalent\n+\n+.. code-block:: python\n+\n+    of = fsspec.open("filecache::s3://bucket/key",\n+                     s3={\'anon\': True}, cache_storage=\'/tmp/files\')\n+\n With the "blockcache" variant, data is downloaded block-wise: only the specific parts of the remote file\n which are accessed. This means that the local copy of the file might end up being much smaller than the\n remote one, if only certain parts of it are required.\n@@ -238,4 +294,7 @@ Whereas "filecache" works for all file system implementations, and provides a re\n libraries to use, "blockcache" has restrictions: that you have a storage/OS combination which supports\n sparse files, that the backend implementation uses files which derive ``from AbstractBufferedFile``,\n and that the library you pass the resultant object to accepts generic python file-like objects. You\n-should not mix block- and file-caches in the same directory.\n+should not mix block- and file-caches in the same directory. "simplecache" is the same as "filecache",\n+except without the options for cache expiry and to check the original source - it can be used where the\n+target can be considered static, and particularly where a large number of target files are expected\n+(because no metadata is written to disc). Only "simplecache" is guaranteed thread/process-safe.\n', 'commit_subject': 'Merge pull request #258 from martindurant/chain-docs', 'repo_name': 'intake/filesystem_spec', 'committer': {'name': 'GitHub', 'email': '1505422b2465e9a84f6fdfaa161078890c593f06@github.com', 'time_sec': '1585162238', 'tz_offset': '-240', 'date': {'seconds': '1585162238'}}, 'commit_tree': '024d4b1dbc1bc1202ab0a62e188a5563bef8055b', 'commit_message': 'Merge pull request #258 from martindurant/chain-docs\n\nUpdate docs after chains', 'language_name': 'Python', 'license': 'bsd-3-clause', 'hunks': '@@ -207,6 +207,53 @@ buffer, you can do\n    with fs.open(path, mode=\'rb\', cache_type=\'readahead\') as f:\n        use_for_something(f)\n \n+URL chaining\n+------------\n+\n+Some implementations proxy or otherwise make use of another filesystem implementation, such\n+as locally caching remote files, i.e., finding out what files exist using the remote implementation,\n+but actually opening the local copies upon access. Other examples include reading from a Dask worker\n+which can see file-systems the client cannot, and accessing a zip file which is being read from\n+another backend.\n+\n+In such cases, you can specify the parameters exactly as specified in the implementation docstrings,\n+for the dask case something like\n+\n+.. code-block:: python\n+\n+    of = fsspec.open(\'dask://bucket/key\', target_protocol=\'s3\', target_options={\'anon\': True})\n+\n+As a shorthand, particularly useful where you have multiple hops, is to "chain" the URLs with\n+the special separator ``"::"``. The arguments to be passed on to each of the implementations referenced\n+are keywed by the protocol names included in the URL. Here is the equivalent to the line above:\n+\n+.. code-block:: python\n+\n+   of = fsspec.open(\'dask::s3://bucket/key\', s3={\'anon\': True})\n+\n+A couple of more complicates cases:\n+\n+.. code-block:: python\n+\n+  of = fsspec.open_files(\'zip://*.csv::simplecache::gcs://bucket/afile.zip\',\n+                         simplecache={\'cache_storage\': \'/stored/zip/files\'},\n+                         gcs={\'project\': \'my-project\'})\n+\n+reads a zip-file from google, stores it locally, and gives access to the contained CSV files. Conversely,\n+\n+.. code-block:: python\n+\n+  of = fsspec.open_files(\'simplecache::zip://*.csv::gcs://bucket/afile.zip\',\n+                         simplecache={\'cache_storage\': \'/stored/csv/files\'},\n+                         gcs={\'project\': \'my-project\'})\n+\n+reads the same zip-file, but extracts the CSV files and stores them locally in the cache.\n+\n+**For developers**: this "chaining" methods works by formatting the arguments passed to ``open_*``\n+into ``target_protocol`` (a simple string) and ``target_options`` (a dict) and also optionally\n+``fo`` (target path, if a specific file is required). In order for an implementation to chain\n+successfully like this, it must look for exactly those named arguments.\n+\n Caching Files Locally\n ---------------------\n \n@@ -230,6 +277,15 @@ a particular local location, the files will persist and can be reused from futur\n you can also set policies to have cached files expire after some time, or to check the remote file system\n on each open, to see if the target file has changed since it was copied.\n \n+With the top-level functions ``open``, ``open_local`` and ``open_files``, you can use the\n+same set of kwargs as the example above, or you can chain the URL - the following would\n+be the equivalent\n+\n+.. code-block:: python\n+\n+    of = fsspec.open("filecache::s3://bucket/key",\n+                     s3={\'anon\': True}, cache_storage=\'/tmp/files\')\n+\n With the "blockcache" variant, data is downloaded block-wise: only the specific parts of the remote file\n which are accessed. This means that the local copy of the file might end up being much smaller than the\n remote one, if only certain parts of it are required.\n@@ -238,4 +294,7 @@ Whereas "filecache" works for all file system implementations, and provides a re\n libraries to use, "blockcache" has restrictions: that you have a storage/OS combination which supports\n sparse files, that the backend implementation uses files which derive ``from AbstractBufferedFile``,\n and that the library you pass the resultant object to accepts generic python file-like objects. You\n-should not mix block- and file-caches in the same directory.\n+should not mix block- and file-caches in the same directory. "simplecache" is the same as "filecache",\n+except without the options for cache expiry and to check the original source - it can be used where the\n+target can be considered static, and particularly where a large number of target files are expected\n+(because no metadata is written to disc). Only "simplecache" is guaranteed thread/process-safe.\n', 'addition_count': 60, 'deletion_count': 1, 'src_file': 'a/docs/source/features.rst', 'tgt_file': 'b/docs/source/features.rst', 'file_extension': '.rst', 'before_file': 'Features of fsspec\n==================\n\nHere follows a brief description of some features of note of ``fsspec`` that provides to make\nit an interesting project beyond some other file-system abstractions.\n\nSerialisability\n---------------\n\nComing out of the Dask stable, it was an important design decision that file-system instances\nbe serialisable, so that they could be created in one process (e.g., the client) and used in\nother processes (typically the workers). These other processes may even be on other machines,\nso in many cases they would need to be able to re-establish credentials, ideally without passing\nsensitive tokens in the pickled binary data.\n\n``fsspec`` instances, generally speaking, abide by these rules, do not include locks, files and other\nthread-local material, and where possible, use local credentials (such as a token file)\nfor re-establishing sessions upon de-serialisation. (While making use of cached instances, where\nthey exist, see below).\n\n``OpenFile`` instances\n----------------------\n\nThe :func:`fsspec.core.OpenFile` class provides a convenient way to prescribe the manner to\nopen some file (local,\nremote, in a compressed store, etc.) which is portable, and can also apply any compression and\ntext-mode to the file. These instances are also serialisable, because they do not contain any open\nfiles.\n\nThe way to work with ``OpenFile`` s is to isolate interaction with in a ``with`` context. It is\nthe initiation of the context which actually does the work of creating file-like instances.\n\n.. code-block:: python\n\n    of = fsspec.open(url, ...)\n    # of is just a place-holder\n    with of as f:\n        # f is now a real file-like object holding resources\n        f.read(...)\n\nFile Buffering and random access\n--------------------------------\n\nMost implementations create file objects which derive from ``fsspec.spec.AbstractBufferedFile``, and\nhave many behaviours in common. A subclass of ``AbstractBufferedFile`` provides\nrandom access for the underlying file-like data (without downloading the whole thing).\nThis is a critical feature in the big-data access model, where each sub-task of an operation\nmay need on a small part of a file, and does not, therefore want to be forced into downloading the\nwhole thing.\n\nThese files offer buffering of both read and write operations, so that\ncommunication with the remote resource is limited. The size of the buffer is generally configured\nwith the ``blocksize=`` kwarg at open time, although the implementation may have some minimum or\nmaximum sizes that need to be respected.\n\nFor reading, a number of buffering schemes are available, listed in ``fsspec.caching.caches``\n(see :ref:`readbuffering`), or "none" for no buffering at all, e.g., for a simple read-ahead\nbuffer, you can do\n\n.. code-block:: python\n\n   fs = fsspec.filesystem(...)\n   with fs.open(path, mode=\'rb\', cache_type=\'readahead\') as f:\n       use_for_something(f)\n\nTransparent text-mode and compression\n-------------------------------------\n\nAs mentioned above, the ``OpenFile`` class allows for the opening of files on a binary store,\nwhich appear to be in text mode and/or allow for a compression/decompression layer between the\ncaller and the back-end storage system. The list of ``fsspec`` supported codec\ncan be retrieved using :func:`fsspec.available_compressions`.\nFrom the user\'s point of view, this is achieved simply by passing arguments to\nthe :func:`fsspec.open_files` or :func:`fsspec.open` functions, and\nthereafter happens transparently.\n\nKey-value stores\n----------------\n\nFile-systems are naturally like dict-like key-value mappings: each (string) path corresponds to some\nbinary data on the storage back-end. For some use-cases, it is very convenient to be able to\nview some path within the file-system as a dict-like store, and the function :func:`fsspec.get_mapper`\ngives a one-stop way to return such an object. This has become useful, for example, in the\ncontext of the `zarr`_ project, which stores it array chunks in keys in any arbitrary mapping-like\nobject.\n\n.. code-block:: python\n\n    mapper = fsspec.get_mapper(\'protocol://server/path\', args)\n    list(mapper)\n    mapper[k] = b\'some data\'\n\n.. _zarr: https://zarr.readthedocs.io/en/stable/\n\nPyArrow integration\n-------------------\n\n`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.fs.FileSystem``),\nand some functions, particularly the loading of parquet, require that the target be compatible.\nAs it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with ``fsspec``\n(this is not by accident).\n\nAt import time, ``fsspec`` checks for the existence of ``pyarrow``, and, if ``pyarrow < 2.0`` is\nfound, adds its base filesystem to the superclasses of the spec base-class.\nFor ``pyarrow >= 2.0``, ``fsspec`` file systems can simply be passed to ``pyarrow`` functions\nthat expect ``pyarrow`` filesystems, and ``pyarrow`` `will automatically wrap them\n<https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems>`_.\n\nIn this manner, all ``fsspec``-derived file-systems are also ``pyarrow`` file-systems, and can be used\nby ``pyarrow`` functions.\n\n\n.. _pyarrow: https://arrow.apache.org/docs/python/\n\nTransactions\n------------\n\n``fsspec`` supports *transactions*, during which writing to files on a remote store are deferred\n(typically put into a temporary location) until the transaction is over, whereupon the whole\ntransaction is finalised in a semi-atomic way, and all the files are moved/committed to their\nfinal destination. The implementation of the details is file-system specific (and not all\nsupport it yet), but the idea is,\nthat all files should get written or none, to mitigate against data corruption. The feature\ncan be used like\n\n.. code-block:: python\n\n    fs = fsspec.filesystem(...)\n    with fs.transaction:\n        with fs.open(\'file1\', \'wb\') as f:\n            f.write(b\'some data\')\n        with fs.open(\'file2\', \'wb\') as f:\n            f.write(b\'more data\')\n\nHere, files 1 and 2 do not get moved to the target location until the transaction context finishes.\nIf the context finishes due to an (uncaught) exception, then the files are discarded and the\nfile target locations untouched.\n\nThe class :func:`fsspec.spec.Transaction` allows for fine-tuning of the operation, and every\n``fsspec`` instance has an instance of this as an attribute ``.transaction`` to give access.\n\nNote that synchronising transactions across multiple instances, perhaps across a cluster,\nis a harder problem to solve, and the implementation described here is only part of the solution.\n\nMount anything with FUSE\n------------------------\n\nAny path of any file-system can be mapped to a local directory using `fusepy <https://pypi.org/project/fusepy/>`_ and\n:func:`fsspec.fuse.run`. This feature is experimental, but basic file listing with\ndetails, and read/write should generally be available to the extent that the\nremote file-system provides enough information. Naturally, if a file-system is read-only,\nthen write operations will fail - but they will tend to fail late and with obscure\nerror messages such as "bad address".\n\nSome specific quirks of some file-systems may cause confusion for FUSE. For example,\nit is possible for a given path on s3 to be both a valid key (i.e., containing binary\ndata, like a file) and a valid prefix (i.e., can be listed to find subkeys, like a\ndirectory). Since this breaks the assumptions of a normal file-system, it may not\nbe possible to reach all paths on the remote.\n\nInstance Caching\n----------------\n\nIn a file-system implementation class is marked as *cachable* (attribute ``.cachable``),\nthen its instances will\nget stored in a class attribute, to enable quick look-up instead of needing to regenerate\npotentially expensive connections and sessions. They key in the cache is a tokenisation of\nthe arguments to create the instance. The cache itself (attribute ``._cache``)\nis currently a simple dict, but could in the future be LRU, or something more complicated,\nto fine-tune instance lifetimes.\n\nSince files can hold on to write caches and read buffers,\nthe instance cache may cause excessive memory usage in some situations; but normally, files\'\n``close`` methods will be called, discarding the data. Only when there is also an unfinalised transaction or\ncaptured traceback might this be anticipated becoming a problem.\n\nTo disable instance caching, i.e., get a fresh instance which is not in the cache\neven for a cachable class, pass ``skip_instance_cache=True``.\n\nListings Caching\n----------------\n\nFor some implementations, getting file listings (i.e., ``ls`` and anything that\ndepends on it) is expensive. These implementations use dict-like instances of\n:class:`fsspec.dircache.DirCache` to manage the listings.\n\nThe cache allows for time-based expiry of entries with the ``listings_expiry_time``\nparameter, or LRU expiry with the ``max_paths`` parameter. These can be\nset on any implementation instance that uses listings caching; or to skip the\ncaching altogether, use ``use_listings_cache=False``. That would be appropriate\nwhen the target location is known to be volatile because it is being written\nto from other sources.\n\nWhen the ``fsspec`` instance writes to the backend, the method ``invalidate_cache``\nis called, so that subsequent listing of the given paths will force a refresh. In\naddition, some methods like ``ls`` have a ``refresh`` parameter to force fetching\nthe listing again.\n\nURL chaining\n------------\n\nSome implementations proxy or otherwise make use of another filesystem implementation, such\nas locally caching remote files, i.e., finding out what files exist using the remote implementation,\nbut actually opening the local copies upon access. Other examples include reading from a Dask worker\nwhich can see file-systems the client cannot, and accessing a zip file which is being read from\nanother backend.\n   with fs.open(path, mode=\'rb\', cache_type=\'readahead\') as f:\n       use_for_something(f)\n\nCaching Files Locally\n---------------------\n\n   fs = fsspec.filesystem("filecache", target_protocol=\'s3\', target_options={\'anon\': True},\n                          cache_storage=\'/tmp/files/\')\n\nEach time you open a remote file on S3, it will first copy it to\na local temporary directory, and then all further access will use the local file. Since we specify\na particular local location, the files will persist and can be reused from future sessions, although\nyou can also set policies to have cached files expire after some time, or to check the remote file system\non each open, to see if the target file has changed since it was copied.\n\nWith the top-level functions ``open``, ``open_local`` and ``open_files``, you can use the\nsame set of kwargs as the example above, or you can chain the URL - the following would\nbe the equivalent\n\n.. code-block:: python\n\n    of = fsspec.open("filecache::s3://bucket/key",\n                     s3={\'anon\': True}, filecache={\'cache_storage\':\'/tmp/files\'})\nyou can also set policies to have cached files expire after some time, or to check the remote file system\non each open, to see if the target file has changed since it was copied.\n\nWith the "blockcache" variant, data is downloaded block-wise: only the specific parts of the remote file\nwhich are accessed. This means that the local copy of the file might end up being much smaller than the\nremote one, if only certain parts of it are required.\n--------------------\n\nlibraries to use, "blockcache" has restrictions: that you have a storage/OS combination which supports\nsparse files, that the backend implementation uses files which derive ``from AbstractBufferedFile``,\nand that the library you pass the resultant object to accepts generic python file-like objects. You\nshould not mix block- and file-caches in the same directory.\n       f.write(b"some data")\n\nThis will open a local file for writing, and when this file is closed, it will be uploaded\nto the target URL, in this case on S3. The file-like object ``f`` can be passed to any\nlibrary expecting to write to a file. Note that we pass parameters to ``S3FileSystem`` using\nthe key ``"s3"``, the same as the name of the protocol.\n\nFile Selector (GUI)\n-------------------\n\nThe module ``fsspec.gui`` contains a graphical file selector interface. It is built\nusing `panel`_, which must be installed in order to use the GUI. Upon instantiation,\nyou can provide the initial URL location (which can be returned to with the "🏠" button),\narguments and filters.\n\n.. _panel: https://panel.holoviz.org/\n\n.. image:: img/gui.png\n\nClicking on a directory will descend into it, and selecting a file will mark it as\nthe output of the interface. You can select any of the known protocols, but should\nprovide any required arguments in the "kwargs" box (as a dictionary) and any\nabsolute URL location before clicking "⇨" to go to that location. If using file filters,\nthey will appear as a list of checkboxes; only those file-endings selected will be\nshown (or if none are selected, all files are shown).\n\nThe interface provides the following outputs:\n\n#. ``.urlpath``: the currently selected item (if any)\n#. ``.storage_options``: the value of the kwargs box\n#. ``.fs``: the current filesystem instance\n#. ``.open_file()``: produces an ``OpenFile`` instance for the current selection\n\nConfiguration\n-------------\n\nYou can set default keyword arguments to pass to any fsspec backend by editing\nconfig files, providing environment variables, or editing the contents of\nthe dictionary ``fsspec.config.conf``.\n\nFiles are stored in the directory pointed to by ``FSSPEC_CONFIG_DIR``,\n``"~/.config/fsspec/`` by default. All \\*.ini and \\*.json files will be\nloaded and parsed from their respective formats and fed into the config dict\nat import time. For example, if there is a file "~/.config/fsspec/conf.json"\ncontaining\n\n.. code-block:: json\n\n   {"file": {"auto_mkdir": true}}\n\nthen any instance of the file system whose protocol is "file" (i.e.,\n``LocalFileSystem``) with be passed the kwargs ``auto_mkdir=True``\n**unless** the user supplies the kwarg themselves.\n\nFor instance:\n\n.. code-block:: python\n\n    import fsspec\n    fs = fsspec.filesystem("file")\n    assert fs.auto_mkdir == True\n    fs = fsspec.filesystem("file", auto_mkdir=False)\n    assert fs.auto_mkdir == False\n\nObviously, you should only define default values that are appropriate for\na given file system implementation. INI files only support string values.\n\nAlternatively, you can provide overrides with environment variables of\nthe style ``FSSPEC_{protocol}_{kwargname}=value``.\n\nConfiguration is determined in the following order, with later items winning:\n\n#. the contents of ini files, and json files in the config directory, sorted\n   alphabetically\n#. environment variables\n#. the contents of ``fsspec.config.conf``, which can be edited at runtime\n#. kwargs explicitly passed, whether with ``fsspec.open``, ``fsspec.filesystem``\n   or directly instantiating the implementation class.\n\n\nAsynchronous\n------------\n\nSome implementations, those deriving from ``fsspec.asyn.AsyncFileSystem``, have\nasync/coroutine implementations of some file operations. The async methods have\nnames beginning with ``_``, and listed in the ``asyn`` module; synchronous or\nblocking functions are automatically generated, which will operate via an\nevent loop in another thread, by default.\n\nSee :doc:`async` for modes of operation and how to implement such file systems.\n\n\nCallbacks\n---------\n\nSome methods support a ``callback=`` argument, which is the entry point to\nproviding feedback on transfers to the user or any other logging service. This\nfeature is new and experimental and supported by varying amounts in the\nbackends.\n\nSee the docstrings in the callbacks module for further details.\n``fsspec.callbacks.TqdmCallback`` can be used to display a progress bar using\ntqdm.\n', 'message': 'Merge pull request #258 from martindurant/chain-docs\n\nUpdate docs after chains', 'author_name': 'GitHub', 'lang': 'rst'}