1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
(backends)=
# {fas}`database` Backends
This page contains general information about the cache backends supported by requests-cache.
The default backend is SQLite, since it requires no extra dependencies or configuration, and has
great all-around performance for the most common use cases.
Here is a full list of backends available, and any extra dependencies required:
Backend | Class | Alias | Dependencies
------------------------------------------------------|----------------------------|----------------|----------------------------------------------------------
 {ref}`sqlite` | {py:class}`.SQLiteCache` | `'sqlite'` |
 {ref}`redis` | {py:class}`.RedisCache` | `'redis'` | [redis-py](https://github.com/redis/redis-py)
 {ref}`mongodb` | {py:class}`.MongoCache` | `'mongodb'` | [pymongo](https://github.com/mongodb/mongo-python-driver)
 {ref}`gridfs` | {py:class}`.GridFSCache` | `'gridfs'` | [pymongo](https://github.com/mongodb/mongo-python-driver)
 {ref}`dynamodb` | {py:class}`.DynamoDbCache` | `'dynamodb'` | [boto3](https://github.com/boto/boto3)
 {ref}`filesystem` | {py:class}`.FileCache` | `'filesystem'` |
 Memory | {py:class}`.BaseCache` | `'memory'` |
<!-- Hidden ToC tree to add pages to sidebar ToC -->
```{toctree}
:hidden:
:glob: true
backends/*
```
## Choosing a Backend
Here are some general notes on choosing a backend:
* All of the backends perform well enough that they usually won't become a bottleneck until you
start hitting around **700-1000 requests per second**
* It's recommended to start with SQLite until you have a specific reason to switch
* If/when you encounter limitations with SQLite, the next logical choice is usually Redis
* Each backend has some unique features that make them well suited for specific use cases; see
individual backend docs for more details
Here are some specific situations where you may want to choose one of the other backends:
* Your application is distributed across multiple machines, without access to a common filesystem
* Your application will make large volumes of concurrent writes (i.e., many nodes/threads/processes caching many different URLs)
* Your application environment only has slower file storage options (like a magnetic drive, or NFS with high latency)
* Your application environment has little or no local storage (like some cloud computing services)
* Your application is already using one of the other backends
* You want to reuse your cached response data outside of requests-cache
* You want to use a specific feature available in one of the other backends
## Specifying a Backend
You can specify which backend to use with the `backend` parameter for either {py:class}`.CachedSession`
or {py:func}`.install_cache`. You can specify one by name, using the aliases listed above:
```python
>>> session = CachedSession('my_cache', backend='redis')
```
Or by instance, which is preferable if you want to pass additional backend-specific options:
```python
>>> backend = RedisCache(host='192.168.1.63', port=6379)
>>> session = CachedSession('my_cache', backend=backend)
```
## Backend Options
The `cache_name` parameter has a different use depending on the backend:
Backend | Cache name used as
----------------|-------------------
SQLite | Database path
Redis | Hash namespace
MongoDB, GridFS | Database name
DynamoDB | Table name
Filesystem | Cache directory
Each backend class also accepts optional parameters for the underlying connection. For example,
the {ref}`sqlite` backend accepts parameters for {py:func}`sqlite3.connect`.
## Testing Backends
If you just want to quickly try out all of the available backends for comparison,
[docker-compose](https://docs.docker.com/compose/) config is included for all supported services.
First, [install docker](https://docs.docker.com/get-docker/) if you haven't already. Then, run:
::::{tab-set}
:::{tab-item} Bash (Linux/macOS)
```bash
pip install -U requests-cache[all] docker-compose
curl https://raw.githubusercontent.com/requests-cache/requests-cache/main/docker-compose.yml -O docker-compose.yml
docker-compose up -d
```
:::
:::{tab-item} Powershell (Windows)
```ps1
pip install -U requests-cache[all] docker-compose
Invoke-WebRequest -Uri https://raw.githubusercontent.com/requests-cache/requests-cache/main/docker-compose.yml -Outfile docker-compose.yml
docker-compose up -d
```
:::
::::
(exporting)=
## Exporting To A Different Backend
If you have cached data that you want to copy or migrate to a different backend, you can do this
with `CachedSession.cache.update()`. For example, if you want to dump the contents of a Redis cache
to JSON files:
```python
>>> src_session = CachedSession('my_cache', backend='redis')
>>> dest_session = CachedSession('~/workspace/cache_dump', backend='filesystem', serializer='json')
>>> dest_session.cache.update(src_session.cache)
>>> # List the exported files
>>> print(dest_session.cache.paths())
'/home/user/workspace/cache_dump/9e7a71a3ff2e.json'
'/home/user/workspace/cache_dump/8a922ff3c53f.json'
```
Or, using backend classes directly:
```python
>>> src_cache = RedisCache()
>>> dest_cache = FileCache('~/workspace/cache_dump', serializer='json')
>>> dest_cache.update(src_cache)
```
(custom-backends)=
## Custom Backends
If the built-in backends don't suit your needs, you can create your own by making subclasses of {py:class}`.BaseCache` and {py:class}`.BaseStorage`:
:::{dropdown} Example
:animate: fade-in-slide-down
:color: primary
:icon: file-code
```python
>>> from requests_cache import CachedSession
>>> from requests_cache.backends import BaseCache, BaseStorage
>>> class CustomCache(BaseCache):
... """Wrapper for higher-level cache operations. In most cases, the only thing you need
... to specify here is which storage class(es) to use.
... """
... def __init__(self, **kwargs):
... super().__init__(**kwargs)
... self.redirects = CustomStorage(**kwargs)
... self.responses = CustomStorage(**kwargs)
>>> class CustomStorage(BaseStorage):
... """Dict-like interface for lower-level backend storage operations"""
... def __init__(self, **kwargs):
... super().__init__(**kwargs)
...
... def __getitem__(self, key):
... pass
...
... def __setitem__(self, key, value):
... pass
...
... def __delitem__(self, key):
... pass
...
... def __iter__(self):
... pass
...
... def __len__(self):
... pass
...
... def clear(self):
... pass
```
:::
You can then use your custom backend in a {py:class}`.CachedSession` with the `backend` parameter:
```python
>>> session = CachedSession(backend=CustomCache())
```
|
