1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
|
.. _sync:
########################
Synchronizer
########################
Synchronizer is a powerful utility with functionality of a basic backup application.
It is able to copy entire folders into the cloud and back to a local drive
or even between two cloud buckets, providing retention policies and many other options.
The **high performance** of sync is credited to parallelization of:
* listing local directory contents
* listing bucket contents
* uploads
* downloads
Synchronizer spawns threads to perform the operations listed above in parallel to shorten
the backup window to a minimum.
Sync Options
============
Following are the important optional arguments that can be provided while initializing `Synchronizer` class.
* ``compare_version_mode``: When comparing the source and destination files for finding whether to replace them or not, `compare_version_mode` can be passed to specify the mode of comparison. For possible values see :class:`b2sdk.v2.CompareVersionMode`. Default value is :py:attr:`b2sdk.v2.CompareVersionMode.MODTIME`
* ``compare_threshold``: It's the minimum size (in bytes)/modification time (in seconds) difference between source and destination files before we assume that it is new and replace.
* ``newer_file_mode``: To identify whether to skip or replace if source is older. For possible values see :class:`b2sdk.v2.NewerFileSyncMode`. If you don't specify this the sync will raise :class:`b2sdk.v2.exception.DestFileNewer` in case any of the source file is older than destination.
* ``keep_days_or_delete``: specify policy to keep or delete older files. For possible values see :class:`b2sdk.v2.KeepOrDeleteMode`. Default is `DO_NOTHING`.
* ``keep_days``: if `keep_days_or_delete` is :py:attr:`b2sdk.v2.KeepOrDeleteMode.KEEP_BEFORE_DELETE` then this specifies for how many days should we keep.
.. code-block:: python
>>> from b2sdk.v2 import ScanPoliciesManager
>>> from b2sdk.v2 import parse_folder
>>> from b2sdk.v2 import Synchronizer, SyncReport
>>> from b2sdk.v2 import KeepOrDeleteMode, CompareVersionMode, NewerFileSyncMode
>>> import time
>>> import sys
>>> source = '/home/user1/b2_example'
>>> destination = 'b2://example-mybucket-b2'
>>> source = parse_folder(source, b2_api)
>>> destination = parse_folder(destination, b2_api)
>>> policies_manager = ScanPoliciesManager(exclude_all_symlinks=True)
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.SIZE,
compare_threshold=10,
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.KEEP_BEFORE_DELETE,
keep_days=10,
)
We have a file (hello.txt) which is present in destination but not on source (my local),
so it will be deleted and since our mode is to keep the delete file,
it will be hidden for 10 days in bucket.
.. code-block:: python
>>> no_progress = False
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
upload f1.txt
delete hello.txt (old version)
hide hello.txt
We changed f1.txt and added 1 byte. Since our compare_threshold is 10, it will not do anything.
.. code-block:: python
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
We changed f1.txt and added more than 10 bytes.
Since our compare_threshold is 10, it will replace the file at destination folder.
.. code-block:: python
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
upload f1.txt
Let's just delete the file and not keep - keep_days_or_delete = DELETE
You can avoid passing keep_days argument in this case because it will be ignored anyways
.. code-block:: python
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.SIZE,
compare_threshold=10, # in bytes
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
delete f1.txt
delete f1.txt (old version)
delete hello.txt (old version)
upload f2.txt
delete hello.txt (hide marker)
As you can see, it deleted f1.txt and it's older versions (no hide this time)
and deleted hello.txt also because now we don't want the file anymore.
also, we added another file f2.txt which gets uploaded.
Now we changed newer_file_mode to SKIP and compare_version_mode to MODTIME.
also uploaded a new version of f2.txt to bucket using B2 web.
.. code-block:: python
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.MODTIME,
compare_threshold=10, # in seconds
newer_file_mode=NewerFileSyncMode.SKIP,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
As expected, nothing happened, it found a file that was older at source
but did not do anything because we skipped.
Now we changed newer_file_mode again to REPLACE and
also uploaded a new version of f2.txt to bucket using B2 web.
.. code-block:: python
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.MODTIME,
compare_threshold=10,
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
delete f2.txt (old version)
upload f2.txt
Handling encryption
-------------------
The `Synchronizer` object may need `EncryptionSetting` instances to perform downloads and copies. For this reason, the
`sync_folder` method accepts an `EncryptionSettingsProvider`, see :ref:`server_side_encryption` for further explanation
and :ref:`encryption_provider` for public API.
Public API classes
==================
.. autoclass:: b2sdk.v2.ScanPoliciesManager()
:inherited-members:
:special-members: __init__
:members:
.. autoclass:: b2sdk.v2.Synchronizer()
:inherited-members:
:special-members: __init__
:members:
.. autoclass:: b2sdk.v2.SyncReport()
:inherited-members:
:special-members: __init__
:members:
.. _encryption_provider:
Sync Encryption Settings Providers
==================================
.. autoclass:: b2sdk.v2.AbstractSyncEncryptionSettingsProvider()
:inherited-members:
:members:
.. autoclass:: b2sdk.v2.ServerDefaultSyncEncryptionSettingsProvider()
:no-members:
.. autoclass:: b2sdk.v2.BasicSyncEncryptionSettingsProvider()
:special-members: __init__
:no-members:
|
