1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
Paging and Paginators
=====================
Globus SDK Client objects have paginated methods which return paginators.
A paginated API is one which returns data in multiple API calls.
This is used in cases where the the full set of results is too large to return all
at once, or where getting all results is slow and a few results are wanted faster.
A good example of paginated data would be search results: the first "page"
of data may be the first 10 results, and the next "page" consists of the
next 10 results.
The number of results per call is the page size.
Each page is an API response with a number of results equal to the page size.
Paging in the Globus SDK can be done by iterating over pages (responses) or by
iterating over items (individual results).
Paginators
----------
A :py:class:`~globus_sdk.paging.Paginator` object is an iterable provided by
the Globus SDK.
Paginators support iteration over pages with the method ``pages()`` and iteration
over items with the method ``items()``.
Paginators have fixed parameters which are set when the paginator is created.
Once a method returns a paginator, you don't need to pass it any additional
data -- ``pages()`` or ``items()`` will operate based on the original parameters to
the paginator.
.. _making_paginated_calls:
Making Paginated Calls
----------------------
Globus SDK client objects define paginated variants of methods. The normal
method is said to be "unpaginated", and returns a single page of results.
The paginated variant, prefixed with ``paginated.``, returns a paginated.
For example, :class:`globus_sdk.TransferClient` has a paginated method,
:py:meth:`~globus_sdk.TransferClient.endpoint_search`. Once you have
a client object, calls to the unpaginated method are done like so:
.. code-block:: python
import globus_sdk
# for information on getting an authorizer, see the SDK Tutorial
tc = globus_sdk.TransferClient(authorizer=...)
# unpaginated calls can still return iterable results!
# endpoint_search() returns an iterable response
for endpoint_info in tc.endpoint_search("tutorial"):
print("got endpoint_id:", endpoint_info["id"])
The paginated variant of this same method is accessed nearly identically. But
instead of calling ``endpoint_search(...)``, we'll invoke
``paginated.endpoint_search(...)``.
Here are three variants of code with the same basic effect:
.. code-block:: python
# note the call to `items()` at the end of this line!
for endpoint_info in tc.paginated.endpoint_search("tutorial").items():
print("got endpoint_id:", endpoint_info["id"])
# equivalently, call `pages()` and iterate over the items in each page
for page in tc.paginated.endpoint_search("tutorial").pages():
for endpoint_info in page:
print("got endpoint_id:", endpoint_info["id"])
# iterating on a paginator without calling `pages()` or `items()` is
# equivalent to iterating on `pages()`
for page in tc.paginated.endpoint_search("tutorial"):
for endpoint_info in page:
print("got endpoint_id:", endpoint_info["id"])
Do I need to use pages()? What is it for?
-----------------------------------------
If your use-case is satisfied with ``items()``, then stick with ``items()``!
``pages()`` iteration is important when there is useful data in the page other
than the individual items.
For example,
:meth:`TransferClient.endpoint_search <globus_sdk.TransferClient.endpoint_search>`
returns the total number of results for the search as a field on each page.
Most use-cases can be solved with ``items()``, and ``pages()`` will be
available to you if or when you need it.
Typed Paginators with Paginator.wrap
------------------------------------
This is an alternate syntax for getting a paginated call. It is more verbose,
but preserves type annotation information correctly. It is therefore preferable
for users who want to type-check their code with ``mypy``.
``Paginator.wrap`` converts any client method into a callable which returns a
paginator. Its usage is very similar to the ``.paginated`` syntax.
.. code-block:: python
import globus_sdk
from globus_sdk.paging import Paginator
tc = globus_sdk.TransferClient(...)
# convert `tc.endpoint_search` into a call returning a paginator
paginated_call = Paginator.wrap(tc.endpoint_search)
# now the result is a paginator and we can use `pages()` or `items()` as
# normal
for endpoint_info in paginated_call("tutorial").items():
print("got endpoint_id:", endpoint_info["id"])
However, if using ``mypy`` to run ``reveal_type``, the results of
``tc.paginated.task_successful_transfers`` and
``Paginator.wrap(tc.task_successful_transfers)`` are very different:
.. code-block:: python
# def (task_id: Union[uuid.UUID, builtins.str], *, query_params: Union[builtins.dict[builtins.str, Any], None] =) -> globus_sdk.services.transfer.response.iterable.IterableTransferResponse
reveal_type(tc.task_successful_transfers)
# def [PageT <: globus_sdk.response.GlobusHTTPResponse] (*Any, **Any) -> globus_sdk.paging.base.Paginator[PageT`-1]
reveal_type(tc.paginated.task_successful_transfers)
# def (task_id: Union[uuid.UUID, builtins.str], *, query_params: Union[builtins.dict[builtins.str, Any], None] =) -> globus_sdk.paging.base.Paginator[globus_sdk.services.transfer.response.iterable.IterableTransferResponse*]
reveal_type(Paginator.wrap(tc.task_successful_transfers))
Paginator Types
---------------
``globus_sdk.paging`` defines several paginator classes and methods. For the
most part, you do not need to interact with these classes or methods except
through ``pages()`` or ``items()``.
The ``paging`` subpackage also defines the ``PaginatorTable``, which is used to
define the ``paginated`` attribute on client objects.
.. autofunction:: globus_sdk.paging.has_paginator
.. autoclass:: globus_sdk.paging.Paginator
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.PaginatorTable
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.MarkerPaginator
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.NextTokenPaginator
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.LastKeyPaginator
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.HasNextPaginator
:members:
:show-inheritance:
.. autoclass:: globus_sdk.paging.LimitOffsetTotalPaginator
:members:
:show-inheritance:
|
