1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
<html><body>
<style>
body, h1, h2, h3, div, span, p, pre, a {
margin: 0;
padding: 0;
border: 0;
font-weight: inherit;
font-style: inherit;
font-size: 100%;
font-family: inherit;
vertical-align: baseline;
}
body {
font-size: 13px;
padding: 1em;
}
h1 {
font-size: 26px;
margin-bottom: 1em;
}
h2 {
font-size: 24px;
margin-bottom: 1em;
}
h3 {
font-size: 20px;
margin-bottom: 1em;
margin-top: 1em;
}
pre, code {
line-height: 1.5;
font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace;
}
pre {
margin-top: 0.5em;
}
h1, h2, h3, p {
font-family: Arial, sans serif;
}
h1, h2, h3 {
border-bottom: solid #CCC 1px;
}
.toc_element {
margin-top: 0.5em;
}
.firstline {
margin-left: 2 em;
}
.method {
margin-top: 1em;
border: solid 1px #CCC;
padding: 1em;
background: #EEE;
}
.details {
font-weight: bold;
font-size: 14px;
}
</style>
<h1><a href="dataflow_v1b3.html">Dataflow API</a> . <a href="dataflow_v1b3.projects.html">projects</a> . <a href="dataflow_v1b3.projects.locations.html">locations</a> . <a href="dataflow_v1b3.projects.locations.jobs.html">jobs</a> . <a href="dataflow_v1b3.projects.locations.jobs.stages.html">stages</a></h1>
<h2>Instance Methods</h2>
<p class="toc_element">
<code><a href="#close">close()</a></code></p>
<p class="firstline">Close httplib2 connections.</p>
<p class="toc_element">
<code><a href="#getExecutionDetails">getExecutionDetails(projectId, location, jobId, stageId, endTime=None, pageSize=None, pageToken=None, startTime=None, x__xgafv=None)</a></code></p>
<p class="firstline">Request detailed information about the execution status of a stage of the job. EXPERIMENTAL. This API is subject to change or removal without notice.</p>
<p class="toc_element">
<code><a href="#getExecutionDetails_next">getExecutionDetails_next()</a></code></p>
<p class="firstline">Retrieves the next page of results.</p>
<h3>Method Details</h3>
<div class="method">
<code class="details" id="close">close()</code>
<pre>Close httplib2 connections.</pre>
</div>
<div class="method">
<code class="details" id="getExecutionDetails">getExecutionDetails(projectId, location, jobId, stageId, endTime=None, pageSize=None, pageToken=None, startTime=None, x__xgafv=None)</code>
<pre>Request detailed information about the execution status of a stage of the job. EXPERIMENTAL. This API is subject to change or removal without notice.
Args:
projectId: string, A project id. (required)
location: string, The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that contains the job specified by job_id. (required)
jobId: string, The job to get execution details for. (required)
stageId: string, The stage for which to fetch information. (required)
endTime: string, Upper time bound of work items to include, by start time.
pageSize: integer, If specified, determines the maximum number of work items to return. If unspecified, the service may choose an appropriate default, or may return an arbitrarily large number of results.
pageToken: string, If supplied, this should be the value of next_page_token returned by an earlier call. This will cause the next page of results to be returned.
startTime: string, Lower time bound of work items to include, by start time.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # Information about the workers and work items within a stage.
"nextPageToken": "A String", # If present, this response does not contain all requested tasks. To obtain the next page of results, repeat the request with page_token set to this value.
"workers": [ # Workers that have done work on the stage.
{ # Information about a worker
"workItems": [ # Work items processed by this worker, sorted by time.
{ # Information about an individual work item execution.
"attemptId": "A String", # Attempt ID of this work item
"endTime": "A String", # End time of this work item attempt. If the work item is completed, this is the actual end time of the work item. Otherwise, it is the predicted end time.
"metrics": [ # Metrics for this work item.
{ # Describes the state of a metric.
"boundedTrie": "", # Worker-computed aggregate value for the "Trie" aggregation kind. The only possible value type is a BoundedTrieNode. Introduced this field to avoid breaking older SDKs when Dataflow service starts to populate the `bounded_trie` field.
"cumulative": True or False, # True if this metric is reported as the total cumulative aggregate value accumulated since the worker started working on this WorkItem. By default this is false, indicating that this metric is reported as a delta that is not associated with any WorkItem.
"distribution": "", # A struct value describing properties of a distribution of numeric values.
"gauge": "", # A struct value describing properties of a Gauge. Metrics of gauge type show the value of a metric across time, and is aggregated based on the newest value.
"internal": "", # Worker-computed aggregate value for internal use by the Dataflow service.
"kind": "A String", # Metric aggregation kind. The possible metric aggregation kinds are "Sum", "Max", "Min", "Mean", "Set", "And", "Or", and "Distribution". The specified aggregation kind is case-insensitive. If omitted, this is not an aggregated value but instead a single metric sample value.
"meanCount": "", # Worker-computed aggregate value for the "Mean" aggregation kind. This holds the count of the aggregated values and is used in combination with mean_sum above to obtain the actual mean aggregate value. The only possible value type is Long.
"meanSum": "", # Worker-computed aggregate value for the "Mean" aggregation kind. This holds the sum of the aggregated values and is used in combination with mean_count below to obtain the actual mean aggregate value. The only possible value types are Long and Double.
"name": { # Identifies a metric, by describing the source which generated the metric. # Name of the metric.
"context": { # Zero or more labeled fields which identify the part of the job this metric is associated with, such as the name of a step or collection. For example, built-in counters associated with steps will have context['step'] = . Counters associated with PCollections in the SDK will have context['pcollection'] = .
"a_key": "A String",
},
"name": "A String", # Worker-defined metric name.
"origin": "A String", # Origin (namespace) of metric name. May be blank for user-define metrics; will be "dataflow" for metrics defined by the Dataflow service or SDK.
},
"scalar": "", # Worker-computed aggregate value for aggregation kinds "Sum", "Max", "Min", "And", and "Or". The possible value types are Long, Double, and Boolean.
"set": "", # Worker-computed aggregate value for the "Set" aggregation kind. The only possible value type is a list of Values whose type can be Long, Double, String, or BoundedTrie according to the metric's type. All Values in the list must be of the same type.
"trie": "", # Worker-computed aggregate value for the "Trie" aggregation kind. The only possible value type is a BoundedTrieNode.
"updateTime": "A String", # Timestamp associated with the metric value. Optional when workers are reporting work progress; it will be filled in responses from the metrics API.
},
],
"progress": { # Information about the progress of some component of job execution. # Progress of this work item.
"currentProgress": 3.14, # The current progress of the component, in the range [0,1].
"dataPoints": [ # History of progress for the component. Points are sorted by time.
{ # A point in the timeseries.
"time": "A String", # The timestamp of the point.
"value": 3.14, # The value of the point.
},
],
},
"startTime": "A String", # Start time of this work item attempt.
"state": "A String", # State of this work item.
"stragglerInfo": { # Information useful for straggler identification and debugging. # Information about straggler detections for this work item.
"causes": { # The straggler causes, keyed by the string representation of the StragglerCause enum and contains specialized debugging information for each straggler cause.
"a_key": { # Information useful for debugging a straggler. Each type will provide specialized debugging information relevant for a particular cause. The StragglerDebuggingInfo will be 1:1 mapping to the StragglerCause enum.
"hotKey": { # Information useful for debugging a hot key detection. # Hot key debugging details.
"detectedHotKeys": { # Debugging information for each detected hot key. Keyed by a hash of the key.
"a_key": { # Information about a hot key.
"hotKeyAge": "A String", # The age of the hot key measured from when it was first detected.
"key": "A String", # A detected hot key that is causing limited parallelism. This field will be populated only if the following flag is set to true: "--enable_hot_key_logging".
"keyTruncated": True or False, # If true, then the above key is truncated and cannot be deserialized. This occurs if the key above is populated and the key size is >5MB.
},
},
},
},
},
"startTime": "A String", # The time when the work item attempt became a straggler.
},
"taskId": "A String", # Name of this work item.
},
],
"workerName": "A String", # Name of this worker
},
],
}</pre>
</div>
<div class="method">
<code class="details" id="getExecutionDetails_next">getExecutionDetails_next()</code>
<pre>Retrieves the next page of results.
Args:
previous_request: The request for the previous page. (required)
previous_response: The response from the request for the previous page. (required)
Returns:
A request object that you can call 'execute()' on to request the next
page. Returns None if there are no more items in the collection.
</pre>
</div>
</body></html>
|
