1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
remctl Streaming Protocol Draft
Introduction
This is a draft of what would become version four of the remctl
protocol. It adds optional support for bidirectional streaming,
allowing the server and client to exchange arbitrary unsequenced data
while a command is running with coordinated termination of the
command.
This draft should not be used for implementation yet. The details of
the protocol may change substantially before it is added to remctl.
Client library API changes are not discussed in this draft, only
protocol issues.
Streaming Overview
A streaming command is a client-initiated operation. The client
requests streaming when it sends the command. The server either
accepts that request or rejects it. If a particular command requires
streaming, the server may reject a non-streaming version of that
command.
Once the server accepts the streaming command, it runs the command,
passing as arguments the arguments provided in the initial command
token as before. From that point on, all command output is
immediately sent back to the client, and the client may also send
data at any time which is passed as input to the command. This
continues without ordering and sequencing until one side or the other
wants to start shutting things down.
The server can indicate the end of an output stream with a token
dedicated to that purpose. This communicates an EOF on that stream to
the client without terminating the command processing. The client can
similarly tell the server that it is done providing input tokens by
sending an end of stream token to the server, after which it is not
permitted to send any more input tokens until the server indicates the
command is finished.
When the command is over, the server sends a token indicating that
fact. This token will also include the exit status of the command.
This token implies end of stream on all output streams. The server is
solely responsible for determining when the command is over. The most
the client can do is indicate the end of the input stream, but it must
still wait for the server to indicate the end of the command.
If the server indicates the end of the command before the client
closes the input stream, the server will continue to consume and
discard input tokens from the client until the client sends an end of
stream token.
There are therefore two ways for a streaming command to end:
* Client sends end of stream token. Server sends end of streaming
command token, possibly preceeded by an arbitrary number of
streaming output or end of stream tokens.
* Server sends end of streaming command token. Client sends end of
stream token, possibly preceeded by an arbitrary number of streaming
input tokens.
The streaming command is not over until the server has sent an end of
command token and the client has sent an end of input stream token.
Only after both of those tokens have been sent do both sides return to
normal processing of remctl commands.
New Tokens
MESSAGE_COMMAND_STREAM
Identical to MESSAGE_COMMAND but starts a streaming command instead of
a regular command. A separate token is used for this purpose to avoid
changing the format of the MESSAGE_COMMAND token to add an additional
flag.
MESSAGE_STREAM_DATA
Used by both the client and the server to send data during a streaming
command. The format is the same as a MESSAGE_OUTPUT token. For
tokens from the client, the stream is required to be 1. Other streams
are reserved for future versions of the protocol.
MESSAGE_STREAM_END
Indicates an end of data on a stream, equivalent to an EOF condition.
The only content of this token is the stream number. After this token
is sent, no further MESSAGE_STREAM_DATA tokens with that stream number
will be sent as part of this command. When sent by the client, this
indicates the end of streaming data from the client and promises that
the client will send no further data until the server sends the end of
command token.
MESSAGE_COMMAND_END
Indicates the end of a streaming command. The format is identical to
the MESSAGE_STATUS token. A separate token is used rather than
reusing the MESSAGE_STATUS token only because the expected client
response is different (the client may need to send a final
MESSAGE_STREAM_END token) and it may be useful for the client to
easily distinguish.
Implementation Issues
The remctl server will be responsible for translating the network
packets from the client into data on standard input for the command
and output from the command into network packets. To do so safely
without making assumptions about ordering of data, it will need to
poll the running command and the client network connection and do
non-blocking reads and writes. However, it may need to block the
client until the command can process data, or block the command while
waiting for client data.
I think the best way of handling this is to add buffering code to the
server with an upper limit on the allowed amount of data buffering.
The remctl server will then accept data from either the command or the
client up to the buffer size even if it can't yet write that data to
its destination. Once the buffer is full, the server will stop
polling that connection until the buffer has been drained, forcing the
other end of that connection to block.
One challenge for such buffering is that the amount of data contained
in a single GSS-API token may vary, and ideally the server should not
partially consume that data. If the server accepts a token, it should
read the entire token. This is not strictly necessary, but I think
partially consuming an output token would significantly increase the
risk of deadlock.
Limitations
This protocol does not address deadlock and instead requires that the
client and server be aware of deadlock issues and find ways of
avoiding deadlock within the implementation of a streaming command.
License
Copyright 2008, 2011
The Board of Trustees of the Leland Stanford Junior University
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved. This file is offered as-is,
without any warranty.
SPDX-License-Identifier: FSFAP
|
