remctl Protocol R. Allbery Stanford University March 2006 remctl: Authenticated Remote Command Execution Allbery [Page 1] remctl: Authenticated Remote Command Execution March 2006 Abstract This document specifies the remctl wire protocol, used to send commands and arguments to a remote system and receive the results of executing that command. The protocol uses GSS-API and Kerberos v5 for authentication, confidentiality, and integrity protection. Both the current (version 2) protocol and the older version 1 protocol are described. The version 1 protocol should only be implemented for backward compatibility. Table of Contents 1. Basic Packet Format . . . . . . . . . . . . . . . . . . . . . 3 2. Network Protocol (version 2) . . . . . . . . . . . . . . . . . 4 2.1. Session Sequence . . . . . . . . . . . . . . . . . . . . . 4 2.2. Message Format . . . . . . . . . . . . . . . . . . . . . . 5 2.3. Protocol Version Negotiation . . . . . . . . . . . . . . . 5 2.4. MESSAGE_COMMAND . . . . . . . . . . . . . . . . . . . . . 5 2.5. MESSAGE_OUTPUT and MESSAGE_STATUS . . . . . . . . . . . . 6 2.6. MESSAGE_ERROR . . . . . . . . . . . . . . . . . . . . . . 7 2.7. MESSAGE_QUIT . . . . . . . . . . . . . . . . . . . . . . . 8 3. Network Protocol (version 1) . . . . . . . . . . . . . . . . . 9 4. Security Considerations . . . . . . . . . . . . . . . . . . . 11 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 12 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 13 Allbery [Page 2] remctl: Authenticated Remote Command Execution March 2006 1. Basic Packet Format The remctl network protocol consists of data packets sent from a client to a server or a server to a client over a TCP connection. Each packet has the following format: 1 byte flags 4 bytes length data payload The flag byte contains one or more of the following values, combined with binary xor: 0x01 TOKEN_NOOP 0x02 TOKEN_CONTEXT 0x04 TOKEN_DATA 0x08 TOKEN_MIC 0x10 TOKEN_CONTEXT_NEXT 0x20 TOKEN_SEND_MIC 0x40 TOKEN_PROTOCOL Only TOKEN_CONTEXT, TOKEN_CONTEXT_NEXT, TOKEN_DATA, and TOKEN_PROTOCOL are used for version 2 packets. The other flags are used only with the legacy version 1 protocol. The length field is a four-byte length in network byte order, specifying the number of octets in the following data payload. The data payload is empty, the results of gss_accept_sec_context, the results of gss_init_sec_context, or a data payload protected with gss_wrap. The maximum size of the data payload of a remctl packet is constrained by the maximum message size in the GSS-API protocol, which for Kerberos v5 is 64KB. Allbery [Page 3] remctl: Authenticated Remote Command Execution March 2006 2. Network Protocol (version 2) 2.1. Session Sequence A remctl connection is always initiated by a client opening a TCP connection to a server. The protocol then proceeds as follows: 1. Client sends message with an empty payload and flags TOKEN_NOOP, TOKEN_CONTEXT_NEXT, and TOKEN_PROTOCOL (0x51). If the client doesn't include TOKEN_PROTOCOL, it is speaking the version 1 protocol, and the server MUST either drop the connection or fall back to the version 1 protocol. This initial message is useless in a pure version 2 protocol world and is done only for backward compatibility with the version 1 protocol. 2. Client calls gss_init_sec_context and replies with the results and TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42). The client must pass GSS_C_MUTUAL_FLAG, GSS_C_REPLAY_FLAG, GSS_C_CONF_FLAG, and GSS_C_INTEG_FLAG as requested flags. 3. Server replies with the results of gss_accept_sec_context and flags TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42). If the server doesn't include TOKEN_PROTOCOL in the flags, it is speaking the version 1 protocol, and the client MUST either drop the connection or fall back to the version 1 protocol. 4. Client passes data to gss_init_sec_context and replies with the results and TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42). The client must pass GSS_C_MUTUAL_FLAG, GSS_C_REPLAY_FLAG, GSS_C_CONF_FLAG, and GSS_C_INTEG_FLAG as requested flags. 5. Server and client repeat, passing in the payload from the last packet from the other side, for as long as GSS-API indicates that continuation is required. If either side drops TOKEN_PROTOCOL from the flags, it is an considered an error and the connect MUST be dropped. (This could be a down-negotiation attack.) After the establishment of the security context, both client and server MUST confirm that GSS_C_MUTUAL_FLAG, GSS_C_REPLAY_FLAG, GSS_C_CONF_FLAG, and GSS_C_INTEG_FLAG are set in the resulting security context and MUST immediately close the connection if this is not the case. 6. After the security context has been established, the client and server exchange commands and responses as described below. All commands are sent with flags TOKEN_DATA and TOKEN_PROTOCOL (0x44) and the data payload of all packets is protected with gss_wrap. The conf_req_flag parameter of gss_wrap MUST be set to non-zero, requesting both confidentiality and integrity services. Allbery [Page 4] remctl: Authenticated Remote Command Execution March 2006 2.2. Message Format All client and server messages will use the following format inside the data payload. This is the format of the message before passing it to gss_wrap for confidentiality and integrity protection. 1 byte protocol version 1 byte message type The protocol version for the version 2 protocol is 2. (Note that the version 1 protocol does not use this message format, and therefore a protocol version of 1 is invalid.) See below for protocol version negotiation. The message type is one of the following constants: 1 MESSAGE_COMMAND 2 MESSAGE_QUIT 3 MESSAGE_OUTPUT 4 MESSAGE_STATUS 5 MESSAGE_ERROR 6 MESSAGE_VERSION The first two message types are client messages and MUST NOT be sent by the server. The remaining message types are server messages and MUST NOT by sent by the client. 2.3. Protocol Version Negotiation If the server ever receives a message from a client that claims a protocol version higher than the server supports, the server MUST otherwise ignore the contents of the message and SHOULD respond with a message type of MESSAGE_VERSION and the following message payload: 1 byte highest supported version The client MUST then either send only messages supported at that protocol version or lower or send MESSAGE_QUIT and close the connection. 2.4. MESSAGE_COMMAND Most client messages will be of type MESSAGE_COMMAND, which has the following format: Allbery [Page 5] remctl: Authenticated Remote Command Execution March 2006 1 byte keep-alive flag 1 byte continue status 4 bytes number of arguments 4 bytes argument length argument ... If the keep-alive flag is 0, the server SHOULD close the connection after processing the command. If it is 1, the server SHOULD leave the connection open (up to a timeout period) and wait for more commands. This is similar to HTTP keep-alive. If the continue status is 1, it indicates that there is more data coming. The server should accept the data sent, buffer or process it, and wait for more data before responding. If the the continue status is 2, it indicates that this message is logically a part of the previous message (which MUST have had a continue status of 1 or 2) and still has more data coming. If the continue status is 3, it says that this message is logically part of the previous message, like 2, but it also says that this is the end of the command. Number of arguments is a four-byte number in network byte order that gives the total number of command arguments. For each argument, there is then a length and argument data pair, where the length is a four-byte number in network byte order indicating the number of octets of data in the following argument. Both number of arguments and length may be 0. 2.5. MESSAGE_OUTPUT and MESSAGE_STATUS The server response to MESSAGE_COMMAND is zero or more MESSAGE_OUTPUT messages followed by either a MESSAGE_STATUS or a MESSAGE_ERROR response. Each MESSAGE_OUTPUT message has the following format: 1 byte output stream 4 bytes output length output The output stream is either 1 for standard output or 2 for standard error. Output length is a four-byte number in network byte order that specifies the length of the following output data. The MESSAGE_STATUS message has the following format: 1 byte exit status MESSAGE_STATUS indicates the command has finished and returns the final exit stauts of the command. Exit status is 0 for success and Allbery [Page 6] remctl: Authenticated Remote Command Execution March 2006 non-zero for failure, where the meaning of non-zero exit statuses is left to the application to define. (This is identical to a Unix command exit status.) Unless the MESSAGE_COMMAND message from the client had the keep-alive flag set to 1, the server MUST close the network connection immediately after sending the MESSAGE_STATUS response message. 2.6. MESSAGE_ERROR At any point before sending MESSAGE_STATUS, the server may respond with MESSAGE_ERROR if some error occurred. This can be the first response after a MESSAGE_COMMAND, or it may be sent after one or more MESSAGE_OUTPUT messages. The format of MESSAGE_ERROR is as follows: 4 bytes error code 4 bytes message length error message The error code is a four-byte number in network byte order indicating the type of error. The error code may be one of the following values: 1 ERROR_INTERNAL Internal server failure 2 ERROR_BAD_TOKEN Invalid format in token 3 ERROR_UNKNOWN_MESSAGE Unknown message type 4 ERROR_BAD_COMMAND Invalid command format in token 5 ERROR_UNKNOWN_COMMAND Unknown command 6 ERROR_ACCESS Access denied Additional error codes may be added without changing the version of the remctl protocol, so clients MUST accept error codes other than the ones above. The message length is a four-byte number in network byte order that specifies the length in octets of the following error message. The error message is a free-form informational message intended for human consumption and MUST NOT be interpreted by an automated process. Software should instead use the error code. Unless the MESSAGE_COMMAND message from the client had the keep-alive flag set to 1, the server MUST close the network connection immediately after sending the MESSAGE_ERROR response message. Otherwise, the server SHOULD still honor that flag, although the server MAY terminate the connection after an unreasonable number of errors. Allbery [Page 7] remctl: Authenticated Remote Command Execution March 2006 2.7. MESSAGE_QUIT MESSAGE_QUIT is a way of terminating the connection cleanly if the client asked for keep-alive and then decided not to use it. There is no message body. Upon receiving this message, the server MUST immediately close the connection. Allbery [Page 8] remctl: Authenticated Remote Command Execution March 2006 3. Network Protocol (version 1) The old network protocol supported only 64KB of data payload, only a single command and response, and had some additional unnecessary protocol components. It SHOULD NOT be used by clients, but MAY be supported by servers for backward compatibility. It is recognized by the server and client by the lack of TOKEN_PROTOCOL in the flags of the initial security context negotiation. The old protocol always uses the following steps: 1. Client opens TCP connection to server. 2. Client sends message with flags TOKEN_NOOP and TOKEN_CONTEXT_NEXT and an empty payload. 3. Client calls gss_init_sec_context and sends message with the results and flags TOKEN_CONTEXT. 4. Server replies with the results of gss_accept_sec_context and flags TOKEN_CONTEXT. 5. Client calls gss_init_sec_context again with the data from the server and replies with the results and flags TOKEN_CONTEXT. 6. Server and client repeat, passing in the payload from the last packet from the other side, for as long as GSS-API indicates that continuation is required. Each of these packets have only TOKEN_CONTEXT set in the flags. 7. Client sends command with flags TOKEN_DATA and TOKEN_SEND_MIC and the following payload format: four byte number of arguments, and then for each argument, a four byte length and then the argument value. All numbers are in network type order. 8. Server accepts and decrypts data, generates a MIC with gss_get_mic, and sends the MIC back to the client with flags TOKEN_MIC. This is the only packet that isn't encrypted with gss_wrap. Client receives and then SHOULD verify this MIC. 9. Server runs the command, collects the output, and sends the output back with flags TOKEN_DATA and the following payload format: four byte exit status, four byte data length, data. All numbers are in network byte order. The exit status is 0 if there were no errors and non-zero otherwise, where the meaning of non-zero values are defined by the application. Allbery [Page 9] remctl: Authenticated Remote Command Execution March 2006 10. Server and client close connection. Allbery [Page 10] remctl: Authenticated Remote Command Execution March 2006 4. Security Considerations The old protocol doesn't provide integrity protection for the flags, but since it always follows the same fixed sequence of operations, this should pose no security concerns in practice. The new protocol only uses the flag field outside of the encrypted section of the packet for initial negotiation and closes the connection if the flags aren't what was expected (avoiding a down-negotiation attack). In the old protocol, the server calculated and sent a MIC back to the client, which then verified that the command as received by the server was correct. Not only does GSS-API already provide integrity protection, but this verification also happens after the server has already started running the command. It has been dropped in the new protocol. Allbery [Page 11] remctl: Authenticated Remote Command Execution March 2006 Appendix A. Acknowledgements The original remctl protocol design was done by Anton Ushakov, with input from Russ Allbery and Roland Schemers. Thank you to David Hoffman and Mike Newton for their review of the version 2 remctl protocol. Allbery [Page 12] remctl: Authenticated Remote Command Execution March 2006 Author's Address Russ Allbery Stanford University 255 Panama Street, MC 4136 Stanford, CA 94305-4136 US Email: rra@stanford.edu URI: http://www.eyrie.org/~eagle/ Allbery [Page 13]