===================================================================== BELOW IS A RUNNING STREAM OF NOTES WHICH DON'T HAVE TO MAKE ANY SENSE ===================================================================== I/O is serialized with timing data IIIIIIiiIIIIIIIiOOOOOoooOOOOooIIIIIiiIIII | || || || | needs splitting the timing: both pieces start at the same time, but different messages, keeping order needs tracking absolute time of each piece to put in the next message can't format timing data right away, only when writing OR store last whitespace time, input/output position >25 IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII ^ ^ We'd better not cut anything for the start write window to log write output to log write input to log read window from terminal write window to slave write output to terminal write input to slave read input from terminal read output from slave why do we even need to be terminated by a signal? 1. because we need to handle out-of-band SIGWINCH when waiting for output and waiting for input to be consumed 2. because we need to handle SIGINT, SIGTERM and SIGHUP timely to restore terminal, instead of simply dying Can we suspend I/O transfer for writing the log entry? We should - this provides flow control However, if we delay data transfer, SIGWINCH might go out of sync with the data. So some data in the kernel buffer and pending write on the slave will be counting on one window size, while the actual window size will be different. Do we really care about window size? Really? Yes we do care about transferring it, so we might as well record it. *** Delay with the transfer should coincide with delay in the log. *** But can it? We have no idea if the slave has read all of its input buffers when it gets SIGWINCH. So it will be out of sync. Still screen handles this anyway. Can we do all this without terminating on a signal? With poll even on write? Would it work on terminals? WE NEED TO HANDLE INTERRUPTED I/O BECAUSE WE NEED TO HANDLE LATENCY LIMIT TIMER Log is always in UTF-8 Convert any other encoding to UTF-8 Write character counts in timing Keep incomplete characters to the next write - don't flush them? Or assume characters are always complete? THAT IS NOT TRUE! OR WAIT! Perhaps it's our code breaking on incomplete characters? VERIFY THIS! Anyway, we can't be sure, we can have a buffer size which won't fit the characters. We have to assume I/O is encoded somehow and for UTF-8 it means multibyte characters. Valid UTF-8 is a smaller subset of arbitrary binary stream. Thus we can't represent everything that's being pushed to the terminal. However, if we need to have our terminal logs searcheable, we need that data in a format that ElasticSearch can understand. What if we add two more timing record types: binary input and binary output? Have them encoded in base64? Write both streams while delay is under 0, then flush their timing What to do with interleaved binary data? We should keep the pieces before flushing, or rather we should flush them as we switch forks. Delay is put in by code above streams. Data lengths are put in by streams. So when a new piece of input comes in, check if delay is still within zero. If it is more than zero: Ask both streams to flush timing, giving them control of timing buffer, then append delay. Hand off the newly received data to the corresponding stream giving it control of remaining length and timing buffer What should the stream reserve space for? Only "cutting" or "flushing" as well? Only cutting, flushing will need more space. if invalid byte encountered if sequence is empty write one byte from input buffer to binary fork else write binary sequence from utf8 buffer else if sequence is incomplete add another byte if not added break else write text sequence from utf8 buffer sequence incomplete and no invalid byte encountered: collecting sequence or invalid byte encountered: writing non-text sequence: either from utf8 buffer or one byte from the input buffer or sequence complete: writing text sequence from utf8 buffer [ ] [ ] [ ] [ ] [ ] ended valid 0 0 sequence in progress 0 1 impossible 1 0 sequence terminated by invalid byte 1 1 sequence completed ended complete 0 0 sequence in progress 0 0 impossible 1 0 seqence terminated by invalid byte 1 1 sequence completed sequence of bytes / \ / \ text binary while true if the sequence has ended If it is valid UTF-8 finish writing it to text fork else finish writing it to binary fork get a sequence the sequence: binary or text length how much is written What if we rename the current utf-8 object to a sequence object? Then it doesn't have to contain only UTF-8 bytes. But would that be useful? We only need to keep incomplete sequences between calls because UTF-8 characters are multi-byte and need to be written atomically. What if we extend our sequence object to keeping words atomically? We'll need to add word length limit as well. Otherwise the rest of the machinery will be there already. So, an incomplete text sequence can be kept between write calls, and can be force-flushed. ------ Can we define the sequence? A sequence can be started. A sequence can be ended. A sequence can be: binary, partial character (binary when incomplete), text (partial character when incomplete?), word (text when incomplete A byte can belong to word, text, binary or unknown (partial character) sequence. Sequence: __________________ character / _____ word / / d0 b0 d0 b1 d0 b2 0a 32 a word is a character sequence a character sequence is a binary sequence ------ __________________ character / ______ word / / d0 b0 d0 b1 d0 b2 0a do we flush incomplete words? do we flush incomplete characters? store last position of a complete sequence type? ff binary started character not started word not started d0 binary started character started word not started Oa binary started character complete word not started d0 b0 binary started character complete word started d0 b0 d0 binary started character started word started d0 b0 d0 b1 d0 b2 binary started character complete word started d0 b0 d0 b1 d0 b2 binary a0 character complete word ended A binary sequence is a sequence of any bytes Non-character sequence is a binary sequence that doesn't contain a valid UTF-8 character. Character sequence is a binary sequence that is a UTF-8 character. Text sequence is a sequence of character sequences. Word sequence is a text sequence that doesn't contain word-break characters. -------------------------- How do we want to use it? forever until we have a complete sequence add a byte to sequence write the sequence byte character atom sequence sequence byte --- Non-character --- Character --- Non-word --- Word So, there are two dimensions: sequencing and alternating. We can start with the first (byte) sequencing level and leave the second (character) sequencing level for later. A two level solution can be like this: until we have a complete character sequence forever until the byte sequence ends add a byte to the sequence if it is a character add the character to the character sequence break write the bytes to binary fork empty the byte sequence write the character sequence to the text fork empty the character sequence So a sequence object provides binary output: valid/non-valid, but only if the sequence has ended. top level stream writing functions return amount of data actually written they write to their forks and also to global metadata inside they need to do some atomic writes, sometimes to both forks at once These second level functions can also accept the location for the remaining length and write to the buffers opportunistically. However, when atomic writing to a few buffers is necessary, we can have intermediate storage for meta pointer and remaining length to ensure that we can roll-back the write accounting So on the second level at the moment we're writing a character, which can be valid or invalid. A valid character is simply an atomic write to the text fork. An invalid character is an atomic write both to text and to binary fork. Either of them can write meta records when switching between valid and invalid runs. These writes should also account for the necessary meta records, but not write them. If the meta record cannot be fit the writing should say that no data was written. We need these operations: streams: tlog_stream_cut write withheld incomplete characters tlog_stream_flush write meta and reset runs io: tlog_io_cut write withheld incomplete stream characters tlog_io_flush flush streams sink: tlog_sink_io_cut write withheld incomplete I/O characters tlog_sink_io_flush flush I/O and write it as a message Abstract transaction: change some fields if some action changed rollback changes abort Should we implement single transaction interface? I.e. if a transaction started on upper levels, we don't start another one? So there is no sub-transaction support? Would that be right? I.e. would there ever be a transaction trying some sub-transaction variants, aborting them and then settling on one variant, proceeding, and then aborting on a later action? Nevertheless, can't we do this with a single transaction storage anyway? An object should save only what makes sense to rollback according to all the possible operations. So, a tlog_io_write can look like this: tlog_io_trx_start(io) tlog_stream_trx_start(input) tlog_stream_trx_start(output) try writing delay to timing if failed: tlog_io_trx_abort(io) tlog_stream_trx_abort(input) tlog_stream_trx_abort(output) try writing stream if failed: tlog_io_trx_abort(io) tlog_stream_trx_abort(input) tlog_stream_trx_abort(output) tlog_io_trx_commit(io) tlog_stream_trx_commit(input) tlog_stream_trx_commit(output) Copying the state should happen only at the top level transaction start. Restoring the state should happen only at the top level transaction abort. But this way we can't do sub-transactions. Yeah, and anyway, if we're doing sub-transactions we're supposed to have another set of fields saved, so we can roll-back properly. So, let's not try to support sub-transactions for now. Let's just assume that only top level transactions save/rollback. So, we'll need a transaction nesting counter. Any level can call start increasing it, or abort/commit decreasing it. But only the top level will actually act on the decision (abort/commit). Perhaps we can come up with a facility to make adding transaction support to an object easier. Ah, it seems we need sub-transactions for writing characters / binary pieces atomically in streams. Or do we? Yeah, pretty much. We abort character / binary piece writing, but we commit pieces written before these. I.e. we're not aborting the whole stack. Another challenge: pmeta. It's not a part of stream object, it's passed into functions, all of them. But perhaps it should be? Then we can save it in a transaction as well. The same about prem. Shan't we try implementing that first? Hmm. It's not like they ever change, after all. Ah, nope. While we can have these pointers kept in the stream objects for ease of calling the object functions, we can't snapshot them as part of streams. They're a part of I/O object and need to be preserved as part of its own transactions, in tlog_io_write. Alright, then the needed depth of the transaction stack is determined by the upper layers. If they need a copy for their transactions, they're supposed to allocate and own it. We shouldn't need to allocate anything dynamically. Shall they be stored on the stack, then? So let's store them on the stack, but copy there only if we need a a new, or sub-transaction. Let's define transaction storage type for each object type, which can simply be the object type. Also let's define backup/restore operations for each object, which can simply be a memcpy. Still, that storage type should be an aggregate for the nested objects. If a function can't fail, it shouldn't accept a transaction pointer. If a function doesn't need to make backups and rollbacks, it shouldn't accept a transaction pointer. However, every type expected to support transactions should define backup/restore functions. On a closer examination and attempting to do it, we'd better not pass pmeta and prem to streams at creation time. This turns out to be more obscure and harder to reason about. If we remove separate window message, then we don't need separate packet type - just supply window size each time, make the sink code or source user compare window sizes and detect changes. This can make the code much simpler. OTOH, I/O data and window changes *do* come separately and such scheme would require the user to cut the I/O themselves and take care of comparing window sizes all the time. So, for a stream of events, having separate packet types, as in "separate even types", makes sense. We can still have one message type for the JSON sink. We still need to preserve window size changes, likely. Can we put them inside timing, as events? What would happen with the player, if there is no window size at the start? How would the *partial* stream matches work?! Shall we simply produce messages with empty I/O and new window sizes? These would have timestamps. We shouldn't send such messages immediately, but only when told to flush, or when we run out of I/O size limit. If window change comes before I/O, the written message should have the window change time position, and timing would have a delay record coming first - this needs a tlog_io object change, so it starts when it's told it starts, not when first I/O comes in. I.e. we can say that window change happens always at message start. No, this is wrong. This means that we won't be able to merge messages across window changes. To satisfy both the need to merge messages and the need to process partial streams, window changes should be able to appear any place in the message (i.e. in "timing"), and each message should have a window size record at the start. Argh. BTW, how are we even going to merge particular stream chunks with audit messages for a specific session? Could Kibana plugin request audit data by itself? Not likely, and it wouldn't be right, anyway. We can tell users: search your messages and then add 'OR something that matches a bunch of relevant audit messages' to the query, and the playback visualisation would filter them out by sessionid and time ranges. That's ugly. Do we have alternatives? I think it is time to actually try making a visualisation. All the objects manipulated by a function are saved on the entrance to the stack frame with depth zero, and restored on return. We need backup/restore so we don't present intermediate state, but rather completely changed or completely unchanged - What do you need sub-transactions for? - For moving in discrete states within another transaction - Why do you need tracking which objects are saved and which are not? - So cyclical ownerships can be saved. So that we can start a transaction at either of these levels: => A -. A -. | | | | B | AND B | | | | | C -' => C -' Each function down the stack needs to store the objects explicitly, in case it is called at the root of the transaction. However, if the function doesn't leave any state inconsistent it doesn't need to accept the transaction argument. So we have object saving for root transaction, then stack increment, then processing, then stack decrement, then object restoration for root transaction. --- We write pieces of input or output with time attached. We write window sizes with time attached We go over each byte sequence of submitted I/O and decide if it's a valid character or not. If it's a valid character, we encode it into text. Otherwise we encode it into binary. We can attach timestamp to each piece of data. Each piece of data has its timestamp. We can't encode data until we know it's a valid character or not. The decision to write the data happens only then, so the decision to terminate a meta record happens only then and the decision to insert a delay record can only happen after that. So how about we have forks again? Can we remove the need for transactions? We would need to be able to back out of writing a delay and subsequent records on flush. To be precise, we would need to be able to play these forward and decide if we want them to do or not. That would mean doing everything twice for all operations, whereas we only need to back out once in a while. What is more complicated? Transactions or backing out? Transactions are easier to think about. Otherwise we would need to pass our temporary copies to all the arbitrary function depths and care about copying/applying them properly in each stack frame. That's what transactions could do for us anyway. --- So if we want to support multiple object roots per transaction we need to separate transaction beginning/ending from transaction object saving/restoring. Also, to be reliable we'll need to remove the need to specify objects to restore for transaction abortion. We can specify objects to backup/restore via initialization of an array variable in a macro. We can then backup these objects in transaction begin and restore them in transaction abort. Can we use the stack for indirect transaction store? First we need to describe object's transaction interface. store_size The size of the object's transaction store pmask Location of the object's transaction mask xfr Transaction data transfer function So each function frame would contain an array describing which objects should be backed up/restored, which would be built from both static, direct object information and dynamic, indirect object information. Or perhaps we should simply resort to dynamic information always. Even with dynamic information, first level can be allocated on the stack and lower levels can do heap allocation for indirect objects, if necessary. We can then remove transaction objects and functions from headers and put it into object files, making them tidier. --- What if we allocate transaction storage within the objects themselves according to the maximum transaction depth? This way we won't have to mess with allocation/deallocation at transaction backup/restore/cleanup time! Then the transaction interface becomes: xfr Transaction data transfer function If we manage to separate them, then it becomes: backup Transaction backup function restore Transaction restore function ---- iface act mask store root header trx - transaction management function TRX_BEGIN TRX_ABORT TRX_COMMIT trx_misc - miscellaneous transaction functions and macros trx_state - transaction state passed between functions trx_state_get_depth trx_iface - object's interface definition trx_basic - macros for basic interface implementation trx_basic_iface - define interface trx_basic_act - define acting function trx_basic_store - basic store definition macros trx_basic_store_sig trx_basic_members - define fields for object structure trx_iface - interface trx_mask - mask trx_store - store list ---- 1 3 input >-------------+-------------------------> input | 3 | 1 output <----------- | ------+-----------------< output | | | | 2 | 2 | \|/ \|/ when to log I/O * after receiving, before sending single timestamp more likely to get out * after sending timestamps may need to be set to the time after sending less likely to get out Should the timestamp be the time we received data, or the time we delivered the data? Neither is guaranteed to be exact - network latency can happen on either side, so no advantage here. So, after receiving and before sending then. Shall we allow interruption of log writing by signal handlers? What shall we do when that happens? SIGTERM/INT/HUP we don't need to log anymore, since data will not be delivered anyway don't write data SIGALRM flush log - that can be done as well after writing, since we're already trying to deliver the log as hard as we can. SIGWINCH that can as well be handled after we finish log writing, if a few resizes are missed, it's not a big deal - delivering the last resize before passing any data is what's important. So if we do partial log writing it is only to handle aborting log writing and not writing extra. This can be useful with large buffers, slow logging backend and overly chatty user application. Shall we mix logging and writing? I.e. deliver what we logged if logging is interrupted and then try to deliver logs again? Shall we actually deliver what we read so far before acting on SIGTERM/INT/HUP? Or shall we simply exit ASAP? It's likely the sysadmins would appreciate the latter more. As well as not finishing logging of undelivered data. With that in mind, there's no use to proceed to writing partially logged data. Because otherwise, SIGWINCH... Wait. What happens when we delay the data in transit and the window gets resized? The data that we've read already was expecting the old window size, we can't push it back so it needs to be delivered anyway. We'll read next portion only after we've delivered the resize. So it's fine. So, going back, we don't need to do anything with SIGWINCH until we're ready to read some more and that's after we've logged the data. Wait. What shall we write first: the SIGWINCH, or the input we've read already? SIGWINCH would be better, as otherwise delivering input first can make the application fill the buffers with response formatted for the old window size. Once again, it doesn't matter what we'll deliver first: the output we read already or the SIGWINCH we received. So it seems it is better to have I/O log writing interrupted to log and deliver window resizes before resuming logging and delivery of I/O. Thus our TTY source needs to drop support for reading window sizes as they need to have higher priority. Since we need to record timing of our window resize deliveries outside the source, we better remove timestamping from the source. Thus we'll have timing neither in source nor in sink. However, sink can have window writing. OTOH, input sent before resize was done with the expectation of the old size. E.g. a command to go to the bottom of the window in vim (L) would need to move the cursor to the old window bottom, not to the new one. So, for the sake of simplicity and, it seems, for the sake of consistency it is better to send the input received before SIGWINCH before sending the new window size. Thus we can preserve the ordering and keep window size retrieval in the source. As to writing as soon as the (interrupted) data logging is complete (i.e. logging/writing interleaving), it might improve interactivity and make the program easier/faster to kill. --- Configuration parameters Log latency (seconds) Maximum message payload size (bytes) Shell to start (path) Log target (file / syslog / journald?) Log target options: file: path syslog: facility level Recording warning to output on startup Command line-specific parameters: argv[0] starting with '-' - login shell - passed the same way to the shell '-c' - command to execute - passed the same way to the shell -?tlog-rec-(.*) - shell, perhaps Mandatory environment parameters: shell It would be good to have all configuration options available via all sources: configuration file command line environment We can parse a particular environment variable value as an option string. We can map command-line options to JSON configuration structure, or the other way around. We can make a mechanism to parse all of these into a common format (JSON?) and merging them according to the priority. E.g. (last overrides first): configuration, environment, command-line. We'll also need some sort of schema verification for configuration. Then we can make a module taking that merged configuration and running the program. What should be the common format? Choices: Custom structure, generic perhaps Hmm, not really, no. JSON merging: Generic repeated walking and merging according to priority. validation: Random access with accumulated state. conversion: from ARGV list: Split into levels on dashes within option names, perhaps? Hmm, options often have very short names, which will have to be normalized. Some sort of mapping, then? How to tell the user about that mapping? Build-time documentation generation? M4, ha-ha? environment: if command-line options, convert to ARGV list - need to handle shell quoting otherwise JSON ARGV list merging: Just concatenate lists from various sources according to priority. validation: Regular getopt loop with parsing and accumulated state conversion: from JSON: Walk and convert automatically, say by concatenating with dashes between levels. Still needs schema to provide JSON-oriented error messages (i.e. line number, field reference). from environment: Split into ARGV list - need to handle shell quoting Values of specific parameters and permitted parameter set have to be validated upon parsing the source format. The overall configuration, required parameter set should be validated only after merging. The (missing or invalid) parameters should be referred by a name common to the configuration sources. So, we can have an m4 file describing containers and parameters, then generate documentation from that, then generate JSON validation, option reading, environment variable reading from that. ----- program name - argv[0] This is not configuration, i.e. not something that the user would like to specify, that's why we don't want it in our options or config file. OTOH, a system administrator or a distro package maintainer might like to change it. Still he's not a user and this is done outside the normal configuration path. Shall we perhaps transfer it out-of-band? But it's sooo convenient to transfer it via JSON. shell - cfg, env, argv[0], -s/--shell option login flag - argv[0], -l/--login option positional arguments argv[x..] If we take away the program name, then we can split those into three origins: name - login flag, shell opts - login flag, shell args - arguments only Then the login flag and shell would be documented properly and overridable by the command-line options. ----- We need to signal the sink to terminate cached characters. That needs to be done with a timestamp. Since in tlog-rec we don't operate on timestamps anymore, getting them with packets from the source, it would be good to have these terminating packets from the source too. The most logical way would be to send these packets when the corresponding streams in the source are ending. There's the problem of the TTY source stream end - it usually happens when either the child or the terminal go away, which are reported as errors, and the source reports all errors to the caller. To return the void packet the source would have to interpret the errors, which might be different between TTYs, PTYs, pipes and files, and interpreting them incorrectly might hide a problem. Let's see what we can get: EBADF - can also be returned if the FD was closed, thus interpreting it as EOF can hide a programming mistake EIO - can also occur on low level I/O error, thus hiding a serious hardware issue We have the void packet signifying the end of the whole source stream now. And it can't have a timestamp as of now. Let's consider the following options: Add a timestamp to the void packet We have to have a completely meaningless packet type to denote an empty, meaningless packet - No Add another type of packet signifying the end of the whole source stream When the source would send such a packet? It can send it only when both input and output are closed. Can we put the function of keeping track of the stream status on the source? Seems alright since it manages and encapsulates them. Add another type of packet allowing specifying the end of input/output stream separately Now what about the potential window size stream? Shall we add that too? Wouldn't it make the whole thing too complicated? Add a new meaning to the I/O packet type which signifies the end of stream (e.g. zero length data) Now, what kind of timestamp would other types of sources specify? E.g. ElasticSearch? It doesn't have anywhere to pull the EOF timestamp from! We shall rethink this. There shouldn't be a timestamp for EOF and it can still be signified by a void packet. We have to accept the fact that sometimes we need to write data that was received before the latest data written to the chunk, and that there can't always be a timestamp associated with that event. E.g. when converting from one storage format to another (e.g. sudo session recording to JSON log and having incomplete characters at the end). To properly handle that we would need to only allow encoding the packet stream as a whole and that's not what we can do. Which timestamp should we have for such data, then? Well, there's nothing left but to use the latest timestamp written to the timing (metadata). Now, how do we signal the cut-off? Shall we keep the function, since we don't need the timestamp anymore, or shall we move to using the void packet? Void packet Symmetrical to EOF from source. Somewhat obscure code. All the non-UTF-8 sinks would need to explicitly ignore the void packet. More code to write, a little code to remove. Function Separate function which complicates the interface and is only needed by one sink so far. Clearer code. All the non-UTF-8 sinks can just not define the cutting function. No code to write. Let's leave it as function for now. Let's have the JSON chunk use the latest timestamp written to the timing for cutting. Wait. We can't use the latest timestamp in the chunk. Because it is possible that a stream will be written only the incomplete character and there will be no timestamp reported to the chunk at all until we actually "cut" the stream. Therefore we must allow the chunk to receive timestamps in the past and assume the latest timestamp in these cases. The streams, though, will still be able to enforce the time succession. So, streams will have to report the last received data timestamp and if chunk sees them as too old, it will use the latest time instead. ----- Do we create a PTY or a pair of pipes for the child? First, effects: If a child uses PTY and we are using non-TTY FDs, then: Child input won't contain expected terminal control symbols Child output will contain unexpected terminal control symbols If a child uses pipes and we are using TTY FDs, then: Child input will contain unexpected terminal control symbols Child output won't contain expected terminal control symbols Next, decision logic: SSH allocates PTY if input is TTY, or if forced What would user expect? Tlog-rec is supposed to start a shell, either interactively or not. When started as the login shell (the primary use case), tlog-rec will either get all descriptors tied to a terminal or all to a non-terminal. What's the use case for starting as a non-login shell? Testing? We won't be outputting to an actual terminal, but will be capturing everything, so we can automate. We will test whatever the user needs. - Not applicable Just the shell being started again, where we need to simply exec the actual user shell. -Not applicable. So what kind of logic can we have? If at least one of in/out/err is a terminal, spawn PTY? If all of in/out/err are terminals, spawn PTY? Which one would be easier for testing? If we're running under expect as we plan, then all FDs will be TTYs. If we want to test non-TTY mode, then we will spawn tlog-rec under pipes, perhaps from shell, Python, or Lua? - Not applicable Which would be more useful? Considering the above the difference is marginal. Which would be easier to implement? If at least one (stdin?) is not a TTY, don't spawn a PTY. However, considering we need to handle open/closed descriptor state and propagate it to the child anyway we can as well check all of the descriptors anyway. The POSIX shell standard says that the shell is considered interactive if both its stdin and stderr are connected to a terminal. We can employ the same logic for deciding whether to spawn the PTY or not. I.e. spawn the PTY only if both stdin and stderr are terminals, ignore stdout. No, tlog-rec is not a shell and in fact any program can be ran under tlog-rec. So we need to employ our own logic. Leaving it at "if any FD is a TTY, spawn PTY" can work as well for now. Do we create two pipes, one, or none? We create two pipes to match the PTY case. We send both child stdout and stderr to our stdout. What do we do when any of stdin/stdout/stderr are closed at the start? We ought to pass the closed/opened state to the child. If our stdout is closed, the child's stdout and stderr are closed. If our stdin is closed, the child's stdin is closed. The child is not affected by our stderr state. We need to ensure that the closed FD is not taken over by another FD. So, for example, error messages directed to the initially closed stderr don't get written somewhere else. How? Perhaps by opening /dev/null? What do we do when only input or output is closed? Perhaps switch to pipes and propagate the status? What do we do when we get an EOF on any of the FDs during operation? We continue the transfer until all FDs get EOF. ---- How can we implement recording and playback to/from journal? We can add a journal sink, which, in addition to the JSON message, can log additional fields, duplicating the data already in JSON, letting us filter journal messages by them. It remains to be seen how that will perform. We can let users configure if tlog-rec should put extra fields into journal, or just the message. The journal source can then accept an arbitrary filter string, similar to what journalctl accepts to filter out the required session. One problem can be specifying the time from which we should play. The journal API provides sd_journal_seek_realtime_usec and sd_journal_seek_monotonic_usec, which we can perhaps use by taking a separate journal source option. --- So, rate limiting is not letting us log more than a specified amount of I/O bytes per time unit. Let's say per hour. The way we calculate that could be having an averaging time window, or just calculate from the start of the recording. current_rate = bytes / hours How would we choose what to drop, though? Should we just do if current_rate > target_rate then drop else pass What should be the quantum of dropping, then? A packet? ---------------- Leaky bucket for raw data, both input and output Either drop or hold the packets Consider providing maximum packet size configuration Consider doing it for encoded data later Add configuration parameter/option for bytes per second ---------------- The pass decision is for a packet. We can simply say every piece of it is written when dropping. We can simply wait on the first piece when delaying. ---------------- We need to either assume that packet timestamps are in local time, or take our own time samples. Because we cannot advance abstract packet recording time, if it is not local time. For that matter, to make the rate limit sink more universal we need to ignore packet timestamps and take our own timestamps. --------------- So, we don't need to know local time if we are just dropping packets, we only need packet timestamps. We can drop packets based on their timestamps, or based on times when they're written. We can only delay packets based on writing timestamps. Having limit based on packet timestamps would be useful for testing. We would need rate limit set on the log sink, right? Right. What if we give the rate limit sink some time source function instead of letting it get its own time? What timestamps should delayed packets have? Original ones, or delayed? Should the recording reflect timestamps the packets were output by the recorded program and input by the user, or the timestamps they were delivered at? If we follow the idea of "recording what users saw on the screen", then it should be delivery time. Ha! Should we actually rate-limit the source instead? Then we can adjust packet timestamps and take time samples freely! For the start we can just give the rate-limit source a clock ID, but in the future we can pass it a function which would generate timestamps, and let us do testing at any actual speed required. We would also have no problem with packet fragments, as sources don't have any. The rate limit sink will have to assume that the time source for packets is the same time source it was given. We can later introduce a notion of "abstract time", which would have timestamp retrieval and time-advancing (sleeping) functions. We can have one specific time for clock_gettime/nanosleep, and one for testing time. Right. We can't rate-limit source, because we will be dropping data sent to the user terminal and entered by the user. We have to rate-limit the log sink. Would it be right to change the timestamp of packets sent to the log? No it likely wouldn't. Should we have delay at the TTY source, but dropping at the log sink? Even if we had delay at the TTY source, would it be right to change packet timestamp? For that matter, what the packet timestamp should be? Time of arrival, or time of departure? And should input and output rather be both measured at the tlog<->program interface, and not input at tty<->tlog and output at tlog<->program interfaces? Or both at tty<->tlog interface, as the user saw it? From the point of correlating with other logs, the right way would be to record time at the tlog<->program interface, as that would be closer to other events in the system. The tty<->tlog interface is just a display, after all. What about window resizes? That should be the same as input. For that matter, if we did throttling (delaying) at the source, then only input and window timestamps should be updated, and we wouldn't really be able to use packet timestamps for rate control. However, since we don't let any packets pass until we log them, and we can't really control logging latency, we can only log source timestamps. But should we log only before passing the I/O? If not, then when should we log it? Should we log it after passing a single packet? Why a single packet then? Why not two? What if it's a single *big* packet? No, we should log *before* passing the packet, or rather we log any data, before passing it on. To avoid the mess of timestamps, we can just assume that timestamp is opaque packet data, and deal only with rate related to the provided clock. Then we can implement rate-limit sink, simply, covering all the features. What if we're asked to write only a part of a packet, and it doesn't pass, but a later part passes? Do we write a fragment only? Apparently yes. So dropping/delaying quanta becomes a packet fragment. Should we allow a part of packet fragment through, if only a part fits? We should let whatever fits through immediately, and drop/delay the rest. We need to do the pkt_pos math properly. Support adding and subtracting. Both should round to the closest possible value, just like floating point does. Should they cap at maximum? ppos + free, cap at end if more than zero, write if less than end, delay until what's left can fit First purpose of pkt_pos: track progress of writing packets. I.e. we need to mark where we were and where we've gone until all of the packet is written Second purpose of pkt_pos: calculating payload size and converting payload size to position to write from and until No, we're not going to be rate-limiting at the packet level. First of all, encoded size can be wildly different, then it will be hard to tell if anything was dropped or not. We need to rate-limit at the JSON writer sink. ------------ So, tests, even unit tests are essentially an outside entity. Or, perhaps, not entirely. If we ship a library, and we hide some of the internal interfaces, we might need to access them for testing purposes. Thus unit tests will be a privileged entity, with access to internal interfaces. Still, we can build a separate library, with those interfaces exposed, so that the unit tests can reach them. Regardless, unit tests and integration tests are not a part of the release distribution, they're outside of it, working on the product of it. Ideally, they need to be in a separate directory, beside tlog. Yet, having two directories side-by-side in the root of a git repo is generally unheard of, and we shouldn't really have a separate repo for the tests, as it will make it difficult to keep them in sync with the code. Shall we make subdirs under top directories instead? E.g. include tlog tltest lib tlog tltest src tlog tltest m4 autotools tlog man tlog doc tlog Let's try.