1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
|
.. -*- mode: rst -*-
.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
.. _chap_design_provenance_capture:
******************
Provenance capture
******************
.. topic:: Specification scope and status
This specification describes the current implementation.
The ability to capture process provenance---the information what activity
initiated by which entity yielded which outputs, given a set of parameters, a
computational environment, and potential input data---is a core feature of
DataLad.
Provenance capture is supported for any computational process that can be
expressed as a command line call. The simplest form of provenance tracking can
be implemented by prefixing any such a command line call with ``datalad run
...``. When executed in the content of a dataset (with the current working
directory typically being in the root of a dataset), DataLad will then:
1. check the dataset for any unsaved modifications
2. execute the given command, when no modifications were found
3. save any changes to the dataset that exist after the command has exited without error
The saved changes are annotated with a structured record that, at minimum,
contains the executed command.
This kind of usage is sufficient for building up an annotated history of a
dataset, where all relevant modifications are clearly associated with the
commands that caused them. By providing more, optional, information to the
``run`` command, such as a declaration of inputs and outputs, provenance
records can be further enriched. This enables additional functionality, such as
the automated re-execution of captured processes.
The provenance record
=====================
A DataLad provenance record is a key-value mapping comprising the following
main items:
- ``cmd``: executed command, which may contain placeholders
- ``dsid``: DataLad ID of dataset in whose context the command execution took place
- ``exit``: numeric exit code of the command
- ``inputs``: a list of (relative) file paths for all declared inputs
- ``outputs``: a list of (relative) file paths for all declared outputs
- ``pwd``: relative path of the working directory for the command execution
A provenance record is stored in a JSON-serialized form in one of two locations:
1. In the body of the commit message created when saving caused the dataset modifications
2. In a sidecar file underneath ``.datalad/runinfo`` in the root dataset
Sidecar files have a filename (``record_id``) that is based on checksum of the
provenance record content, and are stored as LZMA-compressed binary files.
When a sidecar file is used, its ``record_id`` is added to the commit message,
instead of the complete record.
Declaration of inputs and outputs
=================================
While not strictly required, it is possible and recommended to declare all
paths for process inputs and outputs of a command execution via the respective
options of ``run``.
For all declared inputs, ``run`` will ensure that their file content is present
locally at the required version before executing the command.
For all declared outputs, ``run`` will ensure that the respective locations are
writeable.
It is recommended to declare inputs and outputs both exhaustively and precise,
in order to enable the provenance-based automated re-execution of a command. In
case of a future re-execution the dataset content may have changed
substantially, and a needlessly broad specification of inputs/outputs may lead
to undesirable data transfers.
Placeholders in commands and IO specifications
==============================================
Both command and input/output specification can employ placeholders that will
be expanded before command execution. Placeholders use the syntax of the Python
``format()`` specification. A number of standard placeholders are supported
(see the ``run`` documentation for a complete list):
- ``{pwd}`` will be replaced with the full path of the current working directory
- ``{dspath}`` will be replaced with the full path of the dataset that run is invoked on
- ``{inputs}`` and ``{outputs}`` expand a space-separated list of the declared input and output paths
Additionally, custom placeholders can be defined as configuration variables
under the prefix ``datalad.run.substitutions.``. For example, a configuration
setting ``datalad.run.substitutions.myfile=data.txt`` will cause the
placeholder ``{myfile}`` to expand to ``data.txt``.
Selection of individual items for placeholders that expand to multiple values
is possible via the standard Python ``format()`` syntax, for example
``{inputs[0]}``.
Result records emitted by ``run``
=================================
When performing a command execution ``run`` will emit results for:
1. Input preparation (i.e. downloads)
2. Output preparation (i.e. unlocks and removals)
3. Command execution
4. Dataset modification saving (i.e. additions, deletions, modifications)
By default, ``run`` will stop on the first error. This means that, for example,
any failure to download content will prevent command execution. A failing
command will prevent saving a potential dataset modification. This behavior can
be altered using the standard ``on_failure`` switch of the ``run`` command.
The emitted result for the command execution contains the provenance record
under the ``run_info`` key.
Implementation details
======================
Most of the described functionality is implemented by the function
:func:`datalad.core.local.run.run_command`. It is interfaced by the ``run``
command, but also ``rerun``, a utility for automated re-execution based on
provenance records, and ``containers-run`` (provided by the ``container``
extension package) for command execution in DataLad-tracked containerized
environments. This function has a more complex interface, and supports a wider
range of use cases than described here.
|
