1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
|
.. include:: ../global.inc
.. _decorators.transform:
.. index::
pair: @transform; Syntax
.. seealso::
* :ref:`@transform <new_manual.transform>` in the **Ruffus** Manual
* :ref:`Decorators <decorators>` for more decorators
.. |input| replace:: `input`
.. _input: `decorators.transform.input`_
.. |extras| replace:: `extras`
.. _extras: `decorators.transform.extras`_
.. |output| replace:: `output`
.. _output: `decorators.transform.output`_
.. |filter| replace:: `filter`
.. _filter: `decorators.transform.filter`_
.. |matching_regex| replace:: `matching_regex`
.. _matching_regex: `decorators.transform.matching_regex`_
.. |matching_formatter| replace:: `matching_formatter`
.. _matching_formatter: `decorators.transform.matching_formatter`_
.. |suffix_string| replace:: `suffix_string`
.. _suffix_string: `decorators.transform.suffix_string`_
################################################################################################
@transform( |input|_, |filter|_, |output|_, [|extras|_,...] )
################################################################################################
**Purpose:**
Applies the task function to transform data from |input|_ to |output|_ files.
Output file names are specified from |input|_, i.e. from the |output|_
of specified tasks, or a list of file names, or a |glob|_ matching pattern.
String replacement occurs either through suffix matches via :ref:`suffix<decorators.suffix>` or
the :ref:`formatter<decorators.formatter>` or :ref:`regex<decorators.regex>` indicators.
Only out of date tasks (comparing |input|_ and |output|_ files) will be run
**Simple Example**
Transforms ``*.c`` to ``*.o``::
@transform(input = ["1.c", "2.c"], filter = suffix(".c"), output = ".o")
def compile(infile, outfile):
pass
Same example with a regular expression::
@transform(["1.c", "2.c"], regex(r".c$"), ".o")
def compile(infile, outfile):
pass
Both result in the following function calls:
::
# 1.c -> 1.o
# 2.c -> 2.o
compile("1.c", "1.o")
compile("2.c", "2.o")
**Escaping regular expression patterns**
A string like ``universal.h`` in ``add_inputs`` will added *as is*.
``r"\1.h"``, however, performs suffix substitution, with the special form ``r"\1"`` matching everything up to the suffix.
Remember to 'escape' ``r"\1"`` otherwise Ruffus will complain and throw an Exception to remind you.
The most convenient way is to use a python "raw" string.
**Parameters:**
.. _decorators.transform.input:
* **input** = *tasks_or_file_names*
can be a:
#. Task / list of tasks (as in the example above).
File names are taken from the |output|_ of the specified task(s)
#. (Nested) list of file name strings.
File names containing ``*[]?`` will be expanded as a |glob|_.
E.g.:``"a.*" => "a.1", "a.2"``
.. _decorators.transform.filter:
.. _decorators.transform.suffix_string:
* **filter** = *suffix(suffix_string)*
must be wrapped in a :ref:`suffix<decorators.suffix>` indicator object.
The end of each |input|_ file name which matches ``suffix_string`` will be replaced by |output|_.
Input file names which do not match suffix_string will be ignored
The non-suffix part of the match can be referred to using the ``"\1"`` pattern. This
can be useful for putting the output in different directory, for example::
@transform(["1.c", "2.c"], suffix(".c"), r"my_path/\1.o")
def compile(infile, outfile):
pass
This results in the following function calls:
::
# 1.c -> my_path/1.o
# 2.c -> my_path/2.o
compile("1.c", "my_path/1.o")
compile("2.c", "my_path/2.o")
For convenience and visual clarity, the ``"\1"`` can be omitted from the output parameter.
However, the ``"\1"`` is mandatory for string substitutions in additional parameters, ::
@transform(["1.c", "2.c"], suffix(".c"), [r"\1.o", ".o"], "Compiling \1", "verbatim")
def compile(infile, outfile):
pass
Results in the following function calls:
::
compile("1.c", ["1.o", "1.o"], "Compiling 1", "verbatim")
compile("2.c", ["2.o", "2.o"], "Compiling 2", "verbatim")
Since r"\1" is optional for the output parameter, ``"\1.o"`` and ``".o"`` are equivalent.
However, strings in other parameters which do not contain r"\1" will be included verbatim, much
like the string ``"verbatim"`` in the above example.
.. _decorators.transform.matching_regex:
* **filter** = *regex(matching_regex)*
is a python regular expression string, which must be wrapped in
a :ref:`regex<decorators.regex>`\ indicator object
See python `regular expression (re) <http://docs.python.org/library/re.html>`_
documentation for details of regular expression syntax
Each output file name is created using regular expression substitution with ``output``
.. _decorators.transform.matching_formatter:
* **filter** = *formatter(...)*
a :ref:`formatter<decorators.formatter>` indicator object containing optionally
a python `regular expression (re) <http://docs.python.org/library/re.html>`_.
.. _decorators.transform.output:
* **output** = *output*
Specifies the resulting |output|_ file name(s) after string substitution
.. _decorators.transform.extras:
* **extras** = *extras*
Any extra parameters are passed verbatim to the task function
If you are using named parameters, these can be passed as a list, i.e. ``extras= [...]``
Any extra parameters are consumed by the task function and not forwarded further down the pipeline.
If ``regex(matching_regex)`` or ``formatter(...)``` is used, then substitution
is first applied to (even nested) string parameters. Other data types are passed
verbatim.
For example::
@transform(["a.c", "b.c"], regex(r"(.*).c"), r"\1.o", r"\1")
def compile(infile, outfile):
pass
will result in the following function calls::
compile("a.c", "a.o", "a")
compile("b.c", "b.o", "b")
See :ref:`here <decorators.transform_ex>` for more advanced uses of transform.
|
