File: ex_dependencies.html

package info (click to toggle)
tup 0.8-1
  • links: PTS, VCS
  • area: main
  • in suites: trixie
  • size: 21,068 kB
  • sloc: ansic: 256,651; sh: 19,107; perl: 184; python: 67; lisp: 63; makefile: 56
file content (193 lines) | stat: -rw-r--r-- 9,603 bytes parent folder | download | duplicates (2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
<p>Tup handles dependencies a little differently from other build systems. In this example, we'll show how the dependencies that you specify in a Tupfile work together with the dependencies determined automatically during the program's execution.</p>

<p>First setup a test directory with a Tupfile and shell script:</p>
<pre>
<span class="prompt">$</span> tup init tup_test2
<span class="prompt">$</span> cd tup_test2
</pre>

<span class="fileheader">test.sh</span>
<pre class="code">#! /bin/sh
echo "Output from test.sh"
</pre>

<span class="fileheader">Tupfile</span>
<pre class="code">: |&gt; ./test.sh &gt; %o |&gt; output.txt
</pre>

<pre>
<span class="prompt">$</span> chmod +x test.sh
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.006s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] ./test.sh &gt; output.txt
[ tup ] Updated.
<span class="prompt">$</span> cat output.txt
Output from test.sh
</pre>

<p>So our simple script ran and created the <span class="filename">output.txt</span> file. When will tup decide to run the script again to create a new output? The answer is simple: whenever the <span class="filename">test.sh</span> script changes! Watch as <span class="cmd">tup</span> does not re-run the script until the script is touched:</p>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.000s
[ tup ] No tup.config changes.
[ tup ] No Tupfiles to parse.
[ tup ] No files to delete.
[ tup ] No commands to execute.
[ tup ] Updated.
<span class="prompt">$</span> touch test.sh
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.005s
[ tup ] No tup.config changes.
[ tup ] No Tupfiles to parse.
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] ./test.sh &gt; output.txt
[ tup ] Updated.
</pre>

<p>We can change the script to read from other files, as well. Suppose we create a new text file, called <span class="filename">header.txt</span>, that gets cat'd at the top of the script:</p>
<span class="fileheader">header.txt</span>
<pre class="code">This is the file header
</pre>

<span class="modfileheader">test.sh</span>
<pre class="code">#! /bin/sh
<span class="added">cat header.txt</span>
echo "Output from test.sh"
</pre>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] ./test.sh &gt; output.txt
[ tup ] Updated.
<span class="prompt">$</span> cat output.txt
This is the file header
Output from test.sh
</pre>

<p>Note that we don't need to specify the <span class="filename">header.txt</span> dependency in the Tupfile, but tup will still re-run the script if it changes:</p>

<span class="modfileheader">header.txt</span>
<pre class="code">This is the<span class="added"> *new*</span> file header
</pre>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] No Tupfiles to parse.
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] ./test.sh &gt; output.txt
[ tup ] Updated.
<span class="prompt">$</span> cat output.txt
This is the *new* file header
Output from test.sh
</pre>

<p>Similar to the <a href="ex_a_first_tupfile.html">first Tupfile example</a>, tup has executed the script in such a way that it can track file accesses. Tup sees that the script read from <span class="filename">header.txt</span> and automatically adds the dependency. We can see this from the dependency graph:</p>

<pre>
<span class="prompt">$</span> tup graph --stickies . | dot -Tpng &gt; ~/ex_deps_1.png
</pre>
<div class="img"><img src="ex_deps_1.png"></div>

<p>Both the <span class="filename">header.txt</span> and <span class="filename">test.sh</span> files point to the command that executes the test script. If either of those files change, then tup will re-execute the script. Otherwise there's no need to run it again!</p>
<div style="clear:both;"></div>

<h3>Dependencies on Generated Files</h3>
<p>At this point, the <span class="filename">header.txt</span> and <span class="filename">test.sh</span> files are written by you, the author. Tup refers to these as "normal" files (in the tup source code, this is the TUP_NODE_FILE enum). The <span class="filename">output.txt</span> file is generated by tup, so it is called a "generated" file (TUP_NODE_GENERATED in the source). Generated files are treated a little differently in tup. Let's see what happens if we create a new generated file and try to read from it in the test script:</p>
<span class="modfileheader">Tupfile</span>
<pre class="code"><span class="added">: |&gt; echo "generated text" &gt; %o |&gt; generated.txt</span>
: |&gt; ./test.sh &gt; %o |&gt; output.txt
</pre>

<span class="modfileheader">test.sh</span>
<pre class="code">#! /bin/sh
cat header.txt
<span class="added">cat generated.txt</span>
echo "Output from test.sh"
</pre>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/2    ] echo "generated text" &gt; generated.txt
[    2/2    ] ./test.sh &gt; output.txt
 *** tup errors ***
tup error: Missing input dependency - a file was read from, and was not
specified as an input link for the command. This is an issue because the file
was created from another command, and without the input link the commands may
execute out of order. You should add this file as an input, since it is
possible this could randomly break in the future.
 - [8] generated.txt
 *** Command ID=6 ran successfully, but tup failed to save the dependencies.
</pre>

<p>Oops, it seems tup didn't like that. The issue here is that tup has no way of knowing that the command to create <span class="filename">generated.txt</span> must run before the <span class="filename">test.sh</span> script. Therefore, it is possible that tup will schedule them in the wrong order (so that <span class="filename">generated.txt</span> isn't created by the time the script runs), or it may even schedule them in parallel.</p>
<p>To give tup the information that <span class="filename">generated.txt</span> must be created first, we simply list it as an input to the test script:</p>
<span class="modfileheader">Tupfile</span>
<pre class="code">: |&gt; echo "generated text" &gt; %o |&gt; generated.txt
: <span class="added">generated.txt</span> |&gt; ./test.sh &gt; %o |&gt; output.txt
</pre>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] ./test.sh &gt; output.txt
[ tup ] Updated.
<span class="prompt">$</span> cat output.txt
This is the *new* file header
generated text
Output from test.sh
</pre>

<h3>Tupfile Dependencies are for Ordering</h3>
<p>Now we will see what happens if you add an input to the Tupfile, but the input goes unused. Let's add another generated file, but the shell script is unchanged so we don't actually read from it:</p>
<span class="modfileheader">Tupfile</span>
<pre class="code">: |&gt; echo "generated text" &gt; %o |&gt; generated.txt
<span class="added">: |&gt; echo "unused text" &gt; %o |&gt; unused.txt</span>
: generated.txt<span class="added"> unused.txt</span> |&gt; ./test.sh &gt; %o |&gt; output.txt
</pre>

<pre>
<span class="prompt">$</span> tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] echo "unused text" &gt; unused.txt
[ tup ] Updated.
</pre>

<p>Curiously, the <span class="filename">test.sh</span> script did not execute even though <span class="filename">unused.txt</span> is listed as an input. How then were we able to bypass the earlier error message by adding an input in the Tupfile, if those inputs don't actually cause the commands to execute? Let's take a look at our new graph:</p>
<pre>
<span class="prompt">$</span> tup graph . | dot -Tpng &gt; ~/ex_deps_2.png
</pre>
<div class="img"><img src="ex_deps_2.png"></div>
<p>Here we can see that not all of the input dependency arrows are the same. There are solid lines and dotted lines, as well as filled in arrows and empty arrows. The actual details of how tup handles these are beyond the scope of this example. The important thing to see here is that tup still keeps track of the fact that the shell script has a dependency on the <span class="filename">unused.txt</span> file, but because the file was never actually read from by the script, it cannot possibly have an effect on the output file (this fact is represented by the dotted line). Therefore, tup knows that if <span class="filename">unused.txt</span> changes, the script does not need to run again.</p>
<p>However, the presence of the dotted-line dependency means that if <span class="filename">unused.txt</span> is changed and at the same time <span class="filename">test.sh</span> is changed to read from that file, tup is guaranteed to execute them in the correct order. In this sense, inputs in Tupfiles are only for ordering, and dependencies determined automatically during program execution are used for re-executing commands. For more insight into why this is useful, see the <a href="ex_generated_header.html">generated header example</a>.</p>