File: ChangeLog.txt

package info (click to toggle)
python-pyflow 1.1.20-4
  • links: PTS, VCS
  • area: main
  • in suites: bookworm
  • size: 996 kB
  • sloc: python: 4,154; sh: 219; ansic: 15; makefile: 5
file content (232 lines) | stat: -rw-r--r-- 10,280 bytes parent folder | download | duplicates (3)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
v1.1.20 20180302
* Close infrequent race condition in task resolution (STREL-853)
  * Under the race condition error, a non-failing task would be logged as failing
    with the message "TASKNAME has stopped without a traceable cause"
  * The error rate was believed to be very infrequent, but possibly agrevated
    by many very short-running tasks and/or specific python versions.
* Make timezone explicit in logging time stamps
v1.1.19 20180102
* Minor fix/adjustments, primarily doubling the default number of stderr lines
forwarded from failed task to 40.
v1.1.18 20170713
* Add ephemeral sub-workflow option
* Add method to indicate stopped workflows
* Reduce infrequent interpreter shutdown errors
* [#19] Prevent repeated state file tasks on resuming interrupted run
* [#18] Flush task state files on interrupt
v1.1.17 20170526
* Make workflow task specification notice less verbose
* Fix cancelTaskTree to work correctly within subworkflows
* Modify cancelTaskTree to ignore running tasks
v1.1.16 20170523
* Improve task processing time and query accuracy
* Fix task status query methods to work correctly within subworkflows
v1.1.15 20170519
* Add new launchTasksUntil demo
* Add new WorkflowRunner.cancelTaskTree() method
* Add new WorkflowRunner.isTaskDone() method
* Add automated source formatting scripts
v1.1.14 20170112
* STREL-391 improve task throughput for nodes with 100's of cores
v1.1.13 20160414
* fix rare issue with sets of dependent checkpoint tasks
* fix for travis CI script from Dominic Jodoin
v1.1.12 20151203
* lengthen signal file delay tolerance to 4 mins
* [#14] Filter environment variables to remove bash functions. This
eliminates complications between shellshock bash update and SGE.
v1.1.11 20151125
* Improve SGE robustness
v1.1.10 20150927
* Remove old custom cluster SGE logic and standardize on h_vmem
v1.1.9 20150923
* Add windows CI script to run cross-platform tests on master
* Add new cross-platform test script and adjust all unit tests
to run on windows
* Improve error reporting for missing signal file case
v1.1.8 20150918
* Improve windows shell compatibility
* [#10] If an error occurs creating the task visualization script,
issue warning and allow workflow to continue
v1.1.7 20150806
* [#9] improve robustness to filesystem delays for task
wrapper parameter file
* [#9] improve error log specificity when anomolous task
wrapper output occurs in localhost run mode
v1.1.6 20150713
* Fix multithread conflict introduced by [#5] fix
v1.1.5 20150710
* Changed to BSD 2-Clause license
* [#5] fix in-workflow check for nested workflows
v1.1.4 20150527
* added check for local SMTP service before sending email notification
* added extra check aginst workflow methods running outside of pyflow runtime
v1.1.3 20141028
* fix master node memory spike for SGE jobs identified by Lilian Janin
* added Windows OS patches form Tobias Mann
v1.1.2 20131026
* added python distutils setup.py provided by Ryan Kelley
v1.1.1 20130716
* fix issue with new startFromTasks feature when used with subWorkflows
v1.1.0 20130715
* add new features to allow only part of workflow to be run and/or continued
v1.0.1 20130710
* Fix O(exp) scaling problems in DAG handling methods
v1.0.0 20130507
* relicenced to Illumina Open Source Software License v1
v0.6.26 20130304
* fix bug in forceContinue'd workflow runs
v0.6.25 20130221
* Add optional warning and error log which contains all logged warning or error messages.
* allow specification of a custom notification message on successful workflow completion
* allow any task to specify its own retry parameters, overriding the
run-level parameters of the same name
* add retryMode to allow task retry to be applied to local mode
v0.6.24 20121128
* accelerate localhost task sweep so that short-running task workflows can
execute more quickly
* create new mutex option to addTask, this allows a set of tasks to share a mutex
id, causing no more than one in the group to be executed at the same time
v0.6.23 20121018
* change cwd parameter to not require directory to exist
* fix version number search so that an non-installed version does not require git to be installed
v0.6.22 20121002
* fix custom environment option to be more robust in various sge contexts
v0.6.21 20120925
* add option to supply custom environment variables to any task
* fix error message when invalid runmode is given
v0.6.20 20120920
* increase # of retries and timeout length for qstat call
v0.6.19 20120914
* check for and allow OSError on fsync call.
v0.6.18 201207
* ignore available memory limit in non-local run modes
* detect if multiple active pyflow jobs are attempting to use the same data
directory
v0.6.17 20120622
* minor cleanups: add python version to reports and python 2.7.2 warning to
logs
v0.6.16 20120529
* issue error when task specifies itself as a dependency
* fix issue which could cause pyflow to hang when using python 2.7.2 in sge
mode, hang can still occur in local mode. python 2.7.2 should be avoided.
v0.6.15 20120525
* Improved developer logging: dump stack for all threads in python 2.5+ during the update interval
* Additional sge command error logging
* automate pyflow version setting and add this to workflow logs
* improved logging scalability for 10000+ task workflows
* improved API documentation
v0.6.14.1 20120518
* remove typo bug
v0.6.14 20120507
* Add timeouts to sge qsub and qstat commands in case these hang (observed at
low fequency on sd clusters)
* Write SGE job_number to log for evey qsub-ed job
* Write the name of the longest queued and longest running tasks in the status
update report.
* Add new demo demonstrating commandline settings for workflow run arguments
v0.6.13 20120503
* Fix regression in sge make jobs introduced with queue option
v0.6.12 20120429
* Add cwd argument to addTask to change wd before task execution
v0.6.11 20120424
* Remove sub-workflows from status update report
* Dump full qsub arg list for each sge job to temporary logs in case of sge anomoly
* Log sge job number in case of anomolous state at end of sge job
* taskWrapper logs hostname as soon as possible in case of error
* More reliable (but slower) flush used for log writes
* Add option to provide a list of arguments to qsub/qmake (to specify queue
most likely)
* Add option to turn off logging to stderr.
v0.6.10 20120419
* Provide a 'heartbeat' task status update to the log at a specified
interval.
v0.6.9
* Improve robustness against NFS update delay for task wrapper file
* Include more sge error details in Notification email
v0.6.8 20120319
* Better handling on terminal hang-up: capture and ignore SIGHUP and
handle failed writes to stderr. You should still use nohup where needed,
but if you don't, then proper logging and notification will continue.
* flush log writes
v0.6.7 20120316
* add tail of task stderr stream to nofications when tasks fail
* apply ctrl-C task shutdown to SIGTERM as well
v0.6.6 20120315
* include configuration for uscp-prd cluster
* Passive graph creation
* Reduce thread stack size in later versions of python
* More robust ctrl-C shutdown behavior (master workflow on own thread)
v0.6.5 20120308
* bagPipes fix
v0.6.4 20120307
* Consume filehandles more efficiently when running a very large number of
local jobs. Reset SGE default max jobs to something reasonable (128). Allow
logging to continue even after filehandles are exhausted in case it manages
to happen.
v0.6.3 20120305
* Switch sge mode from using qsub sync to a more scalable qsub-and-poll scheme
This immediately removes the 99 job sge limit, and provides the infrastructure
for queueing or running timeout on tasks.
v0.6.2
* allow commands to be submitted as argument lists (in addition to shell
strings, as before). Argument lists allow longer commands and obviate a
variety of quoting issues.
* Change site configuration to an object scheme which simplifies site
customization.
* change qmake tasks from parallel env to dynamic allocation mode
* allow qmake jobs to retry without an expiration window
* improved reporting of failed make jobs
* fixed minor issue with make path cleanup
v0.6.1 20120228
* primarily a bagPipes release
* fixed isForceLocal task bug introduced in 0.6
* fixed make task bug introduced in 0.6
v0.6 20120227
* primarily a bagPipes release
* added task priority option

v0.5.5 20120224
* more robust hostname lookup combines ip query and alias list
* fix runner.bash demo failures on some machines
* fix pyflowTaskWrapper stderr caching
v0.5.4 20120224
* fix configuration for non-sge hosts
v0.5.2 20120222
* chuk pthread fix for pyflow tasks
v0.5.1 20120221
* Added autmatic chuk sge configuration to allow bagPipes to complete
in the uk.
* Various fixes from uk testing: (1) full hostname is correctly found in
the uk now (2) default configuration for email is it now comes form
"pyflow-bot@"YOUR_DOMAIN_NAME now. This is required to correctly
get mail sent from a uk box.
v0.5 20120220
* Cutting version of pyFlow to sync with first bagPipes prototype
* add max SGE jobs to configuration parameters -- default set to 96
* Fix sub-workflows to shutdown properly after task shutdown
* More robust handling of commands with quoting and special characters
* Non-breaking API change: isTaskComplete lets you query whether a task
is in the workflow and completed  -- useful for sane interrupt/resume behavior
* Non-breaking API change: limitNCores(n) and limitMemMb(n) can be used now to
reduce your resource request to the maximum available for this run.

v0.4 20120216
* Added memory to the task resource tracking
* Created pyflowConfig file which contains site specific code --
moved resource to qsub argument translation functions into this
config file
* Non-breaking API change: Added isCmdStable option to addTask to specify that a command
can change on workflow resume.
* Non-breaking API change: all add*() methods return task label

v0.3 20120213
* Full support for recursive WorkflowRunner task specification -- provide any
other WorkflowRunner instance as a task in a workflow() definition.
* Report Exceptions in TaskRunner objects as task errors
* Report list of failed tasks even during a waitForTasks() holding loop.

v0.2 20120207
First versioned released. Major addition is a complete command-line interface
for the bcl converter demo