1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
# batchtools 0.9.17
* Fixed a bug in the finalizer of `ClusterFunctionsMulticore`.
# batchtools 0.9.16
* Fixed a bug in `addExperiments()` in combination with combination method `"bind"` and repls > 1 where experiments have been duplicated.
* `addExperiments()` now also accepts a vector of replications (instead of a single scalar value) for argument `repls`.
* Improved handling of jobs in `ClusterFunctionsSlurm`.
* Fixed a bug in `waitForJobs()`
* Fixed some assertions.
# batchtools 0.9.15
* Maintenance update.
# batchtools 0.9.14
* `batchMap()` now supports unnamed `more.args`.
* Exports are now assigned with `delayedAssign()`.
* Fix an option in the LSF template.
# batchtools 0.9.13
* Maintenance release for R-4.0.0.
# batchtools 0.9.12
* Moved `data.table` from `Depends` to `Imports`.
User scripts might need to explicitly attach `data.table` via `library()` now.
* Fixes for `ClusterFunctionsMulticore`.
* Removed a workaround for `system2()` for R-devel (to be released as R-4.0.0).
* New configuration option `compress` to select the compression algorithm (passed down to `saveRDS()`).
# batchtools 0.9.11
* Removed deprecated function `chunkIds()`.
* New default for argument `fs.timeout` in the cluster function constructor is `0` (was `NA` before).
* Fixed a unit test for OSX.
* Improved stability and documentation.
* Fixed memory usage calculation.
# batchtools 0.9.10
* Exported functions `findConfFile()` and `findTemplateFile()`.
* Dropped support for providing a template file directly as string. A valid file is now always required.
* Fixed writing to `TMPDIR` instead of the R session's temporary directory.
# batchtools 0.9.9
* RDS files are explicitly stored in version 2 to ensure backward compatibility with R versions prior to 3.5.0.
* Package `fs` is now used internally for all file system operations.
* Support for per-site configuration files and templates to be set up by system administrators.
* The print of `getStatus()` now includes a time stamp.
* `chunk()` now optionally shuffles the ids before chunking.
* Support for setting per-job resources in `submitJobs()`.
* Example templates now include resources for `blas.threads` and `omp.threads`.
* Some bug fixes regarding read-only registries.
# batchtools 0.9.8
* Renamed column "memory" in the status table to "mem.used" to avoid name clashes with the resource specification.
* Exported function `assertRegistry()`.
* New function `unwrap()` as alias to `flatten()`.
The latter causes a name clash with package `purrr` and will be deprecated in a future version.
* Registries now contain a unique hash which is updated each time the registry is altered.
Can be utilized to invalidate caches, e.g. the cache of knitr.
# batchtools 0.9.7
* Added a workaround for a test to be compatible with testthat v2.0.0.
* Better and more customizable handling of expired jobs in `waitForJobs()`.
* Package `foreach` is now supported for nested parallelization as an alternative to `parallelMap`.
* Depreciated argument flatten has been removed.
* New helper function `flatten()` to manually unnest/unwrap lists in data frames.
* Removed functions `getProblemIds()` and `getAlgorithmIds()`.
Instead, you can just access `reg$problems` or `reg$algorithms`, respectively.
* The number of the maximum concurrent jobs can now also be controlled via setting resources.
* Internal data base changes to speed up some operations.
Old registries are updated on first load by `loadRegistry()`.
* Fixed a bug where the sleep mechanism between queries was not working.
* Fixed a bug where submit errors on SLURM and TORQUE were not detected as temporary.
# batchtools 0.9.6
* Fixed a bug where the wrong problem was retrieved from the cache. This was only triggered for chunked jobs in
combination with an `ExperimentRegistry`.
# batchtools 0.9.5
* Added a missing routine to upgrade registries created with batchtools prior to v0.9.3.
* Fixed a bug where the registry could not be synced if jobs failed during initialization (#135).
* The sleep duration for `waitForJobs()` and `submitJobs()` can now be set via the configuration file.
* A new heuristic will try to detect if the registry has been altered by a simultaneously running R session.
If this is detected, the registry in the current session will be set to a read-only state.
* `waitForJobs()` has been reworked to allow control over the heuristic to detect expired jobs.
Jobs are treated as expired if they have been submitted but are not detected on the system for `expire.after` iterations
(default 3 iterations, before 1 iteration).
* New argument `writeable` for `loadRegistry()` to allow loading registries explicitly as read-only.
* Removed argument `update.paths` from `loadRegistry()`.
Paths are always updated, but the registry on the file system remains unchanged unless loaded in read-write mode.
* `ClusterFunctionsSlurm` now come with an experimental nodename argument. If set, all communication with the master is
handled via SSH which effectively allows you to submit jobs from your local machine instead of the head node.
Note that mounting the file system (e.g., via SSHFS) is mandatory.
# batchtools 0.9.4
* Fixed handling of `file.dir` with special chars like whitespace.
* All backward slashes will now be converted to forward slashes on windows.
* Fixed order of arguments in `findExperiments()` (argument `ids` is now first).
* Removed code to upgrade registries created with versions prior to v0.9.0 (first CRAN release).
* `addExperiments()` now warns if a design is passed as `data.frame` with factor columns and `stringsAsFactors` is `TRUE`.
* Added functions `setJobNames()` and `getJobNames()` to control the name of jobs on batch systems.
Templates should be adapted to use `job.name` instead of `job.hash` for naming.
* Argument `flatten` of `getJobResources()`, `getJobPars()` and `getJobTable()` is deprecated and will be removed.
Future versions of the functions will behave like `flatten` is set to `FALSE` explicitly.
Single resources/parameters must be extracted manually (or with `tidyr::unnest()`).
# batchtools 0.9.3
* Running jobs now are also included while querying for status "started". This affects `findStarted()`, `findNotStarted()` and `getStatus()`.
* `findExperiments()` now performs an exact string match (instead of matching substrings) for patterns specified via `prob.name` and `algo.name`.
For substring matching, use `prob.pattern` or `algo.pattern`, respectively.
* Changed arguments for `reduceResultsDataTable()`
* Removed `fill`, now is always `TRUE`
* Introduced `flatten` to control if the result should be represented as a column of lists or flattened as separate columns.
Defaults to a backward-compatible heuristic, similar to `getJobPars`.
* Improved heuristic to lookup template files.
Templates shipped with the package can now be used by providing just the file name (w/o extension).
* Updated CITATION
# batchtools 0.9.2
* Full support for array jobs on Slurm and TORQUE.
* Array jobs have been disabled for SGE and LSF (due to missing information about the output format) but will be re-enable in a future release.
Note that the variable `n.array.jobs` has been removed from `JobCollection` in favor of the new variable `array.jobs` (logical).
* `findExperiments()` now has two additional arguments to match using regular expressions.
The possibility to prefix a string with "~" to enable regular expression matching has been removed.
* New function `batchReduce()`.
* New function `estimateRuntimes()`.
* New function `removeRegistry()`.
* Missing result files are now handled more consistently, raising an exception in its defaults if the result is not available.
The argument `missing.val` has been added to `reduceResultsList()` and `reduceResultsDataTable()` and removed from `loadResult()` and `batchMapResults()`.
* Timestamps are now stored with sub-second accuracy.
* Renamed Torque to TORQUE. This especially affects the constructor `makeClusterFunctionsTorque` which now must be called via `makeClusterFunctionsTORQUE()`
* `chunkIds()` has been deprecated. Use `chunk()`, `lpt()` or `binpack()` instead.
* Fixed listing of jobs for `ClusterFunctionsLSF` and `ClusterFunctionsOpenLava` (thanks to @phaverty).
* Job hashes are now prefixed with the literal string 'job' to ensure they start with a letter as required by some SGE systems.
* Fixed handling of `NULL` results in `reduceResultsList()`
* Fixed key lookup heuristic join functions.
* Fixed a bug where `getJobTable()` returned `difftimes` with the wrong unit (e.g., in minutes instead of seconds).
* Deactivated swap allocation for `ClusterFunctionsDocker`.
* The package is now more patient while communicating with the scheduler or file system by using a timeout-based approach.
This should make the package more reliable and robust under heavy load.
# batchtools 0.9.0
Initial CRAN release.
See the vignette for a brief comparison with [BatchJobs](https://cran.r-project.org/package=BatchJobs)/[BatchExperiments](https://cran.r-project.org/package=BatchExperiments).
|
