1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450
|
======================================================================
Version 1.1.1, by Aleksey Cheusov, Fri, 15 Feb 2019 21:58:39 +0300
Fix build using clang
Verify snprintf didn't truncate the output
======================================================================
Version 1.1.0, by Aleksey Cheusov, Wed, 25 Apr 2018 23:21:24 +0300
paexec:
- add new option -0. It works just like in "xargs -0".
- add new option -J.
- add new option -mw=.
- fix help message display by -h.
- -md= now allows no delimiter mode in -g mode.
- -c and -C override each other if one is implied after another.
Add new tool "paargs". It is a wrapper over paexec(1) that
simplifies use of paexec.
Fix transport_broken_rnd test script.
This fixes regression test on Solaris.
Update man page for paexec(1).
======================================================================
Version 1.0.1, by Aleksey Cheusov, Thu, 14 Aug 2014 01:44:39 +0300
Unflushed (broken) stdout seen on Darwin was fixes.
Who knows, this may happen on other systems too.
======================================================================
Version 1.0.0, by Aleksey Cheusov, Sun, 13 Jul 2014 19:10:48 +0300
mk-configure>=0.27.0 is required for build.
Option -X was introduced to paexec(1) for ignoring
calculator's stdout.
PAEXEC_ENV environment variable sets a list of variables passed to
the calculator.
PAEXEC_TRANSPORT environment variable sets the transport unless
option -t was applied.
PAEXEC_NODES environment variable sets the nodes unless
option -n was applied.
pareorder(1) is a synonym for paexec_reorder(1)
======================================================================
Version 0.19.1, by Aleksey Cheusov, Mon, 26 Aug 2013 19:05:53 +0300
paexec: Fix for segfault seen on Linux
Version 0.19.0, by Aleksey Cheusov, Sun, 25 Aug 2013 17:21:03 +0300
This release of paexec was successfully tested on the following
platforms: NetBSD-6.1/amd64, OpenBSD-5.3/i386, FreeBSD-8.3/i386,
Solaris-10/sparc, Solaris-11/amd64 and diverse Linux/{i386,amd64}.
paexec is now selfcontained, libmaa is not needed anymore.
Presentation paexec.pdf was added to presentation/. I hope it will
help easier understand how "paexec" works. It is installed to
${DOCDIR}.
paexec:
- POSIX-2008 getline(3) is used for reading lines
instead of home-made function.
- t '' is equivalent to "no transport", spaces are trimmed.
- "-n +NNN" has higher priority than -t, i.e. if they both are
specified, transport is ignored.
- Fix for -W1.
- Environment variable PAEXEC_EOT was introduced.
- Option -y was added to paexec(1) and paexec_reorder(1).
- Option -C was added to paexec(1).
paexec_reorder:
- Option -x was added to paexec_reorder(1).
A number of fixes and improvements in regression tests.
======================================================================
Version 0.18.0, by Aleksey Cheusov, Thu, 7 Mar 2013 15:17:33 +0300
paexec:
- fixed: NULL dereference when the first line given on input is
empty. Thanks to Sergey Revyako for bug report!
- fixed: entire command passed tp ssh-like transport should be
shquoted. In particular this fixes -x that didn't work with -t.
Thanks to Sergey Revyako for bug report!
- 'paexec -g' accepts empty strings as tasks.
- More regression tests were added
paexec_reorder.1:
- Mistype fix
======================================================================
Version 0.17.0, by Aleksey Cheusov, Sun, 9 Sep 2012 02:07:34 +0300
paexec:
- Option -x was added. With its help paexec can run one command
per task. If -g is also specified, command's exit status is
analysed. Appropriate task and dependants are marked as "failed"
if it is non-zero.
- First character of -n argument must be alphanumeric, `+', `_',
`:' or `/'. Other symbols are reserved for future extentions.
- With '-n :filename' paexec reads a list of nodes from the
specified file.
- With a help of new option '-m t=<eot>' end of task string
may be specified, which is an empty line by default.
- Option -md=<delim> was added that overrides the default
delimiter (space character) between tasks in graph mode (-g).
- Output line that contains failed dependants no longer ends with
unnecessary space.
- Long options were completely removed.
paexec_reorder:
- Fix. "paexec_reorder -g" now handles correctly failed tasks'
output. One extra line after "fatal" is expected.
- Options -m was added. It does the same things as paexec's -m.
More examples of use and regression tests.
Documentation update, clean-ups and improvements.
Regression tests:
- Signals handling was fixed in.
- LC_ALL is always set to C in regression tests, this fixes some
problems in internationalized environment.
mk-configure>=0.23.0 is required at build time
======================================================================
Version 0.16.1, by Aleksey Cheusov, Wed, 23 Mar 2011 00:14:15 +0200
paexec.1 and paexec_reorder.1 are included to paexec-0.16.1.tgz.
Just like in paexec<=0.15.1 pod2man is not needed
for building paexec.
======================================================================
Version 0.16, by Aleksey Cheusov, Fri, 11 Mar 2011 11:58:31 +0200
Project's structure has been reorganized. Now the top-level Makefile
uses mkc.subprj.mk. This adds a lot of flexibility in building the
project and development.
See doc/INSTALL for updated installation instructions.
New modes for reordering tasks were added: -W0 and -W2
See the manual page for details about option -W.
Long options are considered deprecated. They are still supported but
will be removed in the future. Please use POSIX short options. At
the moment use of them produces warning message on stderr.
Fix the compilation bug on old versions of OpenBSD (at least <=3.8)
and probably other systems where intptr_t is declared in inttypes.h
but not in stdint.h
Fix for 'mkcmake test' failure on Solaris and HP-UX because their
/usr/bin/awk sucks. On these platforms now it is possible to run the
following command.
env AWK=/full/path/to/gawk mkcmake all test
======================================================================
Version 0.15.0, by Aleksey Cheusov, Sun, 10 Oct 2010 18:06:31 +0300
After some thoughts I decided to switch from plain mk-files to
mk-configure build system. It provides very cool features and makes
development drammatically easier. It also makes Makefiles much
cleaner and easier. Installation instructions has been updated in
README file.
PAEXEC reads the tasks from stdin and distributes them to all
available hosts or CPU. If some tasks are easy and can be made
quickly while others require much more CPU time it makes sense to
reorder tasks calcultion in order to reduce total calculation time
and reduce computers/CPUs idle time. For this purpose -W<num> option
is added to paexec(1) using weights assigned to each tasks. At the
moment only two values are allowed for <num>: 0 -- do not use
weights at all, 1 -- run heavier tasks first as soon as possible.
New tool paexec_reorder(1) for reordering sliced output of
paexec(1). It is written in runawk. So, you'll need it at run time.
FIXED: 1 byte buffer overflow if -d option is applied.
Documentation update. Tons of new regression tests. Regression
tests framework has been significantly reworked. Clean-ups.
README: notes about my Debian/Lenny/x86 repository.
Minor fixes (warning messages) for different compilers.
======================================================================
Version 0.14.0, by Aleksey Cheusov, Sun, 3 Jan 2010 12:38:54 +0200
fixed: "paexec -z" without -s never worked. Now it is fine.
New options -g|--graph are synonyms for -s|--pos.
New option -w for waiting for failed nodes if they ALL failed.
By default (with -Z) if ALL nodes fail paexec exits with error.
New option -m for setting alternative strings for
success/failure/fatal.
Target for installing directories has been renamed from
install-dirs to installdirs.
Other minor fixes and code clean-ups.
======================================================================
Version 0.13.0, by Aleksey Cheusov, Sat, 7 Mar 2009 19:17:03 +0200
FIXED: When 'paexec -s' retreives 10000 tasks, it allocates
10000*10000*sizeof(int) bytes for detecting cycles, i.e. ~400Mb on
32-bit system or 800Mb on 64-bit system. This is absolutely
inacceptable. This bad algorithm is replaced with new one which
doesn't need quadratic matrix and works much faster.
ADDED: -Z<timeout> option
When I<-z> applied, if a I<command> fails, appropriate node is
marked as broken and is excluded from the following task
distribution. But if B<-Z> applied, every I<timeout> seconds an
attempt to rerun a comand on a failed node is made. I<-Z> implies
I<-z>. This option makes possible to organize clusters over
unreliable networks/hardware.
No EINTR wrappers anymore (iread, xread etc.). SIGCHLD and SIGALRM
are blocked most of the time. They are unblocked before select(2)
and blocked just after it. SA_RESTART is not used anymore.
Minor clean-ups in Makefile.
======================================================================
Version 0.12.1, by Aleksey Cheusov, Tue, 23 Dec 2008 20:33:20 +0200
FIX: support for -z appeared in paexec-0.12.0 was actually
incomplete :-/ Now everything should be fixed. More regression
tests.
Makefile: adapted for FreeBSD make which doesn't support .PARSEDIR
======================================================================
Version 0.12.0, by Aleksey Cheusov, Mon, 22 Dec 2008 21:48:01 +0200
ADDED: -z option. If applied, read/write(2) operations from/to nodes
becomes not critical. In case paexec has lost connection to the
node, it will reassign failed task to another node and, if -s option
applied, will output string "fatal" to stdout. This makes paexec
resistant to the I/O errors, as a result you can create paexec
clusters even over network consisting of unreliable hosts
(Internet?). Failed hosts are marked as such and will not be used
during the current run of paexec. NOTE: "success", "failure" and
"fatal" strings should be used to process the output of 'paexec -s'.
select(2) do not listen stdin (fd=0) anymore. Blocking read(2) is
used to read tasks from stdin.
Makefile: CPPFLAGS -> CFLAGS, FreeBSD make Mk scripts
don't use CPPFLAGS.
======================================================================
Version 0.11.0, by Aleksey Cheusov, Sat, 25 Oct 2008 17:49:47 +0300
paexec -s: before beginning actual work an input tasks graph is
checked for cycles. If they are detected, paexec exits with error.
minor fix in man page
======================================================================
Version 0.10.1, by Aleksey Cheusov, Sat, 25 Oct 2008 02:00:55 +0300
This version of paexec was sucessfully tested on the following
platforms:
NetBSD/x86
NetBSD/Alpha(64bit)
Linux/x86
Linux/x86-64
Interix-3.5/x86
FreeBSD/x86
Solaris-10/x86
Minor fix for rhomb-like dependencies (paexec -s).
Suppose A depends on (B1 and B2), (B1 and B2) depends on C.
If C fails, earlier paexec versions produced
the following output
failure
C B1 A B2 A
That is, A was output twice. Now it is fixed and A
is printed only once. New regression test for this.
By default(!) getopt_long(3) is enabled on the following platforms
(macro): __NetBSD__, __FreeBSD__, __OpenBSD__, __DragonFly__,
__linux__ and __APPLE__. On others only short options provided by
getopt(3) are used. This means that long option may to not work
on your platform.
fix in manual page: accept(2) -> select(2)
code clean-ups
.sinclude removed from Makefile
tests/test.sh:
diff -U ---> standard diff -C10
gawk appeared by mistake removed
======================================================================
Version 0.10.0, by Aleksey Cheusov, Sun, 31 Aug 2008 21:37:59 +0300
Lots of new regression tests
README file: 'make test' is documented
ADDED: -s option
Partially ordered set of tasks are read from stdin.
Instead of autonomous tasks, graph of the tasks is read from
stdin. In this mode every task can either FAIL or SUCCEED. As
always an empty line output by command means end of task. The
line before it shows an EXIT STATUS of the task. The word
"failure" means failure, "success" - success.
See examples/1_div_x/1_div_X_cmd for the sample. An input line
(paexec's stdin) should contain either single task without
spaces inside or two tasks separated by single space character,
e.g. task1<SPC>task2. task1<SPC>task2 line means that task1
must be done before task2 and it is mandatory, that is if task1
fail all dependent tasks (including task2) are also failed
recursively. Tasks having dependencies are started only after
all dependencies are succeeded. When a task succeeds paexec
outputs "success" word just before end_of_task marker (see -e
or -E), otherwise "failure" word is output followed by a list
of tasks failed because of it.
Samples:
tasks (examples/make_package/make_package_tasks file)
textproc/dictem
devel/autoconf wip/libmaa
devel/gmake wip/libmaa
wip/libmaa wip/dict-server
wip/libmaa wip/dict-client
devel/m4 wip/dict-server
devel/byacc wip/dict-server
devel/byacc wip/dict-client
devel/flex wip/dict-server
devel/flex wip/dict-client
devel/glib2
devel/libjudy
command (examples/make_package/make_package_cmd__flex)
#!/usr/bin/awk -f
{
print $0 # print a package name
if ($0 == "devel/flex")
print "failure" # cannot build flex ;-)
else
print "success" # all other packages are ok
print "" # end of task marker
fflush()
}
output of "paexec -s -l -c make_package_cmd__flex -n +10 \
< make_package_tasks"
3 devel/autoconf
3 success
4 devel/gmake
4 success
7 devel/m4
7 success
8 devel/byacc
8 success
9 devel/flex
9 failure
9 devel/flex wip/dict-server wip/dict-client
10 devel/glib2
10 success
11 devel/libjudy
11 success
1 textproc/dictem
1 success
2 wip/libmaa
2 success
======================================================================
Version 0.9.0, by Aleksey Cheusov, Sun, 15 Jun 2008 10:23:52 +0300
-t '' means "no transport". This significantly simplifies writing
shell scripts with paexec. Added: tests for this case.
paexec has no limited internal buffers anymore. All they are
resized automatically as it is needed. PAEXEC_BUFSIZE environment
variable sets an *initial* buffer size, not *maximum* one.
README: note about non-standard function getopt_long, and advice
how to build paexec on platforms with no getopt_long support
(HP-UX, Solaris etc.).
More regressions tests
paexec.1: minor corrections.
'make test' fix: In case regression test fails, 'make test' exits
with non-zero exit status.
paexec can be built with ancient version pmake-1.45 (found in som
Linux distributions).
paexec -h|--help outputs messages to stderr - my new religion :-)
======================================================================
Version 0.8.0, by Aleksey Cheusov, Tue, 4 Mar 2008 00:32:56 +0200
New options implemented: -E, -i and -I
See manual page for details.
Fixes and minor improvements in the documentation
======================================================================
Version 0.7.0, by Aleksey Cheusov, Tue, 26 Feb 2008 23:57:15 +0200
new -e|--eot option implemented.
It prints the end-of-task marker (an empty line) to stdout.
See manual page.
WARNS=4 to see compilation warnings.
several gcc warnings fixed.
fixed: '-n +0' and '-n ""' now fails with error message.
minor fix in man page
minor clean-ups
======================================================================
Version 0.6.0, by Aleksey Cheusov, Thu, 24 Jan 2008 21:31:26 +0200
First publicly available release
======================================================================
|