1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047
|
<!--
%\VignetteIndexEntry{progressr: An Introduction}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{Rprofile}
%\VignetteKeyword{Renviron}
%\VignetteEngine{progressr::selfonly}
-->
<!-- DO NOT EDIT THIS FILE! Edit 'OVERVIEW.md' instead and then rebuild this file with 'make vigs' -->
The **[progressr]** package provides a minimal API for reporting
progress updates in [R](https://www.r-project.org/). The design is to
separate the representation of progress updates from how they are
presented. What type of progress to signal is controlled by the
developer. How these progress updates are rendered is controlled by
the end user. For instance, some users may prefer visual feedback
such as a horizontal progress bar in the terminal, whereas others may
prefer auditory feedback.
<img src="imgs/three_in_chinese.gif" alt="Three strokes writing three in Chinese" style="float: right; margin-right: 1ex; margin-left: 1ex;"/>
Design motto:
> The developer is responsible for providing progress updates but it's
> only the end user who decides if, when, and how progress should be
> presented. No exceptions will be allowed.
## Two Minimal APIs - One For Developers and One For End-Users
<div style="overflow: hidden">
<div style="float: left">
<table style="border: 1px solid #999; box-shadow: 2px 2px 2px #999;">
<tr><th>Developer's API</th></tr>
<tr style="vertical-align: top">
<td>
<p>
1. Set up a progressor with a certain number of steps:
</p>
<pre>
p <- progressor(nsteps)
p <- progressor(along = x)
</pre>
<p>
2. Signal progress:
</p>
<pre>
p() # one-step progress
p(amount = 0) # "still alive"
p("loading ...") # pass on a message
</pre>
</td>
</tr>
</table>
</div>
<div style="float: left"> </div>
<div style="float: left">
<table style="border: 1px solid #999; box-shadow: 2px 2px 2px #999;">
<tr><th>End-user's API</th></tr>
<tr style="vertical-align: top">
<td>
<p>
1a. Subscribe to progress updates from everywhere:
</p>
<pre>
handlers(global = TRUE)
y <- slow_sum(1:5)
y <- slow_sum(6:10)
</pre>
<p>
1b. Subscribe to a specific expression:
</p>
<pre>
with_progress({
y <- slow_sum(1:5)
y <- slow_sum(6:10)
})
</pre>
<p>
2. Configure how progress is presented:
</p>
<pre>
handlers("progress")
handlers("txtprogressbar", "beepr")
handlers(handler_pbcol(enable_after = 3.0))
handlers(handler_progress(complete = "#"))
</pre>
</td>
</table>
</div>
</div>
## A simple example
Assume that we have a function `slow_sum()` for adding up the values
in a vector. It is so slow, that we like to provide progress updates
to whoever might be interested in it. With the **progressr** package,
this can be done as:
```r
slow_sum <- function(x) {
p <- progressr::progressor(along = x)
sum <- 0
for (kk in seq_along(x)) {
Sys.sleep(0.1)
sum <- sum + x[kk]
p(message = sprintf("Adding %g", x[kk]))
}
sum
}
```
Note how there are _no_ arguments in the code that specifies how
progress is presented. The only task for the developer is to decide
on where in the code it makes sense to signal that progress has been
made. As we will see next, it is up to the end user of this code to
decide whether they want to receive progress updates or not, and, if
so, in what format.
### Without reporting on progress
When calling this function as in:
```r
> y <- slow_sum(1:10)
> y
[1] 55
>
```
it will behave as any function and there will be no progress
updates displayed.
### Reporting on progress
If we are only interested in progress for a particular call, we can
do:
```r
> library(progressr)
> with_progress(y <- slow_sum(1:10))
|==================== | 40%
```
However, if we want to report on progress from _every_ call, wrapping
the calls in `with_progress()` might become too cumbersome. If so, we
can enable the global progress handler:
```r
> library(progressr)
> handlers(global = TRUE)
```
so that progress updates are reported on wherever signaled, e.g.
```r
> y <- slow_sum(1:10)
|==================== | 40%
> y <- slow_sum(10:1)
|======================================== | 80%
```
This requires R 4.0.0 or newer. To disable this again, do:
```r
> handlers(global = FALSE)
```
In the below examples, we will assume `handlers(global = TRUE)` is
already set.
## Customizing how progress is reported
### Terminal-based progress bars
The default is to present progress via `utils::txtProgressBar()`,
which is available on all R installations. It presents itself as an
ASCII-based horizontal progress bar in the R terminal. This is
rendered as:

We can tweak this "txtprogressbar" handler to use red hearts for the
bar, e.g.
```r
handlers(handler_txtprogressbar(char = cli::col_red(cli::symbol$heart)))
```
which results in:

Another example is:
```r
handlers(handler_pbcol(
adjust = 1.0,
complete = function(s) cli::bg_red(cli::col_black(s)),
incomplete = function(s) cli::bg_cyan(cli::col_black(s))
))
```
which results in:

To change the default, to, say, `cli_progress_bar()` by the **[cli]**
package, set:
```r
handlers("cli")
```
This progress handler will present itself as:

To instead use `progress_bar()` by the **[progress]** package, set:
```r
handlers("progress")
```
This progress handler will present itself as:

To set the default progress handler, or handlers, in all your R
sessions, call `progressr::handlers(...)` in your
<code>~/.Rprofile</code> startup file.
### Auditory progress updates
Progress updates do not have to be presented visually. They can
equally well be communicated via audio. For example, using:
```r
handlers("beepr")
```
will present itself as sounds played at the beginning, while progressing, and at the end (using different **[beepr]** sounds). There will be _no_ output written to the terminal;
```r
> y <- slow_sum(1:10)
> y
[1] 55
>
```
### Concurrent auditory and visual progress updates
It is possible to have multiple progress handlers presenting progress
updates at the same time. For example, to get both visual and
auditory updates, use:
```r
handlers("txtprogressbar", "beepr")
```
### Silence all progress
To silence all progress updates, use:
```r
handlers("void")
```
### Further configuration of progress handlers
Above we have seen examples where the `handlers()` takes one or more
strings as input, e.g. `handlers(c("progress", "beepr"))`. This is
short for a more flexible specification where we can pass a list of
handler functions, e.g.
```r
handlers(list(
handler_progress(),
handler_beepr()
))
```
With this construct, we can make adjustments to the default behavior
of these progress handlers. For example, we can configure the
`format`, `width`, and `complete` arguments of
`progress::progress_bar$new()`, and tell **beepr** to use a different
`finish` sound and generate sounds at most every two seconds by
setting:
```r
handlers(list(
handler_progress(
format = ":spin :current/:total (:message) [:bar] :percent in :elapsed ETA: :eta",
width = 60,
complete = "+"
),
handler_beepr(
finish = "wilhelm",
interval = 2.0
)
))
```
## Sticky messages
As seen above, some progress handlers present the progress message as
part of its output, e.g. the "progress" handler will display the
message as part of the progress bar. It is also possible to "push"
the message up together with other terminal output. This can be done
by adding class attribute `"sticky"` to the progression signaled.
This works for several progress handlers that output to the terminal.
For example, with:
```r
slow_sum <- function(x) {
p <- progressr::progressor(along = x)
sum <- 0
for (kk in seq_along(x)) {
Sys.sleep(0.1)
sum <- sum + x[kk]
p(sprintf("Step %d", kk), class = if (kk %% 5 == 0) "sticky", amount = 0)
p(message = sprintf("Adding %g", x[kk]))
}
sum
}
```
we get
```r
> handlers("txtprogressbar")
> y <- slow_sum(1:30)
Step 5
Step 10
|==================== | 43%
```
and
```r
> handlers("progress")
> y <- slow_sum(1:30)
Step 5
Step 10
/ [===============>-------------------------] 43% Adding 13
```
## Use regular output as usual alongside progress updates
In contrast to other progress-bar frameworks, output from `message()`,
`cat()`, `print()` and so on, will _not_ interfere with progress
reported via **progressr**. For example, say we have:
```r
slow_sqrt <- function(xs) {
p <- progressor(along = xs)
lapply(xs, function(x) {
message("Calculating the square root of ", x)
Sys.sleep(2)
p(sprintf("x=%g", x))
sqrt(x)
})
}
```
we will get:
```r
> library(progressr)
> handlers(global = TRUE)
> handlers("progress")
> y <- slow_sqrt(1:8)
Calculating the square root of 1
Calculating the square root of 2
- [===========>-----------------------------------] 25% x=2
```
This works because **progressr** will briefly buffer any output
internally and only release it when the next progress update is
received just before the progress is re-rendered in the terminal.
This is why you see a two second delay when running the above example.
Note that, if we use progress handlers that do not output to the
terminal, such as `handlers("beepr")`, then output does not have to be
buffered and will appear immediately.
_Comment_: When signaling a warning using `warning(msg, immediate. =
TRUE)` the message is immediately outputted to the standard-error
stream. However, this is not possible to emulate when warnings are
intercepted using calling handlers, which are used by
`with_progress()`. This is a limitation of R that cannot be worked
around. Because of this, the above call will behave the same as
`warning(msg)` - that is, all warnings will be buffered by R
internally and released only when all computations are done.
## Support for progressr elsewhere
Note that progression updates by **progressr** is designed to work out
of the box for any iterator framework in R. Below is an set of
examples for the most common ones.
### Base R Apply Functions
```r
library(progressr)
handlers(global = TRUE)
my_fcn <- function(xs) {
p <- progressor(along = xs)
lapply(xs, function(x) {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# |==================== | 40%
```
### The foreach package
```r
library(foreach)
library(progressr)
handlers(global = TRUE)
my_fcn <- function(xs) {
p <- progressor(along = xs)
foreach(x = xs) %do% {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
}
}
my_fcn(1:5)
# |==================== | 40%
```
### The purrr package
```r
library(purrr)
library(progressr)
handlers(global = TRUE)
my_fcn <- function(xs) {
p <- progressor(along = xs)
map(xs, function(x) {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# |==================== | 40%
```
### The plyr package
```r
library(plyr)
library(progressr)
handlers(global = TRUE)
my_fcn <- function(xs) {
p <- progressor(along = xs)
llply(xs, function(x, ...) {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# |==================== | 40%
```
Note how this solution does not make use of **plyr**'s `.progress`
argument, because the above solution is more powerful and more
flexible, e.g. we have more control on progress updates and their
messages. However, if you prefer the traditional **plyr** approach,
you can use `.progress = "progressr"`, e.g. `y <- llply(..., .progress
= "progressr")`.
### The knitr package
When compiling ("knitting") an knitr-based vignette, for instance, via
`knitr::knit()`, **[knitr]** shows the progress of code chunks
processed thus far using a progress bar. In **knitr** (>= 1.42) [to
be released as of 2022-12-12], we can use **progressr** for this
progress reporting. To do this, set R option `knitr.progress.fun` as:
```r
options(knitr.progress.fun = function(total, labels) {
p <- progressr::progressor(total, on_exit = FALSE)
list(
update = function(i) p(sprintf("chunk: %s", labels[i])),
done = function() p(type = "finish")
)
})
```
This configures **knitr** to signal progress via the **progressr**
framework. To report on these, use:
```r
progressr::handlers(global = TRUE)
```
### Replace any cli progress bars with progressr updates
The **cli** package is used for progress reporting by many several
packages, notably tidyverse packages. For instance, in **purrr**, you
can do:
```r
y <- purrr::map(1:100, \(x) Sys.sleep(0.1), .progress = TRUE)
```
to report on progress via the **cli** package as `map()` is iterating
over the elements. Now, instead of using the default, built-in
**cli** progress bar, we can customize **cli** to report on progress
via **progressr** instead. To do this, set R option
`cli.progress_handlers` as:
```r
options(cli.progress_handlers = "progressr")
```
With this option set, **cli** will now report on progress according to
your `progressr::handlers()` settings. For example, with:
```r
progressr::handlers(c("beepr", "rstudio"))
```
will report on progress using **beepr** and the RStudio Console
progress panel.
To make **cli** report via **progressr** in all your R session, set
the above R option in your <code>~/.Rprofile</code> startup file.
_Note:_ A **cli** progress bar can have a "name", which can be
specfied in **purrr** function via argument `.progress`,
e.g. `.progress = "processing"`. This name is then displayed in front
of the progress bar. However, because the **progressr** framework
does not have a concept of progress "name", they are silently ignored
when using `options(cli.progress_handlers = "progressr")`.
## Parallel processing and progress updates
The **[future]** framework, which provides a unified API for parallel
and distributed processing in R, has built-in support for the kind of
progression updates produced by the **progressr** package. This means
that you can use it with for instance **[future.apply]**, **[furrr]**,
and **[foreach]** with **[doFuture]**, and **[plyr]** or
**[BiocParallel]** with **doFuture**. In contrast, _non-future_
parallelization methods such as **parallel**'s `mclapply()` and,
`parallel::parLapply()`, and **foreach** adapters like **doParallel**
do _not_ support progress reports via **progressr**.
### future_lapply() - parallel lapply()
Here is an example that uses `future_lapply()` of the **[future.apply]** package to parallelize on the local machine while at the same time signaling progression updates:
```r
library(future.apply)
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
future_lapply(xs, function(x, ...) {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
### foreach() with doFuture
Here is an example that uses `foreach()` of the **[foreach]** package
together with `%dofuture%` of the **[doFuture]** package to
parallelize while reporting on progress. This example parallelizes on
the local machine, it works alsof for remote machines:
```r
library(doFuture) ## %dofuture%
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
foreach(x = xs) %dofuture% {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
}
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
For existing code using the traditional `%dopar%` operators of the
**[foreach]** package, we can register the **[doFuture]** adaptor and
use the same **progressr** as above to progress updates;
```r
library(doFuture)
registerDoFuture() ## %dopar% parallelizes via future
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
foreach(x = xs) %dopar% {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
}
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
### future_map() - parallel purrr::map()
Here is an example that uses `future_map()` of the **[furrr]** package
to parallelize on the local machine while at the same time signaling
progression updates:
```r
library(furrr)
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
future_map(xs, function(x) {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
_Note:_ This solution does not involved the `.progress = TRUE`
argument that **furrr** implements. Because **progressr** is more
generic and because `.progress = TRUE` only supports certain future
backends and produces errors on non-supported backends, I recommended
to stop using `.progress = TRUE` and use the **progressr** package
instead.
### BiocParallel::bplapply() - parallel lapply()
Here is an example that uses `bplapply()` of the **[BiocParallel]**
package to parallelize on the local machine while at the same time
signaling progression updates:
```r
library(BiocParallel)
library(doFuture)
register(DoparParam()) ## BiocParallel parallelizes via %dopar%
registerDoFuture() ## %dopar% parallelizes via future
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
bplapply(xs, function(x) {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
})
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
### plyr::llply(..., .parallel = TRUE) with doFuture
Here is an example that uses `llply()` of the **[plyr]** package to
parallelize on the local machine while at the same time signaling
progression updates:
```r
library(plyr)
library(doFuture)
registerDoFuture() ## %dopar% parallelizes via future
plan(multisession)
library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")
my_fcn <- function(xs) {
p <- progressor(along = xs)
llply(xs, function(x, ...) {
Sys.sleep(6.0-x)
p(sprintf("x=%g", x))
sqrt(x)
}, .parallel = TRUE)
}
my_fcn(1:5)
# / [================>-----------------------------] 40% x=2
```
_Note:_ As an alternative to the above, recommended approach, one can
use `.progress = "progressr"` together with `.parallel = TRUE`. This
requires **plyr** (>= 1.8.7).
### Near-live versus buffered progress updates with futures
As of November 2020, there are four types of **future** backends that are known(*) to provide near-live progress updates:
1. `sequential`,
2. `multicore`,
3. `multisession`, and
4. `cluster` (local and remote)
Here "near-live" means that the progress handlers will report on
progress almost immediately when the progress is signaled on the
worker. For all other future backends, the progress updates are only
relayed back to the main machine and reported together with the
results of the futures. For instance, if `future_lapply(X, FUN)`
chunks up the processing of, say, 100 elements in `X` into eight
futures, we will see progress from each of the 100 elements as they
are done when using a future backend supporting "near-live" updates,
whereas we will only see those updated to be flushed eight times when
using any other types of future backends.
(*) Other future backends may gain support for "near-live" progress
updating later. Adding support for those is independent of the
**progressr** package. Feature requests for adding that support
should go to those future-backend packages.
## Note of caution - sending progress updates too frequently
Signaling progress updates comes with some overhead. In situation
where we use progress updates, this overhead is typically much smaller
than the task we are processing in each step. However, if the task we
iterate over is quick, then the extra time induced by the progress
updates might end up dominating the overall processing time. If that
is the case, a simple solution is to only signal progress updates
every n:th step. Here is a version of `slow_sum()` that signals
progress every 10:th iteration:
```
slow_sum <- function(x) {
p <- progressr::progressor(length(x) / 10)
sum <- 0
for (kk in seq_along(x)) {
Sys.sleep(0.1)
sum <- sum + x[kk]
if (kk %% 10 == 0) p(message = sprintf("Adding %g", x[kk]))
}
sum
}
```
The overhead of progress signaling may depend on context. For
example, in parallel processing with near-live progress updates via
'multisession' futures, each progress update is communicated via a
socket connections back to the main R session. These connections
might become clogged up if progress updates are too frequent.
## Progress updates in non-interactive mode ("batch mode")
When running R from the command line, R runs in a non-interactive mode
(`interactive()` returns `FALSE`). The default behavior of
**progressr** is to _not_ report on progress in non-interactive mode.
To reported on progress also then, set R options `progressr.enable` or
environment variable `R_PROGRESSR_ENABLE` to `TRUE`. For example,
```sh
$ Rscript -e "library(progressr)" -e "with_progress(y <- slow_sum(1:10))"
```
will _not_ report on progress, whereas
```sh
$ export R_PROGRESSR_ENABLE=TRUE
$ Rscript -e "library(progressr)" -e "with_progress(y <- slow_sum(1:10))"
```
will.
## Roadmap
Because this project is under active development, the progressr API is
currently kept at a very minimum. This will allow for the framework
and the API to evolve while minimizing the risk for breaking code that
depends on it. The roadmap for developing the API is roughly:
* [x] Provide minimal API for producing progress updates,
i.e. `progressor()`, `with_progress()`, `handlers()`
* [x] Add support for global progress handlers removing the need for
the user having to specify `with_progress()`,
i.e. `handlers(global = TRUE)` and `handlers(global = FALSE)`
* [ ] Make it possible to create a progressor also in the global
environment (see 'Known issues' below)
* [ ] Add support for nested progress updates
* [ ] Add API to allow users and package developers to design
additional progression handlers
For a more up-to-date view on what features might be added, see
<https://github.com/futureverse/progressr/issues>.
## Appendix
### Known issues
#### A progressor cannot be created in the global environment
It is not possible to create a progressor in the global environment,
e.g. in the the top-level of a script. It has to be created inside a
function, within `with_progress({ ... })`, `local({ ... })`, or a
similar construct. For example, the following:
```r
library(progressr)
handlers(global = TRUE)
xs <- 1:5
p <- progressor(along = xs)
y <- lapply(xs, function(x) {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
})
```
results in an error if tried:
```
Error in progressor(along = xs) :
A progressor must not be created in the global environment unless wrapped in a
with_progress() or without_progress() call. Alternatively, create it inside a
function or in a local() environment to make sure there is a finite life span
of the progressor
```
The solution is to wrap it in a `local({ ... })` call, or more
explicitly, in a `with_progress({ ... })` call:
```r
library(progressr)
handlers(global = TRUE)
xs <- 1:5
with_progress({
p <- progressor(along = xs)
y <- lapply(xs, function(x) {
Sys.sleep(0.1)
p(sprintf("x=%g", x))
sqrt(x)
})
})
# |==================== | 40%
```
The main reason for this is to limit the life span of each progressor.
If we created it in the global environment, there is a significant
risk it would never finish and block all of the following progressors.
#### The global progress handler cannot be set everywhere
It is _not_ possible to call `handlers(global = TRUE)` in all
circumstances. For example, it cannot be called within `tryCatch()`
and `withCallingHandlers()`;
```r
> tryCatch(handlers(global = TRUE), error = identity)
Error in globalCallingHandlers(NULL) :
should not be called with handlers on the stack
```
This is not a bug - neither in **progressr** nor in R itself. It's due
to a conservative design on how _global_ calling handlers should work
in R. If it allowed, there's a risk we might end up getting weird and
unpredictable behaviors when messages, warnings, errors, and other
types of conditions are signaled.
Because `tryCatch()` and `withCallingHandlers()` is used in many
places throughout base R, this means that we also cannot call
`handlers(global = TRUE)` as part of a package's startup process,
e.g. `.onLoad()` or `.onAttach()`.
Another example of this error is if `handlers(global = TRUE)` is used
inside package vignettes and dynamic documents such as Rmarkdown. In
such cases, the global progress handler has to be enabled _prior_ to
processing the document, e.g.
```r
> progressr::handlers(global = TRUE)
> rmarkdown::render("input.Rmd")
```
### Under the hood
When using the **progressr** package, progression updates are
communicated via R's condition framework, which provides methods for
creating, signaling, capturing, muffling, and relaying conditions.
Progression updates are of classes `progression` and
`immediateCondition`(\*). The below figure gives an example how
progression conditions are created, signaled, and rendered.
(\*) The `immediateCondition` class of conditions are relayed as soon
as possible by the **[future]** framework, which means that
progression updates produced in parallel workers are reported to the
end user as soon as the main R session have received them.

_Figure: Sequence diagram illustrating how signaled progression
conditions are captured by `with_progress()`, or the global
progression handler, and relayed to the two progression handlers
'progress' (a progress bar in the terminal) and 'beepr' (auditory)
that the end user has chosen._
### Debugging
To debug progress updates, use:
```r
> handlers("debug")
> with_progress(y <- slow_sum(1:3))
[23:19:52.738] (0.000s => +0.002s) initiate: 0/3 (+0) '' {clear=TRUE, enabled=TRUE, status=}
[23:19:52.739] (0.001s => +0.000s) update: 0/3 (+0) '' {clear=TRUE, enabled=TRUE, status=}
[23:19:52.942] (0.203s => +0.002s) update: 0/3 (+0) '' {clear=TRUE, enabled=TRUE, status=}
[23:19:53.145] (0.407s => +0.001s) update: 0/3 (+0) '' {clear=TRUE, enabled=TRUE, status=}
[23:19:53.348] (0.610s => +0.002s) update: 1/3 (+1) 'P: Adding 1' {clear=TRUE, enabled=TRUE, status=}
M: Adding value 1
[23:19:53.555] (0.817s => +0.004s) update: 1/3 (+0) 'P: Adding 1' {clear=TRUE, enabled=TRUE, status=}
[23:19:53.758] (1.020s => +0.001s) update: 1/3 (+0) 'P: Adding 1' {clear=TRUE, enabled=TRUE, status=}
[23:19:53.961] (1.223s => +0.001s) update: 1/3 (+0) 'P: Adding 1' {clear=TRUE, enabled=TRUE, status=}
[23:19:54.165] (1.426s => +0.001s) update: 1/3 (+0) 'P: Adding 1' {clear=TRUE, enabled=TRUE, status=}
[23:19:54.368] (1.630s => +0.001s) update: 2/3 (+1) 'P: Adding 2' {clear=TRUE, enabled=TRUE, status=}
M: Adding value 2
[23:19:54.574] (1.835s => +0.003s) update: 2/3 (+0) 'P: Adding 2' {clear=TRUE, enabled=TRUE, status=}
[23:19:54.777] (2.039s => +0.001s) update: 2/3 (+0) 'P: Adding 2' {clear=TRUE, enabled=TRUE, status=}
[23:19:54.980] (2.242s => +0.001s) update: 2/3 (+0) 'P: Adding 2' {clear=TRUE, enabled=TRUE, status=}
[23:19:55.183] (2.445s => +0.001s) update: 2/3 (+0) 'P: Adding 2' {clear=TRUE, enabled=TRUE, status=}
[23:19:55.387] (2.649s => +0.001s) update: 3/3 (+1) 'P: Adding 3' {clear=TRUE, enabled=TRUE, status=}
[23:19:55.388] (2.650s => +0.003s) update: 3/3 (+0) 'P: Adding 3' {clear=TRUE, enabled=TRUE, status=}
M: Adding value 3
[23:19:55.795] (3.057s => +0.000s) shutdown: 3/3 (+0) 'P: Adding 3' {clear=TRUE, enabled=TRUE, status=ok}
```
[progressr]: https://cran.r-project.org/package=progressr
[beepr]: https://cran.r-project.org/package=beepr
[cli]: https://cran.r-project.org/package=cli
[progress]: https://cran.r-project.org/package=progress
[purrr]: https://cran.r-project.org/package=purrr
[future]: https://cran.r-project.org/package=future
[foreach]: https://cran.r-project.org/package=foreach
[future.apply]: https://cran.r-project.org/package=future.apply
[doParallel]: https://cran.r-project.org/package=doParallel
[doFuture]: https://cran.r-project.org/package=doFuture
[furrr]: https://cran.r-project.org/package=furrr
[knitr]: https://cran.r-project.org/package=knitr
[pbapply]: https://cran.r-project.org/package=pbapply
[pbmcapply]: https://cran.r-project.org/package=pbmcapply
[plyr]: https://cran.r-project.org/package=plyr
[BiocParallel]: https://www.bioconductor.org/packages/BiocParallel/
|