1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Parallel Workers Running in Linux Containers</title>
<style>
body {
font-family: sans-serif;
line-height: 1.6;
padding-left: 3ex;
padding-right: 3ex;
background-color: white;
color: black;
}
a {
color: #4183C4;
text-decoration: none;
}
h1, h2, h3 {
margin: 2ex 0 1ex;
padding: 0;
font-weight: bold;
-webkit-font-smoothing: antialiased;
cursor: text;
position: relative;
}
h2 {
border-bottom: 1px solid #cccccc;
}
code {
margin: 0 2px;
padding: 0 5px;
white-space: nowrap;
border: 1px solid #eaeaea;
background-color: #f8f8f8;
border-radius: 3px;
}
pre code {
margin: 0;
padding: 0;
white-space: pre;
border: none;
background: transparent;
}
pre {
background-color: #f8f8f8;
border: 1px solid #cccccc;
line-height: 2.5x;
overflow: auto;
padding: 0.6ex 1ex;
border-radius: 3px;
}
pre code {
background-color: transparent;
border: none;
}
</style>
</head>
<body>
<h1>Parallel Workers Running in Linux Containers</h1>
<!--
%\VignetteIndexEntry{Parallel Workers Running in Linux Containers}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{Docker}
%\VignetteKeyword{Apptainer}
%\VignetteEngine{parallelly::selfonly}
-->
<h1>Introduction</h1>
<p>This vignette shows how to set up parallel workers running in Linux
containers, e.g. Docker (<a href="https://www.docker.com/">https://www.docker.com/</a>), Apptainer
(<a href="https://apptainer.org/">https://apptainer.org/</a>), and udocker
(<a href="https://indigo-dc.github.io/udocker/">https://indigo-dc.github.io/udocker/</a>).</p>
<h1>Examples</h1>
<h2>Example: Two parallel workers running in Docker</h2>
<p>This example sets up two parallel workers running Docker image
‘rocker/r-parallel’ (<a href="https://hub.docker.com/r/rocker/r-parallel">https://hub.docker.com/r/rocker/r-parallel</a>).</p>
<pre><code class="language-r">library(parallelly)
cl <- makeClusterPSOCK(
rep("localhost", times = 2L),
## Launch Rscript inside Linux container via Docker
rscript = c(
"docker", "run", "--net=host", "rocker/r-parallel",
"Rscript"
),
## IMPORTANT: Because Docker runs inside a virtual machine (VM) on macOS
## and MS Windows (not Linux), when the R worker tries to connect back to
## the default 'localhost' it will fail, because the main R session is
## not running in the VM, but outside on the host. To reach the host on
## macOS and MS Windows, make sure to use master = "host.docker.internal"
master = if (.Platform$OS.type == "unix") NULL else "host.docker.internal",
)
print(cl)
#> Socket cluster with 2 nodes where 2 nodes are on host 'localhost'
#> (R version 4.3.3 (2024-02-29), platform x86_64-pc-linux-gnu)
</code></pre>
<h2>Example: Two parallel workers running in Apptainer</h2>
<p>This example shows how to set up two parallel workers running Docker
image ‘rocker/r-parallel’
(<a href="https://hub.docker.com/r/rocker/r-parallel">https://hub.docker.com/r/rocker/r-parallel</a>) via Apptainer
(<<a href="https://apptainer.org/">https://apptainer.org/</a>).</p>
<pre><code class="language-r">library(parallelly)
cl <- makeClusterPSOCK(
rep("localhost", times = 2L),
## Launch Rscript inside Linux container via Apptainer
rscript = c(
"apptainer", "exec", "docker://rocker/r-parallel",
"Rscript"
)
)
print(cl)
#> Socket cluster with 2 nodes where 2 nodes are on host 'localhost'
#> (R version 3.6.1 (2019-07-05), platform x86_64-pc-linux-gnu)
</code></pre>
<h2>Example: Two parallel workers running in udocker</h2>
<p>This example shows how to set up two parallel workers running Docker
image ‘rocker/r-parallel’
(<a href="https://hub.docker.com/r/rocker/r-parallel">https://hub.docker.com/r/rocker/r-parallel</a>) via udocker
(<a href="https://indigo-dc.github.io/udocker/">https://indigo-dc.github.io/udocker/</a>).</p>
<pre><code class="language-r">library(parallelly)
cl <- makeClusterPSOCK(
rep("localhost", times = 2L),
## Launch Rscript inside Linux container via Docker
rscript = c(
"udocker", "--quiet", "run", "rocker/r-parallel",
"Rscript"
)
)
print(cl)
#> Socket cluster with 2 nodes where 2 nodes are on host 'localhost'
#> (R version 3.6.1 (2019-07-05), platform x86_64-pc-linux-gnu)
</code></pre>
</body>
</html>
|
