1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
|
.. highlight:: console
.. _installation-ref:
Installation
============
This document describes how to prepare for and install Toil.
.. _dependencyPrep:
Installing System-Level Dependencies
------------------------------------
Toil by itself only needs Python 3.8 or newer.
However, to run CWL and WDL workflows, you need a container engine for running containers. Toil is able to use either `Singularity`_ or `Docker`_. So make sure to install one of those first and configure your system so that `your user has permission to run containers`_.
.. admonition:: Docker Desktop for Mac
If using the proprietary `Docker Desktop for Mac`_, make sure to set your "file sharing implementation" in the ``General`` section of the settings to ``VirtIO``. The default ``gRPC FUSE`` implentation sometimes makes containers see recently created files as empty, which breaks Toil's ability to run containers properly.
.. _Singularity: https://wiki.debian.org/singularity
.. _Docker: https://docs.docker.com/engine/install/
.. _your user has permission to run containers: https://askubuntu.com/a/1389518
.. _Docker Desktop for Mac: https://docs.docker.com/desktop/install/mac-install/
.. _venvPrep:
Preparing Your Python Runtime Environment
-----------------------------------------
It is recommended to install Toil into a Python `virtual environment`_. This is useful
for automatically deploying Python workflows, and is the only supported way to
install Toil for Toil development.
.. _virtual environment: https://virtualenv.pypa.io/en/stable/
If not already present, please install the latest Python ``virtualenv`` using pip_::
$ pip install --user virtualenv
And create a virtual environment called ``venv`` somewhere convenient, usch as under ``~``::
$ python3 -m virtualenv ~/venv
.. _pip: https://pip.readthedocs.io/en/latest/installing/
Now that you've created your virtualenv, activate your virtual environment::
$ source ~/venv/bin/activate
Basic Installation
------------------
If all you want to do is run workflows, you can install Toil with frontends for CWL and WDL like this::
$ pip install toil[cwl,wdl]
If you need only the basic version of Toil for Python workflows, it can be installed like this::
$ pip install toil
Now you're ready to run :ref:`your first Toil workflow <quickstart>`!
(If you need any more extra features, such as AWS support, don't do this yet and instead see :ref:`extras`.)
.. _extras:
Installing Toil with Extra Features
-----------------------------------
Some optional features, called *extras*, are not included in the basic
installation of Toil. To install Toil with all its bells and whistles, first
install any necessary headers and libraries (`python-dev`_, `libffi-dev`_). Then run ::
$ pip install toil[aws,google,mesos,encryption,cwl,wdl,kubernetes,server]
or ::
$ pip install toil[all]
Here's what each extra provides:
+----------------+------------------------------------------------------------+
| Extra | Description |
+================+============================================================+
| ``all`` | Installs all extras (though htcondor is linux-only and |
| | will be skipped if not on a linux computer). |
+----------------+------------------------------------------------------------+
| ``aws`` | Provides support for managing a cluster on Amazon Web |
| | Service (`AWS`_) using Toil's built in :ref:`clusterUtils`.|
| | Clusters can scale up and down automatically. |
| | It also supports storing workflow state. |
+----------------+------------------------------------------------------------+
| ``google`` | Experimental. Stores workflow state in `Google Cloud |
| | Storage`_. |
+----------------+------------------------------------------------------------+
| ``mesos`` | Provides support for running Toil on an `Apache Mesos`_ |
| | cluster. Note that running Toil on other batch systems |
| | does not require an extra. The ``mesos`` extra requires |
| | the following native dependencies: |
| | |
| | * `Apache Mesos`_ (Tested with Mesos v1.0.0) |
| | * :ref:`Python headers and static libraries <python-dev>` |
| | |
| | .. important:: |
| | If launching toil remotely on a mesos instance, |
| | to install Toil with the ``mesos`` extra in a |
| | virtualenv, be sure to create that virtualenv with the |
| | ``--system-site-packages`` flag (only use remotely!):: |
| | |
| | $ virtualenv ~/venv --system-site-packages |
| | |
| | Otherwise, you'll see something like this: |
| | |
| | .. code-block:: python |
| | |
| | ImportError: No module named mesos.native |
| | |
+----------------+------------------------------------------------------------+
| ``htcondor`` | Support for the htcondor batch system. This currently is |
| | a linux only extra. |
+----------------+------------------------------------------------------------+
| ``encryption`` | Provides client-side encryption for files stored in the |
| | AWS job store. This extra requires the |
| | following native dependencies: |
| | |
| | * :ref:`Python headers and static libraries <python-dev>` |
| | * :ref:`libffi headers and library <libffi-dev>` |
+----------------+------------------------------------------------------------+
| ``cwl`` | Provides support for running workflows written using the |
| | `Common Workflow Language`_. |
+----------------+------------------------------------------------------------+
| ``wdl`` | Provides support for running workflows written using the |
| | `Workflow Description Language`_. This extra has no native |
| | dependencies. |
+----------------+------------------------------------------------------------+
| ``kubernetes`` | Provides support for running workflows written using a |
| | `Kubernetes`_ cluster. |
+----------------+------------------------------------------------------------+
| ``server`` | Provides support for Toil server mode, including support |
| | for the GA4GH `Workflow Execution Service`_ API. |
+----------------+------------------------------------------------------------+
Some extras can't install without additional dependencies. If you need any of these extras, make sure to install their dependencies first!
.. _python-dev:
.. topic:: Python headers and static libraries
Needed for the ``mesos``, ``aws``, ``google``, and ``encryption`` extras.
On Ubuntu::
$ sudo apt-get install build-essential python-dev
On macOS::
$ xcode-select --install
.. _libffi-dev:
.. topic:: Encryption specific headers and library
Needed for the ``encryption`` extra.
On Ubuntu::
$ sudo apt-get install libssl-dev libffi-dev
On macOS::
$ brew install libssl libffi
Or see `Cryptography`_ for other systems.
.. _AWS: https://aws.amazon.com/
.. _Apache Mesos: https://mesos.apache.org/gettingstarted/
.. _Google Cloud Storage: https://cloud.google.com/storage/
.. _Workflow Description Language: https://software.broadinstitute.org/wdl/
.. _Workflow Execution Service: https://ga4gh.github.io/workflow-execution-service-schemas/docs/
.. _Cryptography: https://cryptography.io/en/latest/installation/
.. _Homebrew: http://brew.sh/
.. _Kubernetes: https://kubernetes.io/docs/concepts/overview/
.. _buildFromSource:
Installing Plugins
--------------------
Toil also supports plugins that allow Toil to run on different types of batch systems.
To install a plugin from pypi, simply run::
$ pip install [toil-batchsystem-plugin]
To use the batch system, pass the batch system name to the ``--batchSystem`` argument::
$ python sort.py --batchSystem=[batchsystem_name] ...
The current batch system plugins are:
- Task Execution Service (`TES <https://ga4gh.github.io/task-execution-schemas/docs/>`_): `toil_batch_system_tes <https://github.com/adamnovak/toil_batch_system_tes>`_
Building from Source
--------------------
If developing with Toil, you will need to build from source. This allows changes you
make to Toil to be reflected immediately in your runtime environment.
First, clone the source::
$ git clone https://github.com/DataBiosphere/toil.git
$ cd toil
Then, create and activate a virtualenv::
$ virtualenv venv
$ . venv/bin/activate
From there, you can list all available Make targets by running ``make``.
First and foremost, we want to install Toil's build requirements (these are
additional packages that Toil needs to be tested and built but not to be run)::
$ make prepare
Now, we can install Toil in development mode (such that changes to the
source code will immediately affect the virtualenv)::
$ make develop
Or, to install with support for all optional :ref:`extras`::
$ make develop extras=[aws,mesos,google,encryption,cwl]
Or::
$ make develop extras=[all]
To build the docs, run ``make develop`` with all extras followed by ::
$ make docs
To run a quick batch of tests (this should take less than 30 minutes)
run ::
$ export TOIL_TEST_QUICK=True; make test
For more information on testing see :ref:`runningTests`.
|
