1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
Metadata-Version: 2.2
Name: cwl-utils
Version: 0.37
Author-email: Common Workflow Language project contributors <common-workflow-language@googlegroups.com>
License: Apache 2.0
Project-URL: Documentation, https://cwl-utils.readthedocs.io/
Project-URL: Homepage, https://www.commonwl.org
Project-URL: Repository, https://github.com/common-workflow-language/cwl-utils
Project-URL: Issues, https://github.com/common-workflow-language/cwl-utils/issues
Project-URL: Changelog, https://github.com/common-workflow-language/cwl-utils/releases
Project-URL: Related Tools, https://www.commonwl.org/tools/
Keywords: cwl,commonwl,common-workflow-language,sciworkflows
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: File Formats
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.9
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: cwl-upgrader>=1.2.3
Requires-Dist: packaging
Requires-Dist: rdflib
Requires-Dist: requests
Requires-Dist: schema-salad<9,>=8.8.20250205075315
Requires-Dist: ruamel.yaml<0.19,>=0.17.6
Requires-Dist: typing_extensions; python_version < "3.10"
Provides-Extra: pretty
Requires-Dist: cwlformat; extra == "pretty"
Provides-Extra: testing
Requires-Dist: pytest<9; extra == "testing"
Requires-Dist: pytest-mock; extra == "testing"
|Linux Build Status| |Code coverage| |Documentation Status|
.. |Linux Build Status| image:: https://github.com/common-workflow-language/cwl-utils/actions/workflows/ci-tests.yml/badge.svg?branch=main
:target: https://github.com/common-workflow-language/cwl-utils/actions/workflows/ci-tests.yml
.. |Code coverage| image:: https://codecov.io/gh/common-workflow-language/cwl-utils/branch/main/graph/badge.svg
:target: https://codecov.io/gh/common-workflow-language/cwl-utils
.. |Documentation Status| image:: https://readthedocs.org/projects/cwl-utils/badge/?version=latest
:target: https://cwl-utils.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
cwl-utils
---------
Python Utilities and Autogenerated Classes for loading and parsing `CWL
v1.0 <https://github.com/common-workflow-language/cwl-utils/blob/main/cwl_utils/parser/v1_0.py>`__,
`CWL
v1.1 <https://github.com/common-workflow-language/cwl-utils/blob/main/cwl_utils/parser/v1_1.py>`__,
and `CWL
v1.2 <https://github.com/common-workflow-language/cwl-utils/blob/main/cwl_utils/parser/v1_2.py>`__
documents.
Requires Python 3.8+
Installation
------------
::
pip3 install cwl-utils
To install from source::
git clone https://github.com/common-workflow-language/cwl-utils.git
cd cwl-utils
pip3 install .
Usage
-----
Pull the all referenced software container images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-docker-extract`` is useful to cache or pre-pull all software
container images referenced in a CWL CommandLineTool or CWL Workflow
(including all referenced CommandLineTools and sub-Workflows and so on).
The default behaviour is to use the Docker engine to download and save
the software container images in Docker format.
.. code:: bash
cwl-docker-extract path_to_my_workflow.cwl
.. code:: bash
cwl-docker-extract --dir DIRECTORY path_to_my_workflow.cwl
Or you can use the Singularity software container engine to download and
save the software container images and convert them to the Singularity
format at the same time.
.. code:: bash
cwl-docker-extract --singularity --dir DIRECTORY path_to_my_workflow.cwl
Print all referenced software packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-cite-extract`` prints all software packages found (recursively) in the
specified CWL document.
Currently the package name and any listed specs and version field are printed
for all ``SoftwareRequirement`` s found.
.. code:: bash
cwl-cite-extract path_to_my_workflow.cwl
Replace CWL Expressions with concrete steps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-expression-refactor`` refactors CWL documents so that any CWL Expression
evaluations are separate steps (either CWL ExpressionTools or CWL CommandLineTools.)
This allows execution by CWL engines that do not want to support inline expression
evaluation outside of concrete steps, or do not want to directly support CWL's
optional ``InlineJavascriptRequirement`` at all.
.. code:: bash
cwl-expression-refactor directory/path/to/save/outputs path_to_my_workflow.cwl [more_workflows.cwl]
Split a packed CWL document
~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-graph-split`` splits a packed CWL document file into multiple files.
Packed CWL documents use the $graph construct to contain multiple CWL Process
objects (Workflow, CommandLineTool, ExpressionTool, Operation). Typically
packed CWL documents contain a CWL Workflow under the name "main" and the
workflow steps (including any sub-workflows).
.. code:: bash
cwl-graph-split --outdir optional/directory/path/to/save/outputs path_to_my_workflow.cwl
Normalize a CWL document
~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-normalizer`` normalizes one or more CWL document so that for each document,
a JSON format CWL document is produces with it and all of its dependencies packed
together, upgrading to CWL v1.2, as needed. Can optionally refactor CWL
Expressions into separate steps in the manner of cwl-expression-refactor.
.. code:: bash
cwl-normalizer directory/path/to/save/outputs path_to_my_workflow.cwl [more_workflows.cwl]
Generate for Workflow Parameters from a CWL document
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cwl-inputs-schema-gen`` generates a JSON Schema for workflow input parameters from a CWL document.
.. code:: bash
cwl-inputs-schema-gen path_to_my_workflow.cwl
Using the CWL Parsers
~~~~~~~~~~~~~~~~~~~~~
.. code:: python
from pathlib import Path
from ruamel import yaml
import sys
from cwl_utils.parser import load_document_by_uri, save
# File Input - This is the only thing you will need to adjust or take in as an input to your function:
cwl_file = Path("testdata/md5sum.cwl") # or a plain string works as well
# Import CWL Object
cwl_obj = load_document_by_uri(cwl_file)
# View CWL Object
print("List of object attributes:\n{}".format("\n".join(map(str, dir(cwl_obj)))))
# Export CWL Object into a built-in typed object
saved_obj = save(cwl_obj)
print(f"Export of the loaded CWL object: {saved_obj}.")
Development
-----------
Regenerate parsers
~~~~~~~~~~~~~~~~~~
To regenerate install the ``schema_salad`` package and run:
``cwl_utils/parser/cwl_v1_0.py`` was created via
``schema-salad-tool --codegen python https://github.com/common-workflow-language/common-workflow-language/raw/codegen/v1.0/extensions.yml --codegen-parser-info "org.w3id.cwl.v1_0" > cwl_utils/parser/cwl_v1_0.py``
``cwl_utils/parser/cwl_v1_1.py`` was created via
``schema-salad-tool --codegen python https://github.com/common-workflow-language/cwl-v1.1/raw/codegen/extensions.yml --codegen-parser-info "org.w3id.cwl.v1_1" > cwl_utils/parser/cwl_v1_1.py``
``cwl_utils/parser/cwl_v1_2.py`` was created via
``schema-salad-tool --codegen python https://github.com/common-workflow-language/cwl-v1.2/raw/codegen/extensions.yml --codegen-parser-info "org.w3id.cwl.v1_2" > cwl_utils/parser/cwl_v1_2.py``
Release
~~~~~~~
To release CWLUtils, bump the version in ``cwl_utils/__meta__.py``, and
tag that commit with the new version. The
`gh-action-pypi-publish <https://github.com/pypa/gh-action-pypi-publish>`__
should release that tag.
|
