1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
|
==========================================================================
This file describe briefly different useful tools to administrate Mobyle.
==========================================================================
In this document we refer to MOBYLEHOME as the folder which contains the Mobyle installation core files.
1- mobdeploy : to deploy the programs and viewers interfaces
=============================================================
Mobyle allows administrators to deploy programs that are either *local* or *imported* and now viewers that are *local*.
The `mobdeploy` script is used to *deploy* them on the server, making them available to the public.
This script also handles their *indexing*.
Note: In the following, we will talk about Services which means the Programs & the Viewers.
Overview
:::::::::
Once a Mobyle server has been installed in a **MOBYLEHOME** directory on the system, the program definitions (i.e., the xml files that describe them to the Mobyle system) are stored in two directories:
- the MOBYLEHOME/Services/ directory which contains program (Programs/) and viewer (Viewers/) definitions that the administrator has downloaded from distributions,
- and the MOBYLEHOME/Local/Programs directory which contains custom program definitions that have either been created de novo by the server owner, or that he modified to fit his own needs.
In order to deploy the services, the service definitions are first tested to check that these definitions are valid.
Then, there is a predeployment step: the valid service definitions are copied to a temporary directory (tmpservices) the time necessary to generate the index.
(In case of validation problem, the previous service definition installed is temporary kept, if it exists.)
Finally, the whole content of this directory is transferred to a web-accessible directory defined by the server configuration in the **DOCUMENT_ROOT/PROGRAMS_URL** in case of Programs
or **DOCUMENT_ROOT/VIEWERS_URL** for Viewers.
Therefore, these files will be directly accessible from the Web on their corresponding url. This document refers to this directory as **HTDOCS**.
The imported program definitions are downloaded *via* the Web from the **HTDOCS** directory of the remote servers.
These program definitions are stored in **HTDOCS/programs/imports** directory.
The programs and viewers index are stored in the **HTDOCS**/index directory.
Configuration
::::::::::::::
* PROGRAMS *
The configuration of published/imported services is located in the Local/Config/Config.py file.
The deployment of local services (programs, workflows, viewers ) is configured by the **LOCAL_DEPLOY_INCLUDE**, and **LOCAL_DEPLOY_EXCLUDE** directives.
LOCAL_DEPLOY_INCLUDE and LOCAL_DEPLOY_EXCLUDE are python dictionaries with 3 keys programs, workflows and viewers associated to 3 lists of names.
these defined repectively the programs, workflows and viewers and can include jokers.
e.g.::
LOCAL_DEPLOY_INCLUDE = { 'programs' : [ 'golden', 'clustalw*', 'dnadist', 'fitch' ] ,
'workflows': [ '*' ] ,
'viewers' : [ 'jalview' , 'varna' ]
}
LOCAL_DEPLOY_EXCLUDE={'programs' : [ 'clustalw-profile' ] ,
'workflows': [ '*' ] ,
'viewers' : [ 'jalview' , 'varna' ]
}
* VIEWERS *
The important rule about viewer is that the viewer dependencies have to be available within a directory named like the
viewer xml description in MOBYLEHOME/Services/Viewers/.
For instance, for a viewer called 'jalview', MOBYLEHOME/Services/Viewers/ directory has to contain a 'jalview.xml' file and
a 'jalview' directory where are the corresponding dependencies: the archive 'jalviewApplet.jar'.
If the dependency directory is missing, the viewer won't be deployed.
The exportation of services to other portals is defined by the **EXPORTED_SERVICES** directive, which lists these services.
e.g.::
EXPORTED_SERVICES = [ 'golden', 'abiview' ]
The importation of services from other portals is defined by the **PORTALS** directive,
a dictionary that defines external portals the following way:
- the key is the nickname of the portal, as it will appear in the programs list of the portal
- for each portal:
+ url is the path to the directory that contains all the cgis (such as job_submit.py).
+ help is the e-mail of the remote portal maintainer, that receives help requests about his job.
+ repository is the path to the directory that contains all the published programs.
+ services is a dictionary with 2 keys
- 'programs' associated to the list of imported programs.
- 'workflows' associated to the list of imported workflows.
e.g.::
PORTALS={'portal1': {
'url': 'http://otherdomain.fr:port/cgi-bin/MobylePortal',
'help' : 'help@otherdomain.fr',
'repository': 'http://otherdomain.fr:port/MobyleData/',
services: { 'programs': ['clustalw-multialign'],
'workflows' : ['workflow1_of_portal1']
}
}
}
Usage and options
:::::::::::::::::
The ``mobdeploy`` script is located in the **$MOBYLEHOME/Tools** directory.
Before to run it, you should make sure that your current user has the permissions to read and write the files that belong to
the Mobyle user (e.g., apache user).
Here are the available commands:
deploy : deploy services available options [-s --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose]
clean : clean the repository available options [-s --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose]
index : generate indexes available options [-s --server, -p --programs , -w --workflows , -v --viewers, -V --verbose ]
And options:
-h --help
-p --programs : specify the names (comma separated list) of the programs on which the command is applied.
The keyword 'all' mean all programs as defined in Config.py
-w --workflows: specify the names (comma separated list) of the workflows on which the command is applied.
The keyword 'all' mean all workflows as defined in Config.py
-v --viewers : specify the names (comma separated list) of the viewers on which the command is applied.
The keyword 'all' mean all viewers as defined in Config.py
-s --server : specify the names (comma separated list) of the servers on which the command is applied.
The keyword 'all' mean all servers as defined in Config.py
-f --force : force to deploy a program even it is not imported.
This option is for debugging purpose only and shall not be used otherwise.
-V --verbose : increase the verbosity (there is 3 level of verbosity: warning , info , debug . default is warning)
Below are a few examples of how you can use the mobdeploy utility:
mobdeploy deploy install all programs, workflows and viewers from all servers according to the configuration.
mobdeploy -p blast2 deploy deploy the *blast2* program from all servers.
mobdeploy -p blast2 clean remove the *blast2* from all servers.
mobdeploy -s portal1 deploy deploy all the program(s) and workflows imported from *portal1* server according to its configuration.
mobdeploy --server portal1 -p all deploy deploy all the program(s) imported from *portal1* server according to its configuration.
mobdeploy -s local -w pretty_tree deploy deploy the *pretty_tree* workflow from the local server.
mobdeploy -s local -p blast2 deploy deploy the *blast2* program from the *local* server.
mobdeploy -s mylab -p seqret,blast2 deploy deploy the remote program *seqret* and *blast2* from the *mylab* server.
mobdeploy -v all deploy deploy every viewers and its dependencies.
mobdeploy -v jalview,jarna deploy deploy the *jalview* and *jarna* viewers and its dependencies
mobdeploy -p index only update the program index (in case the index file has been lost for instance)
2- mobjobw : to list the running job
=====================================
mobjob permit to monitor quickly the jobs which are on your batch system.
the informations displayed are:
- the key of the job
- the batch system used ( several system could be mixed in Mobyle e.g. SGE and system )
- the queue name if it defined
- the status of the job :
- running : the job is running,
- pending : the job has been queued by the batch system
- hold : the job has been hold by the administrator
- the jobName
- the email of the user
- the submission date
- the IP of the user
- the full qualified name of the user
- the command line
--------------------------------------------------------------------------------
jobkey -- batch system/queue -- status
jobName -- user email -- submission date -- requestor IP\full qualified name
unix command line
--------------------------------------------------------------------------------
e.g.::
--------------------------------------------------------------------------------
K07333355412998 -- SGE/None -- running
phyml -- email@domain.gov -- 01/13/09 18:59:51 -- 111.111.11.11\name.domain.ext
phyml -i infile_data.phylipi -d aa -b 1000 -m JTT -a e
--------------------------------------------------------------------------------
O31005884758996 -- SGE/short -- running
drawtree -- email2@domain.edu -- 01/15/09 12:47:30 -- 250.250.250.250\UNKNOWN
ln -s protpars.outtree intree && drawtree <drawtree.params && ln -s plotfile plotfile.ps
3- mobkill : to kill a job.
===========================
mobkill take one or more jobKeys and kill the corresponding jobs
e.g.::
mobkill N32454430408001 V00863529172897
4- mobclean : to clean the job or/and anonymous session tree
=============================================================
mobclean is a tool to clean your job result and anonymous session directories. By default, it cleans
your directories(job & a. session) depending on your configuration parameters (the ones of your config.py).
The tool allows you to make the cleaning separately and to run it on simulation mode (-n).
Here are the available options:
-J Clean Job Results.
-S Clean anonymous sessions.
-d <val> Delete directories older than <val> days ( positive integer value ).
-h Display this help and exit.
-v With error messages (verbose).
-n In simulation mode (dry_run), nothing is cleaned.
-l <val> Path to the Logfile where put the logs.
And some examples:
mobclean --> By default, the cleaning of jobs & sessions is done
mobclean -S -d 2 to clean the anonymous sessions older than 2 days
mobclean -J -S that is equivalent to the default run
mobclean -l /var/log/mobyle/cleaner_log : put the logs (error and warning) of the mobclean
run into this log file.
mobclean --help
The cleaning of old jobs can be automated with a cron using mobclean.
5- mobtypes : analyse the xml types
====================================
mobtypes help to ensure the coherence in xml typing between the programs xml definitions, which is
of the utmost importance for the chaining.
mobtypes take programs xml file list as argument. If mobtypes is used without option , it build
a repository with all "xml types" (see doc about typing in mobyle) found and their corresponding
python classes. -if a repository is specify ( -r option ) mobtypes display the differences between
this repository and the xml types.
usage: mobtypes < --ref path to a datatype repository > xmlpaths
options:
-h or --help ... Print this message and exit.
-r or --ref ... the path to reference
e.g ::
mobtypes MOBYLEHOME/*.xml
python class | xml class | description
AbstractText | AceAssembly |
AbstractText | AncestorsFile |
AbstractText | BambeTree |
...
mobtypes -r path_to_a_repository MOBYLEHOME/*.xml
--- ../Programs/mobyle_xml_types.txt
+++ xml list
- Binary | AbiTraceFile | binary trace file produce by abi sequencer
+ Binary | HmmBinProfile |
- Binary | AbiTraceFile | binary trace file produce by abi sequencer
means that this line is in the repository and this AbiTraceFile/Binary is not in the xml programs analysed
+ Binary | HmmBinProfile |
means that HmmBinProfile/Binary is found in programs xml analysed but is not present in repository
6- session0_9_updater.py : to update sessions from mobyle 0.9 to mobyle 0.95.1
==============================================================================
update the sessions located in Config USER_SESSIONS_PATH from a Mobyle v0.9
to be compliant with the Mobyle v0.95.1
option :
-f ( --force ) to force the update even an xml version of the session already exists.
-n ( --dry-run ) translate the session.dump in xml, but do not write them.
e.g ::
session0_9_updater 2&> update.log
7- mobvalid: validate program definition xml files
==================================================
mobvalid is a utility that automatically validates a program description, with
respect to the Mobyle format and constraints defined in the Relax-NG and
Schematron definitions. When creating or editing any program definition,
remember to validate it with this tool before installing it.
Usage: mobvalid [OPTION]... FILE...
Validate Mobyle program description files, each FILE being the path to the file.
option:
-h or --help ... Print this message and exit.
|
