1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
<!--?xml version="1.0" encoding="iso-8859-1"?-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"><head><script src="ppdoc_files/menu134.js" type="text/javascript"></script><link media="all" href="ppdoc_files/widget100.css" type="text/css" rel="stylesheet">
<title>Parallel Python - Parallel Python documentation</title>
<meta name="title" content="Parallel Python documentation">
<meta name="author" content="Administrator">
<meta name="description" content="Parallel Python">
<meta name="keywords" content="parallel python">
<meta name="Generator" content="Joomla! - Copyright (C) 2005 - 2006 Open Source Matters. All rights reserved.">
<meta name="robots" content="index, follow">
<script async="" src="ppdoc_files/cbgapi.loaded_0"></script><script language="JavaScript" type="text/javascript" src="ppdoc_files/script.js"></script>
<script language="JavaScript" type="text/javascript"><!-- // --><![CDATA[
var smf_theme_url = "http://www.parallelpython.com/forum/Themes/default";
var smf_images_url = "http://www.parallelpython.com/forum/Themes/default/images";
var smf_scripturl = "http://www.parallelpython.com/component/option,com_smf/Itemid,28/";
var smf_session_id = "ab2f004fcfef43bc462ad40586336fe3";
// ]]></script>
<link rel="help" href="http://www.parallelpython.com/component/option,com_smf/Itemid,28/action,help/" target="_blank">
<link rel="search" href="http://www.parallelpython.com/component/option,com_smf/Itemid,28/action,search/">
<link rel="contents" href="http://www.parallelpython.com/component/option,com_smf/Itemid,28/">
<link rel="alternate" type="application/rss+xml" title="Parallel Python Community Forums - RSS" href="http://www.parallelpython.com/component/option,com_smf/Itemid,28/type,rss;action,.xml/">
<meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7">
<script language="JavaScript" type="text/javascript"><!-- // --><![CDATA[
var current_header = false;
function shrinkHeader(mode)
{ document.cookie = "upshrink=" + (mode ? 1 : 0); document.getElementById("upshrink").src = smf_images_url + (mode ? "/upshrink2.gif" : "/upshrink.gif");
document.getElementById("upshrinkHeader").style.display = mode ? "none" : "";
document.getElementById("upshrinkHeader2").style.display = mode ? "none" : "";
current_header = mode;
}
// ]]></script>
<script language="JavaScript" type="text/javascript"><!-- // --><![CDATA[
var current_header_ic = false;
function shrinkHeaderIC(mode)
{
document.cookie = "upshrinkIC=" + (mode ? 1 : 0);
document.getElementById("upshrink_ic").src = smf_images_url + (mode ? "/expand.gif" : "/collapse.gif");
document.getElementById("upshrinkHeaderIC").style.display = mode ? "none" : "";
current_header_ic = mode;
}
// ]]></script>
<!-- base href="http://www.parallelpython.com/" -->
<link rel="shortcut icon" href="http://www.parallelpython.com/images/favicon.ico">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="netinsert" content="0.0.1.3.4.8.1">
<link href="ppdoc_files/template_css.css" rel="stylesheet" type="text/css">
<link rel="shortcut icon" href="http://www.parallelpython.com/images/favicon.ico">
<div id="_atssh" style="visibility: hidden; height: 1px; width: 1px; position: absolute; z-index: 100000;"><iframe src="ppdoc_files/sh105.htm" style="height: 1px; width: 1px; position: absolute; z-index: 100000; border: 0px none; left: 0px; top: 0px;" title="AddThis utility frame" id="_atssh268"></iframe></div><script src="ppdoc_files/core053.js" type="text/javascript"></script></head>
<body>
<div id="outer" class="outer">
<div class="section1">
<div class="sitename">Parallel Python</div>
<div class="sitedate">
</div>
<div id="pathway_outer" class="pathway_outer">
<div id="pathway" class="pathway" style="float:left;width:50%;"><span class="pathway"><a href="http://www.parallelpython.com/" class="pathway">Home</a> <img src="ppdoc_files/arrow.png" alt="arrow"> Documents </span></div>
<div style="float:right;"><div id="___plusone_0" style="height: 15px; width: 24px; display: inline-block; text-indent: 0px; margin: 0px; padding: 0px; background: none repeat scroll 0% 0% transparent; border-style: none; float: none; line-height: normal; font-size: 1px; vertical-align: baseline;"><iframe title="+1" allowtransparency="true" src="ppdoc_files/fastbutton.htm" name="I0_1356074737799" id="I0_1356074737799" vspace="0" tabindex="0" style="position: static; top: 0px; width: 24px; margin: 0px; border-style: none; left: 0px; visibility: visible; height: 15px;" marginwidth="0" marginheight="0" hspace="0" frameborder="0" scrolling="no" width="100%"></iframe></div></div>
<div class="addthis_toolbox addthis_default_style" style="float: right; display: inline-block; text-align: right;">
<nobr>
<a href="#" title="Facebook" class="addthis_button_preferred_1 addthis_button_facebook at300b"><span class="at16nc at300bs at15nc at15t_facebook at16t_facebook"><span class="at_a11y">Share on facebook</span></span></a>
<a href="#" title="Tweet" class="addthis_button_preferred_2 addthis_button_twitter at300b"><span class="at16nc at300bs at15nc at15t_twitter at16t_twitter"><span class="at_a11y">Share on twitter</span></span></a>
<a href="#" title="Email" class="addthis_button_preferred_3 addthis_button_email at300b"><span class="at16nc at300bs at15nc at15t_email at16t_email"><span class="at_a11y">Share on email</span></span></a>
<a href="#" title="Print" class="addthis_button_preferred_4 addthis_button_print at300b"><span class="at16nc at300bs at15nc at15t_print at16t_print"><span class="at_a11y">Share on print</span></span></a>
<a title="Gmail" target="_blank" href="http://www.addthis.com/bookmark.php?v=250&winname=addthis&pub=linkedweb&source=tbx-250&lng=en&s=gmail&url=http%3A%2F%2Fwww.parallelpython.com%2Fcontent%2Fview%2F15%2F30%2F&title=Parallel%20Python%20-%20Parallel%20Python%20documentation&ate=AT-linkedweb/-/-/50d40ef19ba24535/2/50d408bbf29cdd2c&frommenu=1&uid=50d408bbf29cdd2c&ct=1&tt=0&captcha_provider=nucaptcha" class="addthis_button_preferred_5 addthis_button_gmail at300b"><span class="at16nc at300bs at15nc at15t_gmail at16t_gmail"><span class="at_a11y">Share on gmail</span></span></a>
<a title="StumbleUpon" target="_blank" href="http://www.addthis.com/bookmark.php?v=250&winname=addthis&pub=linkedweb&source=tbx-250&lng=en&s=stumbleupon&url=http%3A%2F%2Fwww.parallelpython.com%2Fcontent%2Fview%2F15%2F30%2F&title=Parallel%20Python%20-%20Parallel%20Python%20documentation&ate=AT-linkedweb/-/-/50d40ef19ba24535/3/50d408bbf29cdd2c&frommenu=1&uid=50d408bbf29cdd2c&ct=1&tt=0&captcha_provider=nucaptcha" class="addthis_button_preferred_6 addthis_button_stumbleupon at300b"><span class="at16nc at300bs at15nc at15t_stumbleupon at16t_stumbleupon"><span class="at_a11y">Share on stumbleupon</span></span></a>
<a href="#" title="Favorites" class="addthis_button_preferred_7 addthis_button_favorites at300b"><span class="at16nc at300bs at15nc at15t_favorites at16t_favorites"><span class="at_a11y">Share on favorites</span></span></a>
<a title="Blogger" target="_blank" href="http://www.addthis.com/bookmark.php?v=250&winname=addthis&pub=linkedweb&source=tbx-250&lng=en&s=blogger&url=http%3A%2F%2Fwww.parallelpython.com%2Fcontent%2Fview%2F15%2F30%2F&title=Parallel%20Python%20-%20Parallel%20Python%20documentation&ate=AT-linkedweb/-/-/50d40ef19ba24535/4/50d408bbf29cdd2c&frommenu=1&uid=50d408bbf29cdd2c&ct=1&tt=0&captcha_provider=nucaptcha" class="addthis_button_preferred_8 addthis_button_blogger at300b"><span class="at16nc at300bs at15nc at15t_blogger at16t_blogger"><span class="at_a11y">Share on blogger</span></span></a>
<a href="#" class="addthis_button_compact at300m"><span class="at16nc at300bs at15nc at15t_compact at16t_compact"><span class="at_a11y">More Sharing Services</span></span></a>
</nobr>
<div class="atclear"></div></div>
<script type="text/javascript">
var addthis_config = {
pubid: 'linkedweb',
};
</script>
<script type="text/javascript" src="ppdoc_files/addthis_widget.js" async=""></script>
<script gapi_processed="true" type="text/javascript" src="ppdoc_files/plusone.js" async=""></script>
</div>
</div>
<div class="section2">
<div id="user3" class="user3">
<table class="moduletable" cellpadding="0" cellspacing="0">
<tbody><tr>
<td>
<ul id="mainlevel-nav"><li><a href="http://www.parallelpython.com/" class="mainlevel-nav">Home</a></li><li><a href="http://www.parallelpython.com/component/option,com_smf/Itemid,29/" class="mainlevel-nav">Forums</a></li><li><a href="http://www.parallelpython.com/content/view/15/30/" class="mainlevel-nav" id="active_menu-nav">Documents</a></li><li><a href="http://www.parallelpython.com/content/view/17/31/" class="mainlevel-nav">Examples</a></li><li><a href="http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,210.msg653#msg653" class="mainlevel-nav">FAQ</a></li><li><a href="http://www.parallelpython.com/content/view/18/32/" class="mainlevel-nav">Downloads</a></li><li><a href="http://www.parallelpython.com/component/option,com_weblinks/Itemid,23/" class="mainlevel-nav">Links</a></li><li><a href="http://www.pythonblogs.com/" class="mainlevel-nav">Blogs</a></li><li><a href="http://www.parallelpython.com/component/option,com_contact/Itemid,3/" class="mainlevel-nav">Contact Us</a></li></ul> </td>
</tr>
</tbody></table>
</div>
<!--commented by Vitalii div class="logo"><img src="http://www.parallelpython.com/templates/vivacious14/images/logo.gif" alt="Free Mambo Templates TY2U.com"></div-->
<div class="topspace">
<table class="moduletable" cellpadding="0" cellspacing="0">
<tbody><tr>
<td>
<a href="http://www.structuredjournal.com/"><img src="ppdoc_files/sj_banner.png" alt="Nutrition facts and analysis" title="Structured journal" style="border:0"></a> </td>
</tr>
</tbody></table>
</div>
</div>
<div class="section3">
<!--div class="search_outer">
<div class="search_inner">
<form action="index.php" method="post">
<input class="inputbox" type="text" name="searchword" size="15" tabindex="1" value="search..." onblur="if(this.value=='') this.value='search...';" onfocus="if(this.value=='search...') this.value='';" />
<br /><input name="submit" type="image" src="http://www.parallelpython.com/templates/vivacious14/images/search.gif" class="searchbutton" tabindex="2" value="Search"/> <div class="advsearch"><a href="/index.php?option=com_search">Advanced Search</a></div>
<input type="hidden" name="option" value="search" />
</form>
</div>
</div-->
</div>
<div class="section5">
</div>
<div class="section6">
<div class="mainbody">
<table class="contentpaneopen">
<tbody><tr>
<td class="contentheading" width="100%">
Parallel Python documentation </td>
<td class="buttonheading" align="right" width="100%">
<a href="http://www.parallelpython.com/index2.php?option=com_content&do_pdf=1&id=15" target="_blank" onclick="window.open('http://www.parallelpython.com/index2.php?option=com_content&do_pdf=1&id=15','win2','status=no,toolbar=no,scrollbars=yes,titlebar=no,menubar=no,resizable=yes,width=640,height=480,directories=no,location=no'); return false;" title="PDF">
<img src="ppdoc_files/pdf_button.png" alt="PDF" name="PDF" align="middle" border="0"></a>
</td>
<td class="buttonheading" align="right" width="100%">
<a href="http://www.parallelpython.com/index2.php?option=com_content&task=view&id=15&pop=1&page=0&Itemid=30" target="_blank" onclick="window.open('http://www.parallelpython.com/index2.php?option=com_content&task=view&id=15&pop=1&page=0&Itemid=30','win2','status=no,toolbar=no,scrollbars=yes,titlebar=no,menubar=no,resizable=yes,width=640,height=480,directories=no,location=no'); return false;" title="Print">
<img src="ppdoc_files/printButton.png" alt="Print" name="Print" align="middle" border="0"></a>
</td>
<td class="buttonheading" align="right" width="100%">
<a href="http://www.parallelpython.com/index2.php?option=com_content&task=emailform&id=15&itemid=30" target="_blank" onclick="window.open('http://www.parallelpython.com/index2.php?option=com_content&task=emailform&id=15&itemid=30','win2','status=no,toolbar=no,scrollbars=yes,titlebar=no,menubar=no,resizable=yes,width=400,height=250,directories=no,location=no'); return false;" title="E-mail">
<img src="ppdoc_files/emailButton.png" alt="E-mail" name="E-mail" align="middle" border="0"></a>
</td>
</tr>
</tbody></table>
<table class="contentpaneopen">
<tbody><tr>
<td colspan="2" valign="top">
<p> <a href="http://www.parallelpython.com/content/view/15/30/#API">Module API</a> <br> <a href="http://www.parallelpython.com/content/view/15/30/#QUICKSMP">Quick start guide, SMP</a><br> <a href="http://www.parallelpython.com/content/view/15/30/#QUICKCLUSTERS">Quick start guide, clusters<br> </a><a href="http://www.parallelpython.com/content/view/15/30/#QUICKCLUSTERSAUTO">Quick start guide, clusters with auto-discovery</a><br> <a href="http://www.parallelpython.com/content/view/15/30/#ADVANCEDCLUSTERS">Advanced guide, clusters</a><br> <a href="http://www.parallelpython.com/content/view/15/30/#COMMANDLINE">Command line arguments, ppserver.py</a><br> <a href="http://www.parallelpython.com/content/view/15/30/#SECURITY">Security and secret key</a><br> <a href="http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,210.msg653#msg653" title="PP FAQ">PP FAQ</a> </p><hr> <p> </p><h1 id="API"> pp 1.6.3 module API</h1> <p> </p><table summary="section" border="0" cellpadding="2" cellspacing="0" width="100%"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br> <font color="#000000" face="helvetica, arial"><a name="Server" title="Server"></a>class <strong>Server</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2">Parallel Python SMP execution server class<br> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br> <dl><dt><a name="Server-__init__" title="Server-__init__"></a><strong>__init__</strong>(self, ncpus<font color="#909090">='autodetect'</font>, ppservers<font color="#909090">=()</font>, secret<font color="#909090">=None</font>, restart<font color="#909090">=False</font>, proto<font color="#909090">=2</font>, socket_timeout<font color="#909090">=3600</font>)</dt><dd>Creates <a href="http://www.parallelpython.com/content/view/15/30/#Server">Server</a> instance<br> </dd><dd> </dd><dd> </dd><dd> ncpus - the number of worker processes to start on the local <br>
computer, if parameter is omitted it will be set to <br> the number of processors in the system<br> ppservers - list of active parallel python execution servers <br> to connect with<br> secret - passphrase for network connections, if omitted a default<br>
passphrase will be used. It's highly recommended to use a <br> custom passphrase for all network connections.<br></dd><dd>restart - whether to restart worker process after each task completion <br> proto - protocol number for pickle module</dd><dd>socket_timeout - socket timeout in seconds which is also the maximum <br> time a remote job could be executed. Increase this value<br> if you have long running jobs or decrease if connectivity<br> to remote ppservers is often lost. <br></dd><dd> <br> With ncpus = 1 all tasks are executed consequently<br> For the best performance either use the default "autodetect" value<br> or set ncpus to the total number of processors in the system</dd></dl><dl><dt><a name="Server-destroy" title="Server-destroy"></a><strong>destroy</strong>(self)</dt><dd>Kills ppworkers and closes open files</dd></dl> <dl><dt><a name="Server-get_active_nodes" title="Server-get_active_nodes"></a><strong>get_active_nodes</strong>(self)</dt><dd>Returns active nodes as a dictionary <br> [keys - nodes, values - ncpus]</dd></dl> <dl><dt><a name="Server-get_ncpus" title="Server-get_ncpus"></a><strong>get_ncpus</strong>(self)</dt><dd>Returns the number of local worker processes (ppworkers)</dd></dl> <dl><dt><a name="Server-get_stats" title="Server-get_stats"></a><strong>get_stats</strong>(self)</dt><dd>Returns job execution statistics as a dictionary</dd></dl><dl><dt><a name="Server-print_stats" title="Server-print_stats"></a><strong>print_stats</strong>(self)</dt><dd>Prints job execution statistics. Useful for benchmarking on <br> clusters</dd></dl> <dl><dt><a name="Server-set_ncpus" title="Server-set_ncpus"></a><strong>set_ncpus</strong>(self, ncpus<font color="#909090">='autodetect'</font>)</dt><dd>Sets the number of local worker processes (ppworkers)<br> <br> ncpus - the number of worker processes, if parammeter is omitted<br>
it will be set to the number of processors in the system</dd></dl> <dl><dt><a name="Server-submit" title="Server-submit"></a><strong>submit</strong>(self, func, args<font color="#909090">=()</font>, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Submits function to the execution queue<br> <br> func - function to be executed<br> args - tuple with arguments of the 'func'<br> depfuncs - tuple with functions which might be called from 'func'<br> modules - tuple with module names to import<br> callback - callback function which will be called with argument <br> list equal to callbackargs+(result,) <br> as soon as calculation is done<br> callbackargs - additional arguments for callback function<br> group - job group, is used when <a href="http://www.parallelpython.com/content/view/15/30/#Server-wait">wait</a>(group) is called to wait for<br> jobs in a given group to finish<br> globals - dictionary from which all modules, functions and classes<br> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Server-wait" title="Server-wait"></a><strong>wait</strong>(self, group<font color="#909090">=None</font>)</dt><dd>Waits for all jobs in a given group to finish.<br> If group is omitted waits for all jobs to finish</dd></dl> <dl><dt><strong>default_port</strong> = 60000</dt></dl> <dl><dt><strong>default_secret</strong> = 'epo20pdosl;dksldkmm'</dt></dl> </td></tr></tbody></table> <p></p><p> </p><table summary="section" border="0" cellpadding="2" cellspacing="0" width="100%"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br> <font color="#000000" face="helvetica, arial"><a name="Template" title="Template"></a>class <strong>Template</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2"><a href="http://www.parallelpython.com/content/view/15/30/#Template">Template</a> class<br> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br> <dl><dt><a name="Template-__init__" title="Template-__init__"></a><strong>__init__</strong>(self, job_server, func, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Creates <a href="http://www.parallelpython.com/content/view/15/30/#Template">Template</a> instance<br> <br> jobs_server - pp server for submitting jobs<br> func - function to be executed<br> depfuncs - tuple with functions which might be called from 'func'<br> modules - tuple with module names to import<br> callback - callback function which will be called with argument <br> list equal to callbackargs+(result,) <br> as soon as calculation is done<br> callbackargs - additional arguments for callback function<br> group - job group, is used when wait(group) is called to wait for<br> jobs in a given group to finish<br> globals - dictionary from which all modules, functions and classes<br> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Template-submit" title="Template-submit"></a><strong>submit</strong>(self, *args)</dt><dd>Submits function with *arg arguments to the execution queue</dd></dl> </td></tr></tbody></table><p></p><p> </p><table summary="section" border="0" cellpadding="2" cellspacing="0" width="100%"> <tbody><tr bgcolor="#55aa55"> <td colspan="3" valign="bottom"> <br> <font color="#ffffff" face="helvetica, arial"><strong>Data</strong></font></td></tr> <tr><td bgcolor="#55aa55"> </td><td> </td> <td width="100%"><strong>copyright</strong> = 'Copyright (c) 2005-2012 Vitalii Vanovschi. All rights reserved'<br> <strong>version</strong> = '1.6.3'</td></tr></tbody></table> <p></p><hr><h1 id="QUICKSMP"> Quick start guide, SMP<br></h1> <p>1) Import pp module:</p><p><strong> import pp</strong></p><p>2)
Start pp execution server with the number of workers set
to the number of processors in the system
</p><p><strong> job_server = pp.Server() </strong></p><p>3) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br> </p><p> ...etc...<br></p><p>4) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p> <hr><h1 id="QUICKCLUSTERS"> Quick start guide, clusters </h1><p><em><strong>On the nodes</strong></em> <br></p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver.py </strong></p><p><strong> node-2> ./ppserver.py</strong></p><p><strong> node-3> ./ppserver.py</strong></p><p><em><strong>On the client</strong></em> <br></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver.py) </p><p><strong> ppservers=("node-1", "node-2", "node-3")</strong><br></p><p>4)
Start pp execution server with the number of workers set
to the number of processors in the system
and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br> </p><p> ...etc...<br></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a></p> <hr><h1 id="QUICKCLUSTERSAUTO"> Quick start guide, clusters with autodiscovery<br> </h1><p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver.py -a<br> </strong></p><p><strong> node-2> ./ppserver.py -a</strong></p><p><strong> node-3> ./ppserver.py -a<br></strong></p><p><em><strong>On the client</strong></em></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Set ppservers list to auto-discovery: </p><p><strong> ppservers=("*",)</strong><br></p><p>4)
Start pp execution server with the number of workers set
to the number of processors in the system
and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br> </p><p> ...etc...<br></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr><h1 id="ADVANCEDCLUSTERS"> Advanced guide, clusters </h1> <p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes (listen to a given port 35000,<br> and local network interface only, accept only connections which know correct secret):</p><p><strong> node-1> ./ppserver.py -p 35000 -i 192.168.0.101 -s "mysecret"<br></strong></p><p><strong> node-2> ./ppserver.py -p 35000 -i 192.168.0.102</strong><strong> -s "mysecret"</strong></p><p><strong> node-3> ./ppserver.py -p 35000 -i 192.168.0.103</strong><strong> -s "mysecret"</strong></p><p><em><strong>On the client</strong></em> <br></p> <p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver.py) </p><p><strong> ppservers=("node-1:35000", "node-2:</strong><strong>35000</strong><strong>", "node-3:</strong><strong>35000</strong><strong>")</strong><br></p><p>4)
Start pp execution server with the number of workers set
to the number of processors in the system,
<br>list of ppservers to connect with and secret key to authorize the connection:</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>, secret="</strong><strong>mysecret</strong><strong>") </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br> </p><p> ...etc...<br></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> 7) Print the execution statistics:<br></p><p><strong> job_server.print_stats()</strong></p><p>To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr><h1 id="COMMANDLINE"> Command line options, ppserver.py </h1> <pre>Usage: ppserver.py [-hda] [-i interface] [-b broadcast] [-p port] [-w nworkers] [-s secret] [-t seconds]<br> Options:<br> -h : this help message<br> -d : debug<br> -a : enable auto-discovery service<br> -i interface : interface to listen<br> -b broadcast : broadcast address for auto-discovery service<br> -p port : port to listen<br> -w nworkers : number of workers to start<br> -s secret : secret for authentication<br> -t seconds : timeout to exit if no connections with clients exist<br> -k seconds : socket timeout in seconds <br> -P pid_file : file to write PID to <br></pre><hr><h1 id="COMMANDLINE"> Security and secret key
<a name="SECURITY" title="SECURITY"></a></h1><p> Due to the security concerns, in Debian we decided to disable default password authentication, and require to run <strong>ppserver</strong> with a non-trivial secret key (<strong>-s</strong> command line argument) which should be paired with the matching <em>secret</em> keyword of PP Server class constructor. An alternative way to set a secret key is by assigning <strong>pp_secret</strong> variable in the configuration file <strong>.pythonrc.py</strong> which should be located in the user home directory (please make this file readable and writable only by user). The secret key set in <strong>.pythonrc.py</strong> could be overridden by command line argument (for <strong>ppserver</strong>) and <em>secret</em> keyword (for PP Server class constructor). Note that passing the password on the command line allows every user to see it (e.g. using <strong>ps(1)</strong>), and that running it on an open/untrusted network can be a security problem as traf fic, including the password, is not encrypted.</p>
<p> </p><hr><h1 id="COMMANDLINE"> ppserver.py stats and PID file example<br></h1><p>To print job execution statistics for ppserver.py send a SIGUSR1 signal to its main process.<br>For instance on UNIX platform following commands will start a server and print its stats:<br><strong>ppserver.py -P /tmp/ppserver.pid </strong></p><p><strong>kill -s SIGUSR1 `cat /tmp/ppserver.pid`</strong><br></p>
</td>
</tr>
</tbody></table>
<span class="article_seperator"> </span>
</div>
</div>
<div class="section7">
<div class="bottom">
<table class="moduletable" cellpadding="0" cellspacing="0">
<tbody><tr>
<td>
<a href="http://www.nutritionvalue.org/"><img src="ppdoc_files/nv_banner.png" alt="Nutrition facts and analysis" title="Nutrition facts and analysis" style="border:0"></a>
<br> </td>
</tr>
</tbody></table>
</div>
<div class="footer">
<div align="center">
<font color="gray">Parallel Python Solutions | Parallel Python Forums | Parallel Python Community</font>
</div>
<br>
<div align="center">
<font color="gray">Parallel Python © 2005 - 2012 Vitalii Vanovschi. All rights reserved</font>
</div>
<script src="ppdoc_files/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-404570-4";
urchinTracker();
</script>
</div>
</div>
</div>
</body></html>
<!-- 1356074737 -->
|
