1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
|
.. _hoc_parnet:
ParallelNetManager
------------------
.. hoc:class:: ParallelNetManager
Syntax:
``pnm = new ParallelNetManager(ncell)``
Description:
Manages the setup and running of a network simulation on a cluster
of workstations or any parallel computer that has mpi installed.
A version for PVM is also available. This class, implemented
in nrn/share/lib/hoc/netparmpi.hoc, presents an interface
compatible with the NetGUI style of network specification, and is implemented
using the :ref:`hoc_ParallelNetwork` methods. Those methods are
available only if NEURON has been built with the configuration option,
--with-mpi. The :file:`netparmpi.hoc` file at last count was only 285 lines long
so if you have questions about how it works that are not answered here,
take a look.
Experience with published network models where there is an order of magnitude
more segments or cells than machines, suggests that superlinear speedup
occurs up to about 20 to 50 machines due to the fact that the parallel
machine has much more effective high speed cache memory than a single
cpu. Basically, good performance will occur if there is a lot for each
machine to do and the amount of effort to simulate each machine's subnet
is about equal. If cell granularity causes load balance to be
a signficant problem see :hoc:meth:`ParallelNetManager.splitcell`.
The "lot for each machine to do" is relative to the
number of spikes that must be exchanged between machines and how often
these exchanges take place. The latter is determined by the minimum
delay between a spike event generated on one machine that must be delivered
to another machine since that defines the interval that each machine
is allowed to integrate before having to share all spikes it generated
which are destined for other machines.
The fundamental requirement for the use of this class is for the
programmer to be able to associate a unique global id (gid) for each
cell and define the connectivity by means of the source_cell_gid and the
target_cell_gid. If the target cell happens to have synapses, we assume
they can be found via a local synapse index into the target cell's synapse list.
We absolutely must use global indices because it
will be the case that when a connection is requested
on a machine that either the source or the target cell or both may not
actually exist on the machine -- the last case is a no-op.
The following describes the author's intention as to how this class can
be used to construct and simulate a parallel network.
It is assumed that every machine
executes exactly the same code (though with different data).
0) So that the concatenation of all the following fragments will
end up being a valid network simulation for a ring of 128 artificial
cells where cell i sends a spike to cell i+1, let's start out with
.. code-block::
none
load_file("nrngui.hoc") // not that I want a gui but I do want the stdrun system
tstop = 1000
Yes, I know that this example is foolish since there is no computation
going on except when a cell receives a spike. I don't expect any benefit
from parallelization but it is simple enough to allow me to focus on the process
of setup and run instead of cluttering the example with a large cell class.
1) load the :file:`netparmpi.hoc` file and create a ParallelNetManager
.. code-block::
none
load_file("netparmpi.hoc")
objref pnm
ncell = 128
pnm = new ParallelNetManager(ncell)
If you know the global number of cells put it in. For the non-MPI
implementation of ParallelNetManager, ncell is absolutely necessary
since that implementation constructs many mapping vectors that allow
it to figure out what cell is being talked about when the gid is
known. The MPI implementation uses dynamically constructed maps and
it is not necessary to know the global number of cells at this time.
Note that ncell refers to the global number of cells and NOT the
number of cells to be created on this machine.
2) Tell the system which gid's are on which machines.
The simplest distribution mechanism is :hoc:func:`round_robin`
.. code-block::
none
pnm.round_robin()
which will certainly give good load balance if the number of each
cell type to be constructed is an integer multiple of the number
of machines. Otherwise specify which gid's are on which machines through
the use of :hoc:meth:`ParallelNetManager.set_gid2node` . Note that you only
HAVE to call \ ``pnm.set_gid2node(gid, myid)`` for the subset of gid's that
are supposed to be associated with this machines
particular \ ``myid = pnm.pc.id`` but it is usually simpler just to call
it for all gid's since the set_gid2node call is a no-op when the second
argument does not match the pc.id. Also, the PVM version REQUIRES that
you call the function for all the gid values.
There are three performance considerations with regard to sprinkling gid
values on machines.
A) By far the most important is load balance. That is
simple if all your cells take the same time to integrate over the same
interval. If cells have very different sizes or cpu's end up with
very different amounts of work to do so that load balance is a
serious problem then :hoc:meth:`ParallelNetManager.splitcell` can be used to
solve it.
B) Of lesser importance but still quite important is to maximize the
delay of NetCon's that span machines. This isn't an issue if all your
NetCon delays are the same. The minimum delay across machines defines
the maximum step size that each machine can integrate before having
to share spikes. In principle, Metis can help with this and C) but don't
waste your time unless you have established that communication overhead
is your rate limiting step. See :hoc:meth:`ParallelNetManager.prstat` and
:hoc:meth:`ParallelContext.wait_time` .
C) I am only guessing that this is less important than B, it is certainly
related, but obviously
things will be better if you minimize the number of spanning NetCon's.
For our ring example it obviously would be best to keep neighboring cells together
but the improvement may be too small to measure.
3) Now create only the cells that are supposed to be on this machine
using :hoc:meth:`ParallelNetManager.register_cell`.
.. code-block::
none
for i=0, ncell-1 if (pnm.gid_exists(i)) {
pnm.register_cell(i, new IntFire1())
}
Notice how we don't construct a cell if the gid does not exist.
You only HAVE to call
register_cell for those gid's which are actually owned by this machine and
need to send spikes to other machines.
If the gid does not exist, then register_cell will call gid_exists for you.
Note that 2) and 3) can
be combined but it is a serious bug if a gid exists on more than one machine.
You can even start connecting
as discussed in item 4) but of course a NetCon presupposes the existence
of whatever cells it needs on this machine.
Of course this presupposes that you have
already read the files that define your cell classes.
We assume your
cell classes for "real" cells follow the NetworkReadyCell policy required by
the NetGUI tool. That is, each "real" cell type has a synapse list, eg. the
first synapse is \ ``cell.synlist.object(0)`` (the programmer will have to
make use of those synapse indices when such cells are the target of a NetCon)
and each "real" cell type has a connect2target method that constructs
a netcon (returns it in the second argument)
with that cell as the source and its first argument as the
synapse or artificial cell object.
Artificial cells can either be unwrapped or follow the NetGUI tool policy
where they are wrapped in a cell class in which the actual artificial cell
is given by the \ ``cell.pp`` field and the cell class also has a
connect2target method.
If you don't know what I've been talking about in the last two paragraphs,
use the NetGUI tool on a single machine to construct a toy network consisting
of a few real and artificial cells and save it to a hoc file for examination.
4) Connect the cells using :hoc:meth:`ParallelNetManager.nc_append`
.. code-block::
none
for i=0, ncell-1 {
pnm.nc_append(i, (i+1)%ncell, -1, 1.1, 2)
}
Again, it only has to be called if i, or i+1, or both, are on this machine.
It is a no-op if neither are on this machine and usually a no-op if only
the source is on this machine since it will only mark the source cell
as output cell, once.
The -1 just refers to the
synapse index which should be -1 for artificial cells.
The delay is 2 ms and the weight is 1.1 which guarantees
that the IntFire1 cell will fire when it receives a spike.
Our example requires a stimulus and this is not an
unreasonable time to stimulate the net.
Let's get the ring going by forcing the gid==4
cell to fire.
.. code-block::
none
// stimulate
objref stim, ncstim
if (pnm.gid_exists(4)) {
stim = new NetStim(.5)
ncstim = new NetCon(stim, pnm.pc.gid2obj(4))
ncstim.weight = 1.1
ncstim.delay = 0
stim.number=1
stim.start=1
}
Note the stimulator does not require a gid even though it is an artificial
cell because its connections do not span machines. But it does have to be
on the machine that has the cell it is connecting to.
5) Have the system figure out the minimum spanning NetCon delay so it knows
the maximum step size.
.. code-block::
none
pnm.set_maxstep(100) // will end up being 2
6) Decide what output to collect
.. code-block::
none
pnm.want_all_spikes()
If you want to record spikes from only a few cells you can use
:hoc:meth:`ParallelNetManager.spike_record` explicitly. If you want to
record range variable trajectories, check that the cell exists with
:hoc:meth:`ParallelNetManager.gid_exists` and then use :hoc:meth:`Vector.record`.
7) Initialize and run.
.. code-block::
none
stdinit()
runtime = startsw()
pnm.psolve(tstop)
runtime = startsw() - runtime
8) Print the results.
.. code-block::
none
for i=0, pnm.spikevec.size-1 {
print pnm.spikevec.x[i], pnm.idvec.x[i]
}
If you save the stdout to a file you can sort the results. A nice idiom
is
\ ``sort -k 1n,1n -k 2n,2n temp1 > temp``
A perhaps more flexible alternative is to separate the master from all the
workers somewhere after item 4) and before item 8) using :hoc:meth:`ParallelContext.runworker`
and then making use of the :hoc:meth:`ParallelNetManager.prun` and
:hoc:meth:`ParallelNetManager.gatherspikes` with the normal ParallelContext control
in a master worker framework.
At any rate, before we quit we have to call it so that the master can
tell all the workers to quit.
.. code-block::
none
pnm.pc.runworker
pnm.pc.done
----
.. hoc:method:: ParallelNetManager.set_gid2node
Syntax:
``pnm.set_gid2node(gid, machine_id)``
Description:
When MPI is being used, this is just
a wrapper for the ParallelContext version of
:hoc:meth:`ParallelContext.set_gid2node` .
----
.. hoc:method:: ParallelNetManager.round_robin
Syntax:
``pnm.round_robin()``
Description:
The gid ranging from 0 to ncell-1
is assigned to machine ``(gid + 1) % nhost``. There is no good reason
anymore for the "+1". :hoc:meth:`ParallelContext.nhost` is the number of machines
available.
----
.. hoc:method:: ParallelNetManager.gid_exists
Syntax:
``result = pnm.gid_exists(gid)``
Description:
Returns 1 if the gid exists on this machine, 2 if it exists and has been
declared to be an output cell. 0 otherwise.
Just a wrapper for :hoc:meth:`ParallelContext.gid_exists` when MPI is being used.
----
.. hoc:method:: ParallelNetManager.create_cell
Syntax:
``cellobject = pnm.create_cell(gid, "obexpr")``
Description:
This is deprecated. Use :hoc:meth:`ParallelNetManager.register_cell` .
If the gid exists on this machine the obexpr is executed in a statement
equivalent to ``pnm.cells.append(obexpr)``. Obexpr should be something like
\ ``"new Pyramid()"`` or any function that returns a cell object. Valid
"real" cell objects should have a connect2target method and a synlist
synapse list field just as the types used by the NetGUI builder.
Artificial cell objects can be bare or enclosed in a wrapper class
using the pp field.
Note: the following has been changed so that the source is always
an outputcell.
At the end of this call, \ ``pnm.gid_exists(gid)`` will return either
0 or 1 because the cell has not yet been declared to be an outputcell.
That will be done when the first connection is requested for which
this cell is a source but the target is on another machine.
----
.. hoc:method:: ParallelNetManager.register_cell
Syntax:
``pnm.register_cell(gid, cellobject)``
Description:
Associate gid and cellobject. If :hoc:meth:`ParallelContext.gid_exists`
is zero then this procedure calls :hoc:meth:`ParallelContext.set_gid2node`
If the cell is "real" or encapsulates a point process artificial cell, then
the cellobject.connect2target is called. The cellobject is declared to
be an :hoc:meth:`ParallelContext.outputcell` .
This method supersedes the create_cell method since it more easily handles
cell creation arguments.
----
.. hoc:method:: ParallelNetManager.nc_append
Syntax:
``netcon = pnm.nc_append(src_gid, target_gid, synapse_id, weight, delay)``
Description:
If the source and target exist on this machine a NetCon is created
and added to the pnm.nclist.
If the target exists and is a real cell
the synapse object is \ ``pnm.gid2obj(target_gid).synlist(synapse_id)``.
If the target exists and is a wrapped artificial cell then the
synapse_id should be -1 and the target artificial cell is
\ ``pnm.gid2obj(target_gid).pp``.
If the target exists and is an ArtificialCell
the synapse_id should be -1 and the target artificial cell is
\ ``pnm.gid2obj(target_gid)``. Note that
the target is an unwrapped artificial cell if
:hoc:meth:`StringFunctions.is_point_process` returns a non-zero value.
If the target exists but not the source, the netcon
is created via :hoc:meth:`ParallelContext.gid_connect` and added to the
pnm.nclist.
If the source exists but not the target, and
:hoc:meth:`ParallelContext.gid_exists` returns
1 (instead of 2) then the cell is marked to be an
:hoc:meth:`ParallelContext.outputcell` .
If the source exists and is a real cell or wrapped artificial
cell \ ``pnm.gid2obj(src_id).connect2target(synapse_target_object, nc)``
is used to
create the NetCon.
If the source exists and is a artificial cell
then the NetCon is created directly.
If neither the source or target exists,
there is nothing to do.
----
.. hoc:method:: ParallelNetManager.want_all_spikes
Syntax:
``pnm.want_all_spikes()``
Description:
Records all spikes of all cells on this machine into the
pnm.spikevec and pnm.idvec Vector objects. The spikevec holds spike times
and the idvec holds the corresponding gid values.
----
.. hoc:method:: ParallelNetManager.spike_record
Syntax:
``pnm.spike_record(gid)``
Description:
Wraps :hoc:meth:`ParallelContext.spike_record` but calls it only if
:hoc:meth:`ParallelContext.gid_exists` is nonzero and records the spikes
into the pnm.spikevec and pnm.gidvec Vector objects.
----
.. hoc:method:: ParallelNetManager.prun
Syntax:
``pnm.prun()``
Description:
All the workers and the master are asked to :hoc:meth:`ParallelNetManager.pinit`
and :hoc:meth:`ParallelNetManager.pcontinue` up to tstop.
----
.. hoc:method:: ParallelNetManager.psolve
Syntax:
``pnm.psolve(tstop)``
Description:
Wraps :hoc:meth:`ParallelContext.psolve` .
----
.. hoc:method:: ParallelNetManager.pinit
Syntax:
``pnm.pinit()``
Description:
All the workers and the master execute a call to
:hoc:meth:`ParallelContext.set_maxstep` to determine the maximum possible step size
and all the workers and the master execute a call to
the stdinit() of the
standard run system.
----
.. hoc:method:: ParallelNetManager.pcontinue
Syntax:
``pnm.pcontinue(tstop)``
Description:
All the workers and the master execute a call to :hoc:meth:`ParallelContext.psolve`
to integrate from the current value of t to the argument value.
----
.. hoc:method:: ParallelNetManager.prstat
Syntax:
``pnm.prstat(0)``
``pnm.prstat(1)``
Description:
Prints a high resolution amount of time all the machines have waited for
spike exchange. If some are much higher than others then there is likely
a load balance problem. If they are all high relative to the simulation
time then spike exchange may be the rate limiting step.
If the argument is 1, then, in addition to wait time, spike_statistics
are printed. The format is
.. code-block::
none
pc.id wait_time(s) nsendmax nsend nrecv nrecv_useful
%d\t %g\t %d\t %d\t %d\t %d\n
.. seealso::
:hoc:meth:`ParallelContext.wait_time`, :hoc:meth:`ParallelContext.spike_statistics`
----
.. hoc:method:: ParallelNetManager.gatherspikes
Syntax:
``pnm.gatherspikes``
Description:
All the workers are asked to post their spikevec and idvec Vectors
for taking by the master and concatenated to the master's spikevec
and idvec Vectors.
----
.. hoc:method:: ParallelNetManager.splitcell
Syntax:
``cas pnm.splitcell(hostcas, hostparent)``
Description:
The cell is split at the currently accessed section and that
section's
parent into two subtrees rooted at the old connection end of the cas
and the old cas connecting point of the parent (latter must be
0 or 1). The cas subtree will be preserved on the host specified
by hostcas and the parent subtree will be destroyed. The parent
subtree
will be preserved on the host specified by hostparent and the cas
subtree destroyed. Hostparent must be either ``hostcas+1`` or ``hostcas-1``.
Splitcell works only if NEURON has been configured with the
--with-paranrn option. A split cell has exactly the same stability
and accuracy properties as if it were on a single machine. Splitcell
cannot be used with variable step methods at this time. A cell can
be split into only two pieces.
Splitcell is implemented using the :hoc:meth:`ParallelContext.splitcell` method
of :hoc:class:`ParallelContext`.
|
