1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML3.2 EN">
<HTML>
<HEAD> <link rel="canonical" href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html" />
<META NAME="GENERATOR" CONTENT="DOCTEXT">
<TITLE>PCTELESCOPE</TITLE>
</HEAD>
<BODY BGCOLOR="FFFFFF">
<div id="version" align=right><b>petsc-3.10.3 2018-12-18</b></div>
<div id="bugreport" align=right><a href="mailto:petsc-maint@mcs.anl.gov?subject=Typo or Error in Documentation &body=Please describe the typo or error in the documentation: petsc-3.10.3 v3.10.3 docs/manualpages/PC/PCTELESCOPE.html "><small>Report Typos and Errors</small></a></div>
<A NAME="PCTELESCOPE"><H1>PCTELESCOPE</H1></A>
Runs a <A HREF="../KSP/KSP.html#KSP">KSP</A> solver on a sub-group of processors. MPI processes not in the sub-communicator are idle during the solve.
<H3><FONT COLOR="#CC3333">Options Database</FONT></H3>
<TABLE border="0" cellpadding="0" cellspacing="0">
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_reduction_factor <r> </B></TD><TD>- factor to use communicator size by. e.g. with 64 MPI processes and r=4, the new sub-communicator will have 64/4 = 16 ranks.
</TD></TR>
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_ignore_dm </B></TD><TD>- flag to indicate whether an attached <A HREF="../DM/DM.html#DM">DM</A> should be ignored
</TD></TR></TABLE>
<DT><B>-pc_telescope_subcomm_type <interlaced,contiguous> </B> -how to define the reduced communicator. see <A HREF="../Sys/PetscSubcomm.html#PetscSubcomm">PetscSubcomm</A> for more.
<br>
</TABLE>
<P>
<P>
<H3><FONT COLOR="#CC3333">Notes</FONT></H3>
The preconditioner is deemed telescopic as it only calls <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() on a single
sub-communicator, in contrast with <A HREF="../PC/PCREDUNDANT.html#PCREDUNDANT">PCREDUNDANT</A> which calls <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() on N sub-communicators.
This means there will be MPI processes which will be idle during the application of this preconditioner.
<P>
The default <A HREF="../KSP/KSP.html#KSP">KSP</A> is PREONLY. If a <A HREF="../DM/DM.html#DM">DM</A> is attached to the <A HREF="../PC/PC.html#PC">PC</A>, it is re-partitioned on the sub-communicator.
Both the Bmat operator and the right hand side vector are permuted into the new DOF ordering defined by the re-partitioned <A HREF="../DM/DM.html#DM">DM</A>.
Currently only support for re-partitioning a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is provided.
Any nullspace attached to the original Bmat operator is extracted, re-partitioned and set on the repartitioned Bmat operator.
<A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>() is not propagated to the sub <A HREF="../KSP/KSP.html#KSP">KSP</A>.
Currently there is no support for the flag -pc_use_amat
<P>
Assuming that the parent preconditioner (<A HREF="../PC/PC.html#PC">PC</A>) is defined on a communicator c, this implementation
creates a child sub-communicator (c') containing fewer MPI processes than the original parent preconditioner (<A HREF="../PC/PC.html#PC">PC</A>).
<P>
<H3><FONT COLOR="#CC3333">Developer Notes</FONT></H3>
During PCSetup, the B operator is scattered onto c'.
Within <A HREF="../PC/PCApply.html#PCApply">PCApply</A>, the RHS vector (x) is scattered into a redundant vector, xred (defined on c').
Then, <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() is executed on the c' communicator.
<P>
The communicator used within the telescoping preconditioner is defined by a <A HREF="../Sys/PetscSubcomm.html#PetscSubcomm">PetscSubcomm</A> using the INTERLACED
creation routine by default (this can be changed with -pc_telescope_subcomm_type). We run the sub <A HREF="../KSP/KSP.html#KSP">KSP</A> on only the ranks within the communicator which have a color equal to zero.
<P>
The telescoping preconditioner is aware of nullspaces and near nullspaces which are attached to the B operator.
In the case where B has a (near) nullspace attached, the (near) nullspace vectors are extracted from B and mapped into
a new (near) nullspace, defined on the sub-communicator, which is attached to B' (the B operator which was scattered to c')
<P>
The telescoping preconditioner is aware of an attached <A HREF="../DM/DM.html#DM">DM</A>. In the event that the <A HREF="../DM/DM.html#DM">DM</A> is of type <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> (2D or 3D -
1D support for 1D DMDAs is not provided), a new <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is created on c' (e.g. it is re-partitioned), and this new <A HREF="../DM/DM.html#DM">DM</A>
is attached the sub <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>(). The design of telescope is such that it should be possible to extend support
for re-partitioning other to <A HREF="../DM/DM.html#DM">DM</A>'s (e.g. <A HREF="../DMPLEX/DMPLEX.html#DMPLEX">DMPLEX</A>). The user can supply a flag to ignore attached DMs.
<P>
By default, B' is defined by simply fusing rows from different MPI processes
<P>
When a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is attached to the parent preconditioner, B' is defined by: (i) performing a symmetric permutation of B
into the ordering defined by the <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> on c', (ii) extracting the local chunks via <A HREF="../Mat/MatCreateSubMatrices.html#MatCreateSubMatrices">MatCreateSubMatrices</A>(), (iii) fusing the
locally (sequential) matrices defined on the ranks common to c and c' into B' using <A HREF="../Mat/MatCreateMPIMatConcatenateSeqMat.html#MatCreateMPIMatConcatenateSeqMat">MatCreateMPIMatConcatenateSeqMat</A>()
<P>
Limitations/improvements include the following.
<A HREF="../Vec/VecPlaceArray.html#VecPlaceArray">VecPlaceArray</A>() could be used within <A HREF="../PC/PCApply.html#PCApply">PCApply</A>() to improve efficiency and reduce memory usage.
<P>
The symmetric permutation used when a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is encountered is performed via explicitly assmbleming a permutation matrix P,
and performing P^T.A.P. Possibly it might be more efficient to use <A HREF="../Mat/MatPermute.html#MatPermute">MatPermute</A>(). We opted to use P^T.A.P as it appears
<A HREF="../Vec/VecPermute.html#VecPermute">VecPermute</A>() does not supported for the use case required here. By computing P, one can permute both the operator and RHS in a
consistent manner.
<P>
Mapping of vectors is performed in the following way.
Suppose the parent comm size was 4, and we set a reduction factor of 2; this would give a comm size on c' of 2.
Using the interlaced creation routine, the ranks in c with color = 0 will be rank 0 and 2.
We perform the scatter to the sub-comm in the following way.
[1] Given a vector x defined on comm c
<P>
rank(c) : _________ 0 ______ ________ 1 _______ ________ 2 _____________ ___________ 3 __________
x : [0, 1, 2, 3, 4, 5] [6, 7, 8, 9, 10, 11] [12, 13, 14, 15, 16, 17] [18, 19, 20, 21, 22, 23]
<P>
scatter to xtmp defined also on comm c so that we have the following values
<P>
rank(c) : ___________________ 0 ________________ _1_ ______________________ 2 _______________________ __3_
xtmp : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [ ] [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] [ ]
<P>
The entries on rank 1 and 3 (ranks which do not have a color = 0 in c') have no values
<P>
<P>
[2] Copy the value from rank 0, 2 (indices with respect to comm c) into the vector xred which is defined on communicator c'.
Ranks 0 and 2 are the only ranks in the subcomm which have a color = 0.
<P>
rank(c') : ___________________ 0 _______________ ______________________ 1 _____________________
xred : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
<P>
<P>
Contributed by Dave May
<P>
<H3><FONT COLOR="#CC3333">Reference</FONT></H3>
Dave A. May, Patrick Sanan, Karl Rupp, Matthew G. Knepley, and Barry F. Smith, "Extreme-Scale Multigrid Components within PETSc". 2016. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC '16). DOI: 10.1145/2929908.2929913
<P>
<H3><FONT COLOR="#CC3333">See Also</FONT></H3>
<A HREF="../PC/PCTelescopeGetKSP.html#PCTelescopeGetKSP">PCTelescopeGetKSP</A>(), <A HREF="../PC/PCTelescopeGetDM.html#PCTelescopeGetDM">PCTelescopeGetDM</A>(), <A HREF="../PC/PCTelescopeGetReductionFactor.html#PCTelescopeGetReductionFactor">PCTelescopeGetReductionFactor</A>(), <A HREF="../PC/PCTelescopeSetReductionFactor.html#PCTelescopeSetReductionFactor">PCTelescopeSetReductionFactor</A>(), <A HREF="../PC/PCTelescopeGetIgnoreDM.html#PCTelescopeGetIgnoreDM">PCTelescopeGetIgnoreDM</A>(), <A HREF="../PC/PCTelescopeSetIgnoreDM.html#PCTelescopeSetIgnoreDM">PCTelescopeSetIgnoreDM</A>(), <A HREF="../PC/PCREDUNDANT.html#PCREDUNDANT">PCREDUNDANT</A>
<BR><P><B></B><H3><FONT COLOR="#CC3333">Level</FONT></H3>advanced<BR>
<H3><FONT COLOR="#CC3333">Location</FONT></H3>
</B><A HREF="../../../src/ksp/pc/impls/telescope/telescope.c.html#PCTELESCOPE">src/ksp/pc/impls/telescope/telescope.c</A>
<BR><A HREF="./index.html">Index of all PC routines</A>
<BR><A HREF="../../index.html">Table of Contents for all manual pages</A>
<BR><A HREF="../singleindex.html">Index of all manual pages</A>
</BODY></HTML>
|
