1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML3.2 EN">
<HTML>
<HEAD> <link rel="canonical" href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCTELESCOPE.html" />
<META NAME="GENERATOR" CONTENT="DOCTEXT">
<TITLE>PCTELESCOPE</TITLE>
</HEAD>
<BODY BGCOLOR="FFFFFF">
<div id="version" align=right><b>petsc-3.14.5 2021-03-03</b></div>
<div id="bugreport" align=right><a href="mailto:petsc-maint@mcs.anl.gov?subject=Typo or Error in Documentation &body=Please describe the typo or error in the documentation: petsc-3.14.5 v3.14.5 docs/manualpages/PC/PCTELESCOPE.html "><small>Report Typos and Errors</small></a></div>
<A NAME="PCTELESCOPE"><H1>PCTELESCOPE</H1></A>
Runs a <A HREF="../KSP/KSP.html#KSP">KSP</A> solver on a sub-communicator. MPI ranks not in the sub-communicator are idle during the solve.
<H3><FONT COLOR="#CC3333">Options Database</FONT></H3>
<TABLE border="0" cellpadding="0" cellspacing="0">
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_reduction_factor <r> </B></TD><TD>- factor to reduce the communicator size by. e.g. with 64 MPI ranks and r=4, the new sub-communicator will have 64/4 = 16 ranks.
</TD></TR>
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_ignore_dm </B></TD><TD>- flag to indicate whether an attached <A HREF="../DM/DM.html#DM">DM</A> should be ignored.
</TD></TR>
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_subcomm_type <interlaced,contiguous> </B></TD><TD>- defines the selection of MPI ranks on the sub-communicator. see <A HREF="../Sys/PetscSubcomm.html#PetscSubcomm">PetscSubcomm</A> for more information.
</TD></TR>
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_ignore_kspcomputeoperators </B></TD><TD>- flag to indicate whether <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A> should be used on the sub-<A HREF="../KSP/KSP.html#KSP">KSP</A>.
</TD></TR>
<TR><TD WIDTH=40></TD><TD ALIGN=LEFT VALIGN=TOP><B>-pc_telescope_use_coarse_dm </B></TD><TD>- flag to indicate whether the coarse <A HREF="../DM/DM.html#DM">DM</A> should be used to define the sub-communicator.
</TD></TR></TABLE>
<P>
<P>
<H3><FONT COLOR="#CC3333">Notes</FONT></H3>
Assuming that the parent preconditioner (<A HREF="../PC/PC.html#PC">PC</A>) is defined on a communicator c, this implementation
creates a child sub-communicator (c') containing fewer MPI ranks than the original parent preconditioner (<A HREF="../PC/PC.html#PC">PC</A>).
The preconditioner is deemed telescopic as it only calls <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() on a single
sub-communicator, in contrast with <A HREF="../PC/PCREDUNDANT.html#PCREDUNDANT">PCREDUNDANT</A> which calls <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() on N sub-communicators.
This means there will be MPI ranks which will be idle during the application of this preconditioner.
Additionally, in comparison with <A HREF="../PC/PCREDUNDANT.html#PCREDUNDANT">PCREDUNDANT</A>, <A HREF="../PC/PCTELESCOPE.html#PCTELESCOPE">PCTELESCOPE</A> can utilize an attached <A HREF="../DM/DM.html#DM">DM</A>.
<P>
The default type of the sub <A HREF="../KSP/KSP.html#KSP">KSP</A> (the <A HREF="../KSP/KSP.html#KSP">KSP</A> defined on c') is PREONLY.
<P>
There are three setup mechanisms for PCTelescope. Features support by each type are described below.
In the following, we will refer to the operators B and B', these are the Bmat provided to the <A HREF="../KSP/KSP.html#KSP">KSP</A> on the
communicators c and c' respectively.
<P>
[1] Default setup
The sub-communicator c' is created via <A HREF="../Sys/PetscSubcommCreate.html#PetscSubcommCreate">PetscSubcommCreate</A>().
Explicitly defined nullspace and near nullspace vectors will be propogated from B to B'.
Currently there is no support define nullspaces via a user supplied method (e.g. as passed to <A HREF="../Mat/MatNullSpaceSetFunction.html#MatNullSpaceSetFunction">MatNullSpaceSetFunction</A>()).
No support is provided for <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>().
Currently there is no support for the flag -pc_use_amat.
<P>
[2] <A HREF="../DM/DM.html#DM">DM</A> aware setup
If a <A HREF="../DM/DM.html#DM">DM</A> is attached to the <A HREF="../PC/PC.html#PC">PC</A>, it is re-partitioned on the sub-communicator c'.
c' is created via <A HREF="../Sys/PetscSubcommCreate.html#PetscSubcommCreate">PetscSubcommCreate</A>().
Both the Bmat operator and the right hand side vector are permuted into the new DOF ordering defined by the re-partitioned <A HREF="../DM/DM.html#DM">DM</A>.
Currently only support for re-partitioning a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is provided.
Any explicitly defined nullspace or near nullspace vectors attached to the original Bmat operator (B) are extracted, re-partitioned and set on the re-partitioned Bmat operator (B').
Currently there is no support define nullspaces via a user supplied method (e.g. as passed to <A HREF="../Mat/MatNullSpaceSetFunction.html#MatNullSpaceSetFunction">MatNullSpaceSetFunction</A>()).
Support is provided for <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>(). The user provided function and context is propagated to the sub <A HREF="../KSP/KSP.html#KSP">KSP</A>.
This is fragile since the user must ensure that their user context is valid for use on c'.
Currently there is no support for the flag -pc_use_amat.
<P>
[3] Coarse <A HREF="../DM/DM.html#DM">DM</A> setup
If a <A HREF="../DM/DM.html#DM">DM</A> (dmfine) is attached to the <A HREF="../PC/PC.html#PC">PC</A>, dmfine is queried for a "coarse" <A HREF="../DM/DM.html#DM">DM</A> (call this dmcoarse) via <A HREF="../DM/DMGetCoarseDM.html#DMGetCoarseDM">DMGetCoarseDM</A>().
<A HREF="../PC/PCTELESCOPE.html#PCTELESCOPE">PCTELESCOPE</A> will interpret the coarse <A HREF="../DM/DM.html#DM">DM</A> as being defined on a sub-communicator of c.
The communicator associated with dmcoarse will define the c' to be used within <A HREF="../PC/PCTELESCOPE.html#PCTELESCOPE">PCTELESCOPE</A>.
<A HREF="../PC/PCTELESCOPE.html#PCTELESCOPE">PCTELESCOPE</A> will check that c' is in fact a sub-communicator of c. If it is not, an error will be reported.
The intention of this setup type is that <A HREF="../PC/PCTELESCOPE.html#PCTELESCOPE">PCTELESCOPE</A> will use an existing (e.g. user defined) communicator hierarchy, say as would be
available with using multi-grid on unstructured meshes.
This setup will not use the command line options -pc_telescope_reduction_factor or -pc_telescope_subcomm_type.
Any explicitly defined nullspace or near nullspace vectors attached to the original Bmat operator (B) are extracted, scattered into the correct ordering consistent with dmcoarse and set on B'.
Currently there is no support define nullspaces via a user supplied method (e.g. as passed to <A HREF="../Mat/MatNullSpaceSetFunction.html#MatNullSpaceSetFunction">MatNullSpaceSetFunction</A>()).
There is no general method to permute field orderings, hence only <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>() is supported.
The user must use <A HREF="../Sys/PetscObjectComposeFunction.html#PetscObjectComposeFunction">PetscObjectComposeFunction</A>() with dmfine to define the method to scatter fields from dmfine to dmcoarse.
Propogation of the user context for <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>() on the sub <A HREF="../KSP/KSP.html#KSP">KSP</A> is attempted by querying the <A HREF="../DM/DM.html#DM">DM</A> contexts associated with dmfine and dmcoarse. Alternatively, the user may use <A HREF="../Sys/PetscObjectComposeFunction.html#PetscObjectComposeFunction">PetscObjectComposeFunction</A>() with dmcoarse to define a method which will return the appropriate user context for <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>().
Currently there is no support for the flag -pc_use_amat.
This setup can be invoked by the option -pc_telescope_use_coarse_dm or by calling <A HREF="../PC/PCTelescopeSetUseCoarseDM.html#PCTelescopeSetUseCoarseDM">PCTelescopeSetUseCoarseDM</A>(pc,<A HREF="../Sys/PETSC_TRUE.html#PETSC_TRUE">PETSC_TRUE</A>);
Further information about the user-provided methods required by this setup type are described here <A HREF="../PC/PCTelescopeSetUseCoarseDM.html#PCTelescopeSetUseCoarseDM">PCTelescopeSetUseCoarseDM</A>().
<P>
<H3><FONT COLOR="#CC3333">Developer Notes</FONT></H3>
During PCSetup, the B operator is scattered onto c'.
Within <A HREF="../PC/PCApply.html#PCApply">PCApply</A>, the RHS vector (x) is scattered into a redundant vector, xred (defined on c').
Then, <A HREF="../KSP/KSPSolve.html#KSPSolve">KSPSolve</A>() is executed on the c' communicator.
<P>
The communicator used within the telescoping preconditioner is defined by a <A HREF="../Sys/PetscSubcomm.html#PetscSubcomm">PetscSubcomm</A> using the INTERLACED
creation routine by default (this can be changed with -pc_telescope_subcomm_type). We run the sub <A HREF="../KSP/KSP.html#KSP">KSP</A> on only the ranks within the communicator which have a color equal to zero.
<P>
The telescoping preconditioner is aware of nullspaces and near nullspaces which are attached to the B operator.
In the case where B has a (near) nullspace attached, the (near) nullspace vectors are extracted from B and mapped into
a new (near) nullspace, defined on the sub-communicator, which is attached to B' (the B operator which was scattered to c')
<P>
The telescoping preconditioner can re-partition an attached <A HREF="../DM/DM.html#DM">DM</A> if it is a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> (2D or 3D -
support for 1D DMDAs is not provided). If a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is found, a topologically equivalent <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is created on c'
and this new <A HREF="../DM/DM.html#DM">DM</A> is attached the sub <A HREF="../KSP/KSP.html#KSP">KSP</A>. The design of telescope is such that it should be possible to extend support
for re-partitioning other to <A HREF="../DM/DM.html#DM">DM</A>'s (e.g. <A HREF="../DMPLEX/DMPLEX.html#DMPLEX">DMPLEX</A>). The user can supply a flag to ignore attached DMs.
Alternatively, user-provided re-partitioned DMs can be used via -pc_telescope_use_coarse_dm.
<P>
With the default setup mode, B' is defined by fusing rows (in order) associated with MPI ranks common to c and c'.
<P>
When a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is attached to the parent preconditioner, B' is defined by: (i) performing a symmetric permutation of B
into the ordering defined by the <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> on c', (ii) extracting the local chunks via <A HREF="../Mat/MatCreateSubMatrices.html#MatCreateSubMatrices">MatCreateSubMatrices</A>(), (iii) fusing the
locally (sequential) matrices defined on the ranks common to c and c' into B' using <A HREF="../Mat/MatCreateMPIMatConcatenateSeqMat.html#MatCreateMPIMatConcatenateSeqMat">MatCreateMPIMatConcatenateSeqMat</A>()
<P>
Limitations/improvements include the following.
<A HREF="../Vec/VecPlaceArray.html#VecPlaceArray">VecPlaceArray</A>() could be used within <A HREF="../PC/PCApply.html#PCApply">PCApply</A>() to improve efficiency and reduce memory usage.
A unified mechanism to query for user contexts as required by <A HREF="../KSP/KSPSetComputeOperators.html#KSPSetComputeOperators">KSPSetComputeOperators</A>() and <A HREF="../Mat/MatNullSpaceSetFunction.html#MatNullSpaceSetFunction">MatNullSpaceSetFunction</A>().
<P>
The symmetric permutation used when a <A HREF="../DMDA/DMDA.html#DMDA">DMDA</A> is encountered is performed via explicitly assmbleming a permutation matrix P,
and performing P^T.A.P. Possibly it might be more efficient to use <A HREF="../Mat/MatPermute.html#MatPermute">MatPermute</A>(). We opted to use P^T.A.P as it appears
<A HREF="../Vec/VecPermute.html#VecPermute">VecPermute</A>() does not supported for the use case required here. By computing P, one can permute both the operator and RHS in a
consistent manner.
<P>
Mapping of vectors (default setup mode) is performed in the following way.
Suppose the parent communicator size was 4, and we set a reduction factor of 2; this would give a comm size on c' of 2.
Using the interlaced creation routine, the ranks in c with color = 0 will be rank 0 and 2.
We perform the scatter to the sub-communicator in the following way.
[1] Given a vector x defined on communicator c
<P>
<PRE>
rank(c) local values of x
------- ----------------------------------------
0 [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0 ]
1 [ 6.0, 7.0, 8.0, 9.0, 10.0, 11.0 ]
2 [ 12.0, 13.0, 14.0, 15.0, 16.0, 17.0 ]
3 [ 18.0, 19.0, 20.0, 21.0, 22.0, 23.0 ]
</PRE>
<P>
scatter into xtmp defined also on comm c, so that we have the following values
<P>
<PRE>
rank(c) local values of xtmp
------- ----------------------------------------------------------------------------
0 [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0 ]
1 [ ]
2 [ 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0 ]
3 [ ]
</PRE>
<P>
The entries on rank 1 and 3 (ranks which do not have a color = 0 in c') have no values
<P>
<P>
[2] Copy the values from ranks 0, 2 (indices with respect to comm c) into the vector xred which is defined on communicator c'.
Ranks 0 and 2 are the only ranks in the subcomm which have a color = 0.
<P>
<PRE>
rank(c') local values of xred
-------- ----------------------------------------------------------------------------
0 [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0 ]
1 [ 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0 ]
</PRE>
<P>
Contributed by Dave May
<P>
<H3><FONT COLOR="#CC3333">Reference</FONT></H3>
Dave A. May, Patrick Sanan, Karl Rupp, Matthew G. Knepley, and Barry F. Smith, "Extreme-Scale Multigrid Components within PETSc". 2016. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC '16). DOI: 10.1145/2929908.2929913
<P>
<H3><FONT COLOR="#CC3333">See Also</FONT></H3>
<A HREF="../PC/PCTelescopeGetKSP.html#PCTelescopeGetKSP">PCTelescopeGetKSP</A>(), <A HREF="../PC/PCTelescopeGetDM.html#PCTelescopeGetDM">PCTelescopeGetDM</A>(), <A HREF="../PC/PCTelescopeGetReductionFactor.html#PCTelescopeGetReductionFactor">PCTelescopeGetReductionFactor</A>(), <A HREF="../PC/PCTelescopeSetReductionFactor.html#PCTelescopeSetReductionFactor">PCTelescopeSetReductionFactor</A>(), <A HREF="../PC/PCTelescopeGetIgnoreDM.html#PCTelescopeGetIgnoreDM">PCTelescopeGetIgnoreDM</A>(), <A HREF="../PC/PCTelescopeSetIgnoreDM.html#PCTelescopeSetIgnoreDM">PCTelescopeSetIgnoreDM</A>(), <A HREF="../PC/PCREDUNDANT.html#PCREDUNDANT">PCREDUNDANT</A>
<BR><P><B></B><H3><FONT COLOR="#CC3333">Level</FONT></H3>advanced<BR>
<H3><FONT COLOR="#CC3333">Location</FONT></H3>
</B><A HREF="../../../src/ksp/pc/impls/telescope/telescope.c.html#PCTELESCOPE">src/ksp/pc/impls/telescope/telescope.c</A>
<BR><A HREF="./index.html">Index of all PC routines</A>
<BR><A HREF="../../index.html">Table of Contents for all manual pages</A>
<BR><A HREF="../singleindex.html">Index of all manual pages</A>
</BODY></HTML>
|
