# Changes: 3.16

% STYLE GUIDELINES:
% * Capitalize sentences
% * Use imperative, e.g., Add, Improve, Change, etc.
% * Don't use a period (.) at the end of entries
% * If multiple sentences are needed, use a period or semicolon to divide sentences, but not at the end of the final sentence

```{rubric} General:
```

- Change `MPIU_Allreduce()` to always returns a MPI error code that
  should be checked with `CHKERRMPI(ierr)`
- Add support for A64FX and Cray compilers
- Add support for ESSL 5.2 and later; drop support for ESSL \<=5.1
- Add support for NEC-SX Tsubasa Vector Engine
- Add support for NVIDIA HPC SDK

```{rubric} Configure/Build:
```

- Configure requires Python-2.7 or Python-3.4+.
- Remove `--with-kokkos-cuda-arch`. One can use `-with-cuda-gencodearch` to specify the cuda arch for Kokkos. Usually not needed since PETSc auto detects that
- For `--download-hdf5`, disable `--download-hdf5-fortran-bindings` by default
- Add OpenCascade package to PETSc and allow `--download-opencascade`
- Add support for hypre in device mode for both NVIDIA and AMD GPUs
- Extend detection of C++ dialect to C++17. Configure now also takes minimum and maximum required C++ dialect of packages into account when choosing the C++ dialect

```{rubric} Sys:
```

- Add `PetscDevice` class to manage discovered GPU devices
- Add `PetscDeviceKind`
- Add `PetscDeviceCreate()`, `PetscDeviceConfigure()`, and `PetscDeviceDestroy()`
- Add `PetscDeviceContext` class to manage asynchronous GPU compute support via a fork-join model
- Add `PetscStreamType` and `PetscDeviceContextJoinMode`
- Add `PetscDeviceContextCreate()`, `PetscDeviceContextDestroy()`, `PetscDeviceContextSetDevice()`, `PetscDeviceContextGetDevice()`,
  `PetscDeviceContextSetStreamType()`, `PetscDeviceContextGetStreamType()`, `PetscDeviceContextSetUp()`, `PetscDeviceContextDuplicate()`,
  `PetscDeviceContextQueryIdle()`, `PetscDeviceContextWaitForContext()`, `PetscDeviceContextFork()`, `PetscDeviceContextJoin()`,
  `PetscDeviceContextSynchronize()`, `PetscDeviceContextGetCurrentContext()`, `PetscDeviceContextSetCurrentContext()`, and
  `PetscDeviceContextSetFromOptions()`
- Deprecate `petsccublas.h` and `petschipblas.h` in favor of `petscdevice.h` and `petscdevicetypes.h`
- Add GPU event timers to capture kernel execution time accurately
- Remove `WaitForCUDA()` and `WaitForHIP()` before `PetscLogGpuTimeEnd()`
- Add MPIU_REAL_INT and MPIU_SCALAR_INT datatypes to be used for reduction operations
- Add MPIU_MAXLOC and MPIU_MINLOC operations
- Add `CHKERRCXX()` to catch C++ exceptions and return a PETSc error code
- Remove `PetscStack` routines from public headers, this class should now be considered private

```{rubric} PetscViewer:
```

- `PetscViewerHDF5PushGroup()`: if input path begins with `/`, it is
  taken as absolute, otherwise relative to the current group
- Add `PetscViewerHDF5HasDataset()`
- `PetscViewerHDF5HasAttribute()`,
  `PetscViewerHDF5ReadAttribute()`,
  `PetscViewerHDF5WriteAttribute()`,
  `PetscViewerHDF5HasDataset()` and
  `PetscViewerHDF5HasGroup()`
  support absolute paths (starting with `/`)
  and paths relative to the current pushed group
- Add input argument to `PetscViewerHDF5ReadAttribute()` for default
  value that is used if attribute is not found in the HDF5 file
- Add `PetscViewerHDF5PushTimestepping()`,
  `PetscViewerHDF5PopTimestepping()` and
  `PetscViewerHDF5IsTimestepping()` to control timestepping mode.
- One can call `PetscViewerHDF5IncrementTimestep()`,
  `PetscViewerHDF5SetTimestep()` or `PetscViewerHDF5GetTimestep()` only
  if timestepping mode is active
- Error if timestepped dataset is read/written out of timestepping mode, or
  vice-versa

```{rubric} PetscDraw:
```

```{rubric} AO:
```

```{rubric} IS:
```

```{rubric} VecScatter / PetscSF:
```

```{rubric} PF:
```

```{rubric} Vec:
```

- Add `VecMean()` to calculate arithmetic mean of elements of a vector
- Add `VecBoundToCPU()` to query information set with `VecBindToCPU()`

```{rubric} PetscSection:
```

- Extend `PetscSectionView()` for section saving to HDF5
- Add `PetscSectionLoad()` for section loading from HDF5

```{rubric} PetscPartitioner:
```

```{rubric} Mat:
```

- `MATMPIKAIJ` now tracks the object state of the AIJ matrix describing the blockwise action of the KAIJ matrix and
  automatically rebuilds internal data structures before executing operations with the KAIJ matrix if the state has changed
- Factorization types now provide their preferred ordering (which
  may be `MATORDERINGEXTERNAL`) to prevent PETSc PCFactor from, by
  default, picking an ordering when it is not ideal
- Deprecate `MatFactorGetUseOrdering()`; Use
  `MatFactorGetCanUseOrdering()` instead
- Add `--download-htool` to use hierarchical matrices with the new
  type `MATHTOOL`
- Add `MATCENTERING` special matrix type that implements action of the
  centering matrix
- Remove -mat_mumps_icntl_7 1 option, use -pc_factor_mat_ordering_type \<type> to have PETSc perform the ordering (sequential only)
- Add `MATSOLVERSPQR` - interface to SuiteSparse QR factorization
- Add `MatSeqAIJKron()` - Kronecker product of two `MatSeqAIJ`
- Add `MatNormalGetMat()` to retrieve the underlying `Mat` of a `MATNORMAL`
- Add `MatNormalHermitianGetMat()` to retrieve the underlying `Mat` of a `MATNORMALHERMITIAN`
- Add `VecCreateMPICUDA()` and `VecCreateMPIHIP()` to create MPI device vectors
- Add accessor routines for device index data of `MATSEQAIJCUSPARSE` matrices: `MatSeqAIJCUSPARSEGetIJ()` and `MatSeqAIJCUSPARSERestoreIJ()`
- Add accessor routines for device data of `MATSEQAIJCUSPARSE` matrices: `MatSeqAIJCUSPARSEGetArray()`, `MatSeqAIJCUSPARSERestoreArray()`, `MatSeqAIJCUSPARSEGetArrayRead()`, `MatSeqAIJCUSPARSERestoreArrayRead()`. `MatSeqAIJCUSPARSEGetArrayWrite()`, `MatSeqAIJCUSPARSERestoreArrayWrite()`
- Add support for `MATHYPRE` matrices on NVIDIA and AMD GPUs
- `MatPreallocatorPreallocate` performance [significantly improved](https://gitlab.com/petsc/petsc/-/merge_requests/4273)
- Add `MatGetColumnReductions()` developer routine to calculate reductions over columns of a matrix
- Add `MatGetColumnSums()`, `MatGetColumnSumsRealPart()`, `MatGetColumnSumsImaginaryPart()` to compute sums over matrix columns
- Add `MatGetColumnMeans()`, `MatGetColumnMeansRealPart()`, `MatGetColumnMeansImaginaryPart()` to compute arithmetic means over matrix columns
- Add `MatBoundToCPU()` to query information set with `MatBindToCPU()`
- Rename `MATHARA` in `MATH2OPUS`, supporting distributed memory operations with hierarchical matrices

```{rubric} PC:
```

- Add `PCSetPreSolve()`
- Add `PCQR` - interface to SuiteSparse QR factorization for `MatSeqAIJ`,
  `MATNORMAL`, and `MATNORMALHERMITIAN`
- Add support for `MATNORMAL` in `PCASM` and `PCHPDDM`
- Add support for BoomerAMG from `PCHYPRE` to run on NVIDIA and AMD GPUs
- `PCShellGetContext()` now takes `void*` as return argument
- Rename `PCHARA` in `PCH2OPUS`, supporting distributed memory operations with hierarchical matrices

```{rubric} KSP:
```

- `KSPGetMonitorContext()` now takes `void*` as return argument
- `KSPGetConvergenceContext()` now takes `void*` as return argument

```{rubric} SNES:
```

- Add `SNESSetComputeMFFunction()`
- Add support for `-snes_mf_operator` for use with `SNESSetPicard()`
- `SNESShellGetContext()` now takes `void*` as return argument

```{rubric} SNESLineSearch:
```

```{rubric} TS:
```

- Add `-ts_type irk` - fully implicit Runge-Kutta solvers
- Add `TSTrajectory` interface to the CAMS library for optimal offline checkpointing for multistage time stepping schemes
- Add option `-ts_trajectory_memory_type <revolve | cams | petsc>` to switch checkpointing schedule software
- Add option `-ts_trajectory_max_units_ram` to specify the maximum number of allowed checkpointing units

```{rubric} TAO:
```

- `TaoShellGetContext()` now takes `void*` as return argument

```{rubric} DM/DA:
```

- Change management of auxiliary data in DM from object composition
  to `DMGetAuxiliaryVec()`/`DMSetAuxiliaryVec()`, `DMCopyAuxiliaryVec()`
- Remove `DMGetNumBoundary()` and `DMGetBoundary()` in favor of DS
  counterparts
- Remove `DMCopyBoundary()`
- Change interface for `DMAddBoundary()`, `PetscDSAddBoundary()`,
  `PetscDSGetBoundary()`, `PetscDSUpdateBoundary()`
- Add `DMDAVecGetArrayDOFWrite()` and `DMDAVecRestoreArrayDOFWrite()`
- `DMShellGetContext()` now takes `void*` as return argument

```{rubric} DMSwarm:
```

- Add `DMSwarmGetCellSwarm()` and `DMSwarmRestoreCellSwarm()`

```{rubric} DMPlex:
```

- Add a `PETSCVIEWEREXODUSII` viewer type for `DMView()`/`DMLoad()` and
  `VecView()`/`VecLoad()`. Note that not all DMPlex can be saved in exodusII
  format since this file format requires that the numbering of cell
  sets be compact
- Add `PetscViewerExodusIIOpen()` convenience function
- Add `PetscViewerExodusIISetOrder()` to
  generate "2nd order" elements (i.e. tri6, tet10, hex27) when using
  `DMView` with a `PETSCVIEWEREXODUSII` viewer
- Change `DMPlexComputeBdResidualSingle()` and
  `DMPlexComputeBdJacobianSingle()` to take a form key
- Add `DMPlexTopologyLoad()`, `DMPlexCoordinatesLoad()`, and
  `DMPlexLabelsLoad()` for incremental loading of a `DMPlex` object
  from an HDF5 file
- Add `DMPlexTopologyView()`, `DMPlexCoordinatesView()`, and
  `DMPlexLabelsView()` for incremental saving of a `DMPlex` object
  to an HDF5 file
- Add `DMPlexSectionView()` saving a `PetscSection` in
  association with a `DMPlex` mesh
- Add `DMPlexSectionLoad()` loading a `PetscSection` in
  association with a `DMPlex` mesh
- Add `DMPlexGlobalVectorView()` and `DMPlexLocalVectorView()` saving
  global and local vectors in association with a data layout on a `DMPlex` mesh
- Add `DMPlexGlobalVectorLoad()` and `DMPlexLocalVectorLoad()` loading
  global and local vectors in association with a data layout on a `DMPlex` mesh
- Add `DMPlexIsSimplex()` to check the shape of the first cell
- Add `DMPlexShape` to describe prebuilt mesh domains
- Add `DMPlexCreateCoordinateSpace()` to make an FE space for the coordinates
- Add the automatic creation of a Plex from options, see `DMSetFromOptions()`
- The old options for `DMPlexCreateBoxMesh()` NO LONGER WORK. They have been changed to make the interface more uniform
- Replace `DMPlexCreateSquareBoundary()` and `DMPlexCreateCubeBoundary()` with `DMPlexCreateBoxSurfaceMesh()`
- Remove `DMPlexCreateReferenceCellByType()`
- The number of refinements is no longer an argument to `DMPlexCreateHexCylinderMesh()`
- Add `DMSetLabel()`
- Replace `DMPlexComputeJacobianAction()` with `DMSNESComputeJacobianAction()`
- Add `DMSNESCreateJacobianMF()`
- Change `DMPlexComputeBdResidualSingle()` to take `PetscFormKey` instead of explicit label/value/field arguments
- Add `DMPlexInflateToGeomModel()` which pushes refined points out to a geometric boundary
- Separate EGADS and EGADSLite functionality, add `DMPlexCreateEGADSLiteFromFile()`
- Remove `DMPlexReverseCell()` and `DMPlexOrientCell()` in favor of `DMPlexOrientPoint()`
- Remove `DMPlexCompareOrientations()` in favor of `DMPolytopeMatchOrientation()`
- Add `DMPlexGetCompressedClosure()` and `DMPlexRestoreCompressedClosure()`
- Add `DMPlexMetricCreate` as a helper function for creating a (P1) Riemannian metric.
- Add `DMPlexMetricCreateUniform` as a helper function for creating a uniform metric.
- Add `DMPlexMetricCreateIsotropic` as a helper function for creating an isotropic metric.
- Add `DMPlexMetricEnforceSPD` for enforcing that a metric is symmetric positive-definite.
- Add `DMPlexMetricNormalize` to apply L-p metric normalization.
- Add `DMPlexMetricAverage` to average an arbitrary number of metrics.
- Add `DMPlexMetricAverage2` to average two metrics.
- Add `DMPlexMetricAverage3` to average three metrics.
- Add `DMPlexMetricIntersection` to intersect an arbitrary number of metrics.
- Add `DMPlexMetricIntersection2` to intersect two metrics.
- Add `DMPlexMetricIntersection3` to intersect three metrics.

```{rubric} FE/FV:
```

- Change `PetscFEIntegrateBdResidual()` and
  `PetscFEIntegrateBdJacobian()` to take both `PetscWeakForm` and form
  key
- Add `PetscFEGeomGetPoint()` and `PetscFEGeomGetCellPoint` to package up geometry handling

```{rubric} DMNetwork:
```

- Add `DMNetworkCreateIS()` and `DMNetworkCreateLocalIS()`
- Remove nv from `DMNetworkAddSubnetwork()`

```{rubric} DMStag:
```

- Add `DMStagStencilToIndexLocal()`

```{rubric} DT:
```

- Add `PetscWeakFormCopy()`, `PetscWeakFormClear()`, `PetscWeakFormRewriteKeys()` and `PetscWeakFormClearIndex()`
- Add `PetscDSDestroyBoundary()` and `PetscDSCopyExactSolutions()`
- `PetscDSGetContext()` now takes `void*` as return argument
- Add `PetscWeakFormReplaceLabel()` to change labels after mesh modification

```{rubric} Fortran:
```

- Add support for `PetscInitialize(filename,help,ierr)`,
  `PetscInitialize(ierr)` in addition to current `PetscInitialize(filename,ierr)`