1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
|
# MPICH Internal Datatypes
MPICH used to directly use MPI builtin datatype internally. There are a few issues -
* The datatype constants such as MPI_INT are decided by `configure`. This is because
internally we use handle bits to determine the builtin type size and `configure`
figures out the type size. Thus, the MPI datatype constants are not true
constants in a strict sense. They change between platforms or compilers.
* Not all MPI datatypes are supported. Configure defines unsupported datatype to
MPI_DATATYPE_NULL. This may result in compilation error when application build
on different platforms.
* MPI datatypes explodes with addition of new language bindings and new type
aliases. Treating every MPI datatype independently creates ever-increasing
support burden.
* MPI ABI proposal for MPI 5 requires implementations to dynamically configuring
datatypes. For example, MPI_Abi_set_fortran_info can set and enable Fortran
datatypes such as MPI_INTEGER and MPI_REAL at runtime. Thus, the original
`configure` design won't work.
To address these issues, we redesigned MPICH's builtin datatype support by using
a set of *internal types*. Internal types are defined in src/include/mpir_datatype.h,
with names such as `MPIR_INT32`, `MPIR_INT64`, `MPIR_FLOAT32`, `MPIR_FLOAT64`,
etc. Because they are all fixed-width types, the handle bits are also fixed and
all internal code that derives type info from handle bits still work.
In contrast, MPI builtin types such as MPI_INT, are now referred to as
*external* builtin types. The support of *external* builtin types is by mapping
them to *internal types*. For example, `configure` will define `MPIR_INT_INTERNAL`
to `MPIR_INT32` and these `_INTERNAL` macros will be used to construct a
internal table - `MPIR_Internal_types[]` in `src/mpi/datatype/typeutilc` that
maps external builtin type to internal types. The table can be updated at
runtime.
For communication, we usually only need the type size and treat the data as
bytes. If you directly use *external* builtin types *internally*, it may still
work as long as the type size derived from the handle bits match. This is true
for common platforms such as x86-64 but cannot be relied on. Thus, we require
internal code always use *internal* types, e.g. `MPIR_INT_INTERNAL` rather than
`MPI_INT`. There are assertions in critical places to catch incidents when
external types are direct used internally.
To perform reduction operation, internal types need the corresponding "C" type.
This is set by configure and defined in macros such as `MPIR_INT32_CTYPE`. The
`_CTYPE` macros may be not defined if the corresponding internal type does not
have a corresponding C type or it is not supported by the platform and compiler.
For example, `MPIR_INT128_CTYPE` may be undefined (in contrast, `MPIR_INT128` is
always defined).
Yaksa kernels may also need map to basic C type to work even for communications.
## External/Internal Type Conversions
The binding layer will use the macro `MPIR_DATATYPE_REPLACE_BUILTIN` to convert
MPI builtin datatypes from user input into internal types before passing into
internal routines. Thus, internal routines only need support the finite set of
internal types. Because we don't need all handle bits to encode an internal
type, we use the last byte to encode the original MPI builtin type which it is
converted from. For example, `MPI_INT` will be converted to `0x4c810405`
althout `MPIR_INT32` is `0x4c810400`. Thus, we can recover the original
datatype when we need to, such as the case of calling user callbacks.
Because of this, if internally we need to switch and compare to internal
constants, we need mask off the index bits. There are following helper macros:
* `MPIR_DATATYPE_GET_ORIG_BUILTIN(type)` - retrieve the original builtin type
* `MPIR_DATATYPE_GET_RAW_INTERNAL(type)` - mask off the index bits
It will be good to know there is no impact to derived datatypes.
## Migration Tips
Downstream migration is simple but unfortunately tedious. Basically, you need to
find every places where *external* builtin types are directly used and replace
them by its internal macros. Commonly used internal types include - MPI_INT,
MPI_BYTE, MPI_AINT, etc. Replace them by `MPIR_INT_INTERNAL`, `MPIR_BYTE_INTERNAL`,
`MPIR_AINT_INTERNAL`. Of course, if the internal use case is fixed-size types,
you can directly use internal types such as MPIR_INT64.
|
