The distributed_descriptor class template#
This page describes the distributed_descriptor class template and its member functions, which belongs to the oneapi::mkl::experimental::dft namespace and is declared in oneapi/mkl/experimental/distributed_dft.hpp (file to be included).
namespace oneapi::mkl::experimental::dft {
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
class distributed_descriptor;
}
The usage of prepended namespace specifiers oneapi::mkl::experimental::dft is omitted below for conciseness while the namespace specifier oneapi::mkl::dft is kept for clarity.
Users of the distributed DFT DPC++ interface of oneMKL must use instances of a specialization of this class template to specify and fully configure the required global DFT computation(s): successfully-committed objects of a (specialized) distributed_descriptor class are required arguments to the distributed DFT specific compute functions.
Template parameters#
The distributed_descriptor class template is parameterized by two non-type template parameters, in the following order:
a value of type
oneapi::mkl::dft::precision, that determines the floating-point format to be considered by its instances;a value of type
oneapi::mkl::dft::domain, that determines the type of forward domain to be considered by its instances.
Instances of a distributed_descriptor class specialized with value oneapi::mkl::dft::precision::SINGLE (resp. oneapi::mkl::dft::precision::DOUBLE) for the former are referred to as “single-precision descriptors” (resp. “double-precision descriptors”). Similarly, instances of a distributed_descriptor class specialized with value oneapi::mkl::dft::domain::COMPLEX (resp. oneapi::mkl::dft::domain::REAL) for the latter are referred to as “complex descriptors” (resp. “real descriptors”).
Constructors, assignment operators, and destructor#
This section describes the constructors and destructor of the distributed_descriptor class template.
namespace oneapi::mkl::experimental::dft {
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
class distributed_descriptor {
public:
// parameterized constructors:
distributed_descriptor(MPI_Comm Comm, std::vector<std::int64_t> dimensions);
// destructor
~distributed_descriptor();
// unsupported copy constructor and assignment operator:
distributed_descriptor(const distributed_descriptor&) = delete;
distributed_descriptor& operator=(const distributed_descriptor&) = delete;
// unsupported move constructor and assignment operator:
distributed_descriptor(distributed_descriptor&&) = delete;
distributed_descriptor& operator=(distributed_descriptor&&) = delete;
}
}
Parameterized constructors#
The parameterized constructors of any distributed_descriptor class allocate memory for an object’s data structures and default-configure it for the precision, forward domain, and length(s) of the transform it defines. These constructors do not trigger any significant computational work in preparation of the DFT that the object defines upon creation; all such tasks are operated when the object is committed to its DFT definition and to a given sycl::queue instance.
Input parameter#
The input parameter of the constructor is detailed in the tables below. The tables use the notations presented in the introduction when referring to parameters of the DFT operation defined by the object.
Name |
Type |
Description |
|---|---|---|
|
|
The MPI communicator containing the processes among which the given DFT problem will be distributed. |
|
|
Vector of size \(d\), containing \(\lbrace n_{1}, n_{2}, \ldots, n_{d} \rbrace\), in that order (lengths of the global DFT) |
Exceptions#
The parameterized constructors may throw an oneapi::mkl::exception if
\(d < 2\) or \(d > 3\);
The dimensions have length less than the number of processes.
Any MPI related error occurs.
The construction of the
distributed_descriptorobject fails to allocate its required resources.
Copy constructor and assignment operator#
Copy construction and assignment are not supported.
Move constructor and assignment operator#
Move constructor and assignment operator are not supported.
Destructor#
The destructor of any distributed_descriptor class frees all resources allocated for and by objects of that class within the respective MPI process.
Configuration-setting member functions#
This section describes the overloaded configuration-setting member functions set_value of the distributed_descriptor class template.
namespace oneapi::mkl::experimental::dft {
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
class distributed_descriptor {
using real_scalar_t = std::conditional_t<prec == oneapi::mkl::dft::precision::DOUBLE, double, float>;
public:
void set_value(oneapi::mkl::dft::config_param, oneapi::mkl::dft::config_value);
void set_value(oneapi::mkl::dft::config_param, std::int64_t);
void set_value(distributed_config_param, std::int64_t);
void set_value(oneapi::mkl::dft::config_param, const std::vector<std::int64_t>&);
template <typename T, std::enable_if_t<std::is_integral_v<T>, bool> = true>
void set_value(distributed_config_param param, T value) {
set_value(param, static_cast<std::int64_t>(value));
}
void set_value(oneapi::mkl::dft::config_param, real_scalar_t);
template <typename T, std::enable_if_t<std::is_integral_v<T>, bool> = true>
void set_value(oneapi::mkl::dft::config_param param, T value) {
set_value(param, static_cast<std::int64_t>(value));
}
template <typename T, std::enable_if_t<std::is_floating_point_v<T>, bool> = true>
void set_value(oneapi::mkl::dft::config_param param, T value) {
set_value(param, static_cast<real_scalar_t>(value));
}
void set_value(distributed_config_param param,
const std::vector<std::int64_t> &lower_bound,
const std::vector<std::int64_t> &upper_bound,
const std::vector<std::int64_t> &strides);
}
}
The set_value functions enable oneMKL users to assign a configuration value to any writable configuration parameter either oneapi::mkl::dft::config_param or distributed_config_param of a distributed_descriptor object. Most of the set_value functions are similar to what was discussed in configuration-setting-functions and should be referenced for setting global transform configuration like forward/backward scale, etc. These params should be uniformly set across all processes. Also note that the deprecated functionality is not supported by the distributed DFT, trying to set any of the deprecated parameters will throw a oneapi::mkl::invalid_argument exception. Below, we will discuss distributed DFT specific set_value overload functions and also document the default configuration values associated with the respective configuration parameters.
Note
Regarding the
oneapi::mkl::dft::FWD_STRIDESandoneapi::mkl::dft::BWD_STRIDES, the global layout of the transform is expected to be conforming to the default layout discussed in configuring strides for forward and backward domains. For distributed DFT, in the case of real transforms, along with supporting padded data in forward domain for both in-place and out-of-place operations, the out-of-place operation additionally supports packed strides in forward domain.
Configuration values may be successfully and correctly set for a distributed_descriptor object but found invalid later on, when attempting to commit that object. Assessing the validity of a distributed_descriptor object’s configuration requires knowledge (and analysis) of all its configuration values (considered “frozen” when the object is committed). Any call to set_value resulting in a configuration change for a committed distributed_descriptor object effectively uncommits that object: indeed, such a change typically invalidates the object’s compute-readiness preparation steps operated when it was last committed. As a consequence, any such operation changes the object’s (read-only) configuration value associated with configuration parameter oneapi::mkl::dft::config_param::COMMIT_STATUS from oneapi::mkl::dft::config_value::COMMITTED to oneapi::mkl::dft::config_value::UNCOMMITTED.
Setting slab distribution#
The following member functions
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
void distributed_descriptor<prec, dom>::set_value(distributed_config_param param, std::int64_t value);
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
template <typename T, std::enable_if_t<std::is_integral_v<T>, bool> = true>
void distributed_descriptor<prec, dom>::set_value(distributed_config_param param, T value) {
set_value(param, static_cast<std::int64_t>(value));
}
enable users to set the dimension to be slab decomposed in the respective domain(forward or backward).
Accepted configuration parameters |
Accepted values |
Default value |
|---|---|---|
|
|
|
|
|
|
Note
Only the default slab distribution values are supported as noted above. Attempting to use a non-default value will result in an oneapi::mkl::unimplemented exception being thrown.
Setting custom distribution#
The following member function
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
void distributed_descriptor<prec, dom>::set_value(distributed_config_param param,
const std::vector<std::int64_t> &lower_bound,
const std::vector<std::int64_t> &upper_bound,
const std::vector<std::int64_t> &strides);
enables users to set the per-process rectangle/block’s lower and upper bounds along with the strides within that local portion.
Parameter |
Accepted values |
Definition |
|---|---|---|
|
|
|
|
|
An |
|
|
An |
|
|
An |
Note
Currently, custom distribution is not implemented. Attempting to set custom distribution will throw a oneapi::mkl::unimplemented exception.
Exceptions#
The configuration-setting member functions may throw
an
std::runtime_errorexception if an issue is found with the calling object;a
oneapi::mkl::invalid_argumentexception ifthe parameter being set is not writable;
the parameter being set is rejected, e.g., if it is inconsistent with the type of configuration value being used;
the configuration value being set is rejected for the specific configuration parameter being set.
a
oneapi::mkl::unimplementedexception if - the parameter being set is yet to be implemented.
Configuration-querying member functions#
This section describes the overloaded configuration-querying member functions get_value of the distributed_descriptor class template.
namespace oneapi::mkl::experimental::dft {
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
class distributed_descriptor {
using real_scalar_t = std::conditional_t<prec == oneapi::mkl::dft::precision::DOUBLE, double, float>;
public:
// for the type of forward domain:
void get_value(oneapi::mkl::dft::config_param, oneapi::mkl::dft::domain*) const;
// for the floating-point format:
void get_value(oneapi::mkl::dft::config_param, oneapi::mkl::dft::precision*) const;
// for integer-valued parameters:
void get_value(oneapi::mkl::dft::config_param, std::int64_t*) const;
void get_value(distributed_config_param, std::int64_t*) const;
// for vector-valued parameters:
void get_value(oneapi::mkl::dft::config_param, std::vector<std::int64_t>*) const;
// for real-valued parameters:
void get_value(oneapi::mkl::dft::config_param, real_scalar_t*) const;
// for custom distribution:
void get_value(distributed_config_param param,
std::vector<std::int64_t> *lower_bound,
std::vector<std::int64_t> *upper_bound,
std::vector<std::int64_t> *strides) const;
// for other parameters:
void get_value(oneapi::mkl::dft::config_param, oneapi::mkl::dft::config_value*) const;
}
}
The get_value functions enable oneMKL users to query the configuration value associated with a oneapi::mkl::dft::config_param or distributed_config_param of a distributed_descriptor object. Most of the get_value functions are similar to what was discussed in configuration-querying-functions and should be referenced for querying transform configuration like forward/backward scale, etc. The calling distributed_descriptor object is left unchanged by any call to a configuration-querying member function. Also note that the deprecated functionality is not supported by the distributed DFT, trying to query any of the deprecated parameters will throw an exception. Below, we will discuss distributed DFT specific get_value overload functions.
Querying integer-valued parameters#
The following member function
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
void distributed_descriptor<prec, dom>::get_value(distributed_config_param param, std::int64_t* value_ptr) const;
enables users to query the configuration value for the configuration parameters in the table below.
Accepted configuration parameters |
Value written by oneMKL |
|---|---|
|
|
The distributed DFT may require more memory than that which can be deduced from the data distribution for the forward or backward domains within each process. The configuration parameters distributed_config_param::fwd_local_data_size_bytes and distributed_config_param::bwd_local_data_size_bytes can be used to query the process-specific number of bytes in the respective domain (forward or backward) for which the device-accessible memory must be allocated and then initialized with input data. This must be done after the distributed_descriptor has been committed (only after which the exact number of bytes that must be allocated is known).
Querying custom distribution#
The following member function
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
void get_value(distributed_config_param param,
std::vector<std::int64_t> *lower_bound,
std::vector<std::int64_t> *upper_bound,
std::vector<std::int64_t> *strides) const;
enables users to query the custom distribution bounds for that process.
Parameter |
Accepted values |
Definition |
|---|---|---|
|
|
|
|
|
A pointer to |
|
|
A pointer to |
|
|
An |
Note
Currently, custom distribution is not implemented. Attempting to query custom distribution will throw a oneapi::mkl::unimplemented exception.
Exceptions#
The configuration-querying member functions may throw
an
std::runtime_errorexception if an issue is found with the calling object;a
oneapi::mkl::uninitializedexception if the calling object is uncommitted yet queried about a configuration parameter that requires the calling object to be committed;a
oneapi::mkl::unimplementedexception if the queried parameter corresponds to a feature that is not implemented for the calling object;a
oneapi::mkl::invalid_argumentexception ifthe pointer arguments are
nullptr;the parameter being queried is rejected, e.g., if it is inconsistent with the pointer type used as second argument;
the size of the vector pointed by
value_ptris not as required;the configuration value to be returned cannot be safely or accurately converted into the desired type, e.g., if querying a scaling factor that happens to be \(0.5\) using the configuration-querying member function specific to integer-valued configuration parameters.
Configuration- and queue-committing member function#
This section describes the configuration- and queue-committing member function commit of the distributed_descriptor class template.
namespace oneapi::mkl::experimental::dft {
template <oneapi::mkl::dft::precision prec, oneapi::mkl::dft::domain dom>
class distributed_descriptor {
public:
void commit(sycl::queue &user_queue);
}
}
Invoking this function notifies that the calling object’s configuration is completed and triggers it to complete the required initialization steps (e.g., pre-computing data, exploring various factorizations, assessing suitability of various algorithms) relevant to enqueueing the DFT computations that it defines to the user-provided user_queue. That sycl::queue object is mapped to a physical device within the available devices by the MPI.
Upon successful completion, the calling object is “committed” (its configuration value associated with config_param::COMMIT_STATUS is then config_value::COMMITTED). distributed_descriptor objects must be committed to be used in any compute function. Note that all the processes should successfully commit their distributed_descriptor object to obtain correct results.
Changing any configuration setting of a committed object effectively leaves it “uncommitted” (its configuration value associated with config_param::COMMIT_STATUS is then config_value::UNCOMMITTED). As a consequence, it is best to avoid any call to any set_value member function after invoking the commit member function.
Input parameter#
Name |
Type |
Description |
|---|---|---|
|
|
Queue to which the local DFT computations are to be enqueued by the calling object, when used in compute functions thereafter |
Exceptions#
The configuration- and queue-committing member function may throw
- a
oneapi::mkl::unimplementedexception, e.g., if the calling object’s configuration is not supported (yet); Non-default slab distribution is used.
- a
- a
oneapi::mkl::exceptionexception if, Device is not an Intel® Data Center GPU Max Series.
The environment variable
I_MPI_OFFLOADis not set to1.SYCL Backend is not Level Zero.
Default packed layouts are not used for the global array.
Batching is used (setting
oneapi::mkl::dft::config_param::NUMBER_OF_TRANSFORMS>1).Failure to allocate and initialize resources required for the
distributed_descriptorobject.
- a