C++#
-
namespace twoscale#
Functions
-
template<typename M, typename V>
class coarseManager# - #include <coarseManager.h>
- Template Parameters:
M – The matrix type
V – The vector type
-
template<>
class coarseManager# - #include <coarseManager.h>
Specialization of twoscale::coarseManager class with M,V being respectively a PETSc Mat and Vec.
Public Functions
-
coarseManager(Mat &&PSR, Mat &&PE, Vec &&W, std::vector<std::int64_t> diag_dof_eliminated, std::unordered_map<std::int32_t, twoscale_dolfinx::enrichedDofIDs> &&enriched_dof_id_)#
Class constructor that check communicator coherance of its arguments and initialize privates members
Note
User won’t call it directelly a priori but will use twoscale_dolfinx::generateCoarseManager function
- Parameters:
PSR – Standard part restriction to passe from coarse to fine scale (interpolation coefficients).
PE – Enriched part of operator Q to passe from coarse to fine scale (interpolation coefficients).
W – Vector representing Dirichlet BC values imposed at coarse level and projected on fine scale field (PSD.XDc with PSD eliminate column of PS and XDc Vector representing Dirichlet BC values imposed at coarse level )
diag_dof_eliminated – Set of index (global) coresponding to all eliminated rows/columns that will be used to fill diagonal term of \(PSR^t.A_{ff}.PSR\) block with one.
enriched_dof_id_ – Map of global dofs index pair (enriched coarse dof,coresponding fine dof) indexed by topological index of enriched nodes at coarse level (passed to twoscale_dolfinx::patchManager to update \(PE^t\))
Update PE, the enriched part of the Q operator. When called for the first time it transfert PE coefficiant into patch object owning the row corresponding to coarse enriched node. In following call those coefficiants are used directely from patches.
- Parameters:
pm – Patch manager used to update PE.
func – Function applyed to patch solution to obtain enrichment function
-
void projectStdCoarse(Vec xcs, Vec xf)#
Project, using the standard scale operator, an enriched vector at coarse scale into a vector at fine scale
- Parameters:
xcs – vector at coarse scale representing coarse enriched field
xf – vector at fine scale representing fine field
-
coarseManager(Mat &&PSR, Mat &&PE, Vec &&W, std::vector<std::int64_t> diag_dof_eliminated, std::unordered_map<std::int32_t, twoscale_dolfinx::enrichedDofIDs> &&enriched_dof_id_)#
-
template<typename T>
class enrichedFunction# - #include <enrichedFunctions.h>
Subclassed by twoscale::enrichedShiftFunction< T, bs >
-
template<typename T, int bs>
class enrichedShiftFunction : public twoscale::enrichedFunction<T># - #include <enrichedFunctions.h>
-
template<typename T1, typename T2>
class equalPair# - #include <util.h>
-
template<typename T1, typename T2>
class hashPair# - #include <util.h>
-
class patch#
- #include <patchManager.h>
Subclassed by twoscale::patchMPIDirect, twoscale::patchMPIMagma, twoscale::patchOMPDirect
-
class patchMPIDirect : public twoscale::patch#
- #include <patchManager.h>
No openMP only MPI (i.e. each patchs can’t be treated in parrallel by multiple threads). Linear system resolution use parallel direct solver Local patch use sequential resolution Distributed patch use parrallel resolution
Public Functions
-
void generateProblem(Mat Aff, Vec bf, Vec computf_, MPI_Comm comm, MPI_Comm univ, std::int32_t nbdl, std::int32_t bs)#
Function that generate patch system of equation using given fine scale system.
Function that update \(PE^t\) terms relate to this patch (object) based on current solution of this patch system The first time this method is called the \(PE^t\) rows associated to this patch are copied in internal storage so that during next calls updating can be done using this storage as \(PE^t\) no longer contains interpolation coefficients but resulting terms from last call.
- Template Parameters:
F – Type representing functor that transform patch solution into enrichment function
- Parameters:
PEt – [inout] Operator to update.
enriched_dof_id – [in] Used only in the first call, this container gives way to identify row(s) of \(PE^t\) related to this patch and the fine dofs related to enriched node
func – [in] Functor to transform field solution of this patch problem into enrichment function
comm – [in] General communicator on which coarse/fine meshes are distributed
univ – [in] Communicator on which this patch is distributed
bs – [in] Block size supposed to be the same for coarse and fine field
rstart – [in] First row of \(PE^t\) stored in this proc
rend – [in] Past the last row of \(PE^t\) stored in this proc
-
void solveProblem(Vec xf, MPI_Comm comm, MPI_Comm univ, std::int32_t bs)#
Solve problem for this patch
- Parameters:
xf – [in] Petsc Vector storing fine field solution approximation (from last TwoScale Loop for example)
comm – [in] General communicator on which coarse/fine meshes are distributed
univ – [in] Communicator on which this patch is distributed
bs – [in] Block size supposed to be the same for coarse and fine field
-
void generateProblem(Mat Aff, Vec bf, Vec computf_, MPI_Comm comm, MPI_Comm univ, std::int32_t nbdl, std::int32_t bs)#
-
class patchMPIMagma : public twoscale::patch#
- #include <patchManager.h>
variant to implement patchMPIMagma use Magma or any GPU solver for patch resolution. All patch must use GPU Distributed patch needs to gather the Matrix on 1 proc/GPU to launch computation on one device. From experience in eXlibris using parallel direct solver on distributed patch and Magma resolution on local patch alleviate the possibility of increasing scale jump and get gains.
-
class patchOMPDirect : public twoscale::patch#
- #include <patchManager.h>
variant to implement patchOMPDirect use OpenMP to compute local patch in multi-threaded mode using a thread safe direct solver like Pardiso and usual parallel direct solver for distributed patch. From experience in eXlibris this hybrid computation did not show high gain but implementation might has been poorly done.
-
template<typename M>
class scaleJump# - #include <scaleJump.h>
Class that store fine and coarse mesh with the connections in between them.
- Template Parameters:
M – The mesh type
Public Functions
-
inline scaleJump(M &&fine_mesh_, M &&coarse_mesh_, bool needs_mpc_, std::vector<std::int32_t> &&cell_child_offset_, std::vector<std::int32_t> &&cell_child_, std::vector<std::int32_t> &&face_child_offset_, std::vector<std::int32_t> &&face_child_, std::int32_t max_nb_face_child_, std::vector<std::int32_t> &&sface_child_offset_, std::vector<std::int32_t> &&sface_child_, std::vector<std::int32_t> &&cell_parent_, std::vector<std::int32_t> &&enriched_nodes_, std::vector<std::int32_t> &&extra_enriched_nodes_, std::vector<std::int32_t> &&support_, std::vector<std::int32_t> &&surrounding_cells_, std::unordered_set<std::int32_t> &&fine_node_master_, std::vector<std::int32_t> &&coarse_interface_master_)#
Class constructor
Note
User won’t call it directelly a priori but will use twoscale_dolfinx::topDown function
- Parameters:
fine_mesh_ The – fine mesh instance of type M
coarse_mesh_ The – coarse mesh instance of type M
needs_mpc_ – A boolean indicating if mpc are required or not
cell_child_offset_, cell_child_ – The vector cell_child_ store fine scale cell local id packed by group corresponding to coarse scale cell.
-
const std::shared_ptr<M> getFineMesh() const#
Function that give a shared pointer of the fine scale mesh.
-
const std::shared_ptr<M> getCoarseMesh() const#
Function that give a shared pointer of the coarse scale mesh.
-
std::span<const std::int32_t> getChildren(std::int32_t coarse_cell_idx) const#
Function that return a list of fine scale cells local ids. These cells are embodied in coarse scale cell corresponding to local index given as argument. If no fine cell is associated to this coarse cell it return an empty list.
Warning
In the context of a distributed mesh, a ‘local index’ is a numbering system used by each process to identify entities that it store. This is usually complemented by a global numbering system covering all processes. Here the FEniCs numbering is implicitly used.
- Parameters:
coarse_cell_idx – Index of the coarse scale cell for which we want embedded fine scale cells (children)
-
std::span<const std::int32_t> getFaceChilds(std::int32_t coarse_face_idx) const#
Function that return a list of fine scale faces (in 3D and edges in 2D or nodes in 1D) local ids. These faces are embodied in coarse scale face corresponding to local index given as argument. If no fine face is associated to this coarse face it return an empty list.
Warning
In the context of a distributed mesh, a ‘local index’ is a numbering system used by each process to identify entities that it store. This is usually complemented by a global numbering system covering all processes. Here the FEniCs numbering is implicitly used.
- Parameters:
coarse_face_idx – Index of the coarse scale face for which we want embedded fine scale faces (children)
-
std::span<const std::int32_t> getEnriched() const#
Function that return a list of coarse scale nodes local ids. These nodes corresponds to the enriched nodes provided by the user at creation of the instance.
-
std::span<const std::int32_t> getExtraEnriched() const#
Function that return a list of coarse scale nodes local ids. These nodes corresponds to the enriched nodes added to treat blending problem at interfaces of enriched/non enriched area.
-
std::span<const std::int32_t> getSupport() const#
Function that return a list of coarse scale cells local ids. These cells corresponds to the support of the enriched nodes given by scaleJump::getEnriched
-
std::span<const std::int32_t> getSurroundingCells() const#
Function that return a list of coarse scale cells local ids. These cells corresponds to the support of the enriched nodes given by scaleJump::getExtraEnriched minus the cells provided by scaleJump::getSupport
-
bool isFineMasterNode(std::int32_t i) const#
Function that return true if index given as argument correspond to a local fine scale id of a nodes which is a master node in a MPC relationship.
- Parameters:
i – fine scale local index of the node to test
Public Members
-
const bool needs_mpc#
Member indicating if MPC are mandatory or not (i.e. if mesh jump exist in fine scale mesh).
-
template<typename M, typename V>
-
namespace twoscale_dolfinx#
Typedefs
-
template<typename T, std::size_t D>
using mdspan_t = MDSPAN_IMPL_STANDARD_NAMESPACE::mdspan<T, MDSPAN_IMPL_STANDARD_NAMESPACE::dextents<std::size_t, D>># multi dimensional span type shortcut to basix mdspan implementation declaration
- Template Parameters:
T – Type of the data stored in the array warped by mdspan
D – integer describing number of dimension of the mdspan
Functions
Function that generate a twoscale::coarseManager<Mat, Vec> object from arguments. The coarse scale system ( \(A_{cc}\), \(B_{c}\)) is in PETSc nested format managed by coarseManager instance created by this function. Only operators used to compute sub blocks are created by this function and provided to coarseManager instance (via the constructor). The field to describe coarse enriched system are split in two independent fields:
A simple coarse regular field to deal with standard dof (std). Its space (blocked) is given by the user (coarse_space). Boundary conditions (Dirichlet) given by the user (bc) are eliminated at operator level and define two set :
D for Dirichlet eliminated dofs
R for Retained free dofs
A simple coarse regular field to deal with enriched dof (enr). This field is defined only on the support of enriched nodes defined in j (given by user as parameter). Its space is created automatically from coarse_space characteristics and either the sub or full coarse mesh. If all nodes are enriched coarse_space is used for this field.
The operator that are constructed are thus:
\(P_{SR}\) size nb_fine x (nb_R): pass from retained standard dof to fine scale dof (given by user: fine_space)
\(P_E^t\) size nb_fine x (nb_enr): pass from enriched dof to fine scale dof (given by user: fine_space)
If bc use non null imposed value the following are also create by this function:
\(XD_{c}\) size nb_D the vector of imposed value on coarse_space
\(P_{SD}\) size nb_fine x (nb_D) a temporary coarse to fine operator
\(W=P_{SD}.XR_D\) size nb_fine : vector of coarse bc projected on fine field
Note
This function implemented is only available with PETSc
- Template Parameters:
T – The field scalar type (e.g. float,std::complex<double>,…).
U – The mesh geometry/space scalar type (float or double).
- Parameters:
j – The twoscale::scaleJump instance that give all jump information (mesh transition information)
fine_space – The fine scale discretization space
coarse_space – The coarse scale discretization space (regular space i.e. not enriched)
mpc – if any, multi point constrain used to connect fine field dof at support interface
bc – a vector of Dirichlet boundary conditions, if any, to be imposed at coarse scale
- Template Parameters:
T – The field scalar type (e.g. float,std::complex<double>,…).
- Template Parameters:
T – The field scalar type (e.g. float,std::complex<double>,…).
U – The mesh geometry/space scalar type (float or double).
- Parameters:
j – The scaleJump object describing coarse/fine mesh and their relations
space – The fine scale disretization space
-
void setColor(twoscale::patchMPIDirect *patch, std::int64_t *trans_table, int proc_id)#
-
bool checkColor(twoscale::patchMPIDirect *patch, std::int64_t *trans_table, int proc_id)#
Function to generate a patchManager instance from scaleJump instance and dolfinx::fem::Function
- Template Parameters:
P – The type of patch managed (e.g. patchMPIDirect, …)
U – The mesh geometry/space scalar type (float or double).
- Parameters:
sj – [in] The scaleJump object describing the fine and coarse scale and there relationship.
fine_space – [in] The discretization space at fine scale level
-
template<typename T, dolfinx::mesh::MarkerFn<T> U, dolfinx::mesh::MarkerFn<T> W, typename Z>
twoscale::scaleJump<dolfinx::mesh::Mesh<T>> topDown(dolfinx::mesh::Mesh<T> &mesh, U enriched, W crit, std::uint8_t &control, std::uint8_t level, bool clustering_dual_graph = true, bool accurate_weight = false, dolfinx::mesh::MeshTags<Z> const *tags = nullptr)# Function creating a scale jump in a top down manner. Starting from
a coarse mesh distributed on some process
a set of coarse enriched nodes
a geometric criterion functor to locate the areas where mesh needs to be refined.
a level of refinement
this function do the following:
identify the support of all enriched nodes
load balance the original coarse mesh distribution so that each procces get a peace of the support
isolate the sub-mesh coresponding to the support
from this new distributed sub-mesh loop ‘level of refinement’ time to refine mesh locate in areas identified by criterion
reconnect this refined sub-mesh with ignored part of the coarse mesh. It form a new composite mesh made of coarse and fine element that will be connected later by MPC relation for dangling nodes.
compute all relation between new coarse mesh and new composite mesh for future MPC creation and TwoScale processing.
Note
User is responsible to provides a crit functor in coherence with enriched nodes support (i.e. if refined area are not in the support, refinement except the first level won’t be done)
Warning
tags argument is completely ignored for now. Implementation has to be done and certainnely somme API change to provide to the user this MeshTags as it won’t make sens to incude it in scaleJump object.
- Template Parameters:
T – The mesh geometry scalar type (float or double).
U – Based on the dolfinx concept represents a function for geometry marking
W – Based on the dolfinx concept represents a function for geometry marking
Z – The type that is stored by MeshTags argument
- Parameters:
mesh – [in] The coarse mesh in its original distribution.
enrich – [in] The set of enriched node provided by the user as a Dolfinx Marker function
crit – [in] The refinement area selection functor provided by the user as a Dolfinx Marker function with an extra control parameter. This parameter may be used by crit to tune area selection (e.g. focus on a specific location while mesh discretization becomes smaller)
control – [in] The control parameter that drive the refinement area selection functor crit. It is set to current refinement level during refinement loop and passed to crit as second argument.
level – [in] The level of refinement (i.e. the number of time areas identified by crit are refined by dolfinx::refinement::refine)
clustering_dual_graph – [in] A Boolean to force usage of clustering strategy
accurate_weight – [in] A Boolean forcing accurate weight computation by refining mesh with coarse mesh in its original distribution. Once weights are computed this refined mesh is dropped. By default (i.e. accurate_weight==false) an uniform estimate based on level and dimension is used for all weights of all support cells.
tags – [in] Optional MeshTags object to propagate during refinement
- Returns:
A scaleJump object encapsulating all new meshes and their relationship.
-
bool useNest()#
Function that tells if library is using or no the nested matrix strategy.
-
template<typename S, typename T>
void scatter_fwd(S &scatterer, std::span<const T> l, std::span<T> g)# Function encapsulating a simple scatter fwd operation It use MPI neighbourhood collective communication
- Parameters:
l – [in] Buffer containing local data to send
g – [out] Buffer containing local ghost data to be updated
- Template Parameters:
S – Type of the scatterer object
T – Type of the data to be exchanged
-
template<typename S, typename T, typename BinaryOp>
void scatter_rev(S &scatterer, std::span<T> l, std::span<const T> g, BinaryOp op)# Function encapsulating a simple scatter rev operation It use MPI neighbourhood collective communication
- Parameters:
l – [out] Buffer containing local data to be updated
g – [in] Buffer containing local ghost data to send
op – [in] binary operator that takes current local value and received corresponding value as argument. It returns local value to set.
- Template Parameters:
S – Type of the scatterer object
T – Type of the data to be exchanged
S – Type of the binary operator argument
-
template<typename S, typename T, typename BinaryOp>
void scatter_rev_fwd(S &scatterer, std::span<T> l, std::span<T> g, BinaryOp op)# Function encapsulating a scatter rev followed by fwd operation It use MPI neighbourhood collective communication
- Parameters:
l – [inout] Buffer containing local data to update and send
g – [inout] Buffer containing local ghost data to send and update
op – [in] binary operator that takes current local value and received corresponding value as argument. It returns local value to set.
- Template Parameters:
S – Type of the scatterer object
T – Type of the data to be excanged
S – Type of the binary operator argument
-
class enrichedDofIDs#
- #include <util.h>
Utility class to store enriched dofs indexes and their associated fine scale dofs indexes, both related to matrix \(PE^t\) Only the block index is stored: for coarse it is an implicit block as mixed space have bs=1. For fine it is an explicit block Eliminates row by Dirichlet boundary are encoded in eliminated_coarse_encoding
Public Members
-
PetscInt global_coarse_block_dof_idx#
enriched dof block index at coarse scale in global numbering (PETSc \(PE^t\) row numbering)
-
PetscInt global_fine_block_dof_idx#
dof block index at fine scale corresponding to global_coarse_block_dof_idx in global numbering (PETSc \(PE^t\) column numbering) This give in a patch a way to identify dofs that can be used for shift enrichment function for example
-
std::int64_t eliminated_coarse_encoding#
encoding using basic binary encoding:
each component of a block are encode on one different bit in increasing power of 2 (component 0 -> \(2^0\), component 1 -> \(2^1\), ….)
a bit is set to 1 if eliminated 0 otherwise
Thus eliminated_coarse_encoding is null if the full block is retained. And it is non null if at leas one component is eliminated
-
PetscInt global_coarse_block_dof_idx#
-
template<typename P>
class patchManager# - #include <patchManager.h>
Note
P could be suppressed as pointer to base do the job but it may help to do some specialization for some patch type in some methods
- Template Parameters:
P – The type of patch managed (e.g. patchMPIDirect, …)
Public Functions
-
patchManager(std::vector<P*> &&patches_, std::int32_t nbdl, std::int32_t bs_d, MPI_Comm comm_)#
Constructor bassed on patches provided as argument
Warning
This constructor is not intended to be called by the user who must use twoscale_dolfinx::generatePatchManager function instead.
- Parameters:
patches_ – [in] vector of pointers to patches of type P. The new instance acquire the control on them.
nbdl – [in] Number of block dofs local to this process
bs_d – [in] Block size associated to dofs
comm_ – [in] Communicator associated to fine/coarse mesh distribution
-
~patchManager()#
Destructor required to avoid internal dummy_patch pointer to be unfreed.
-
patchManager(const patchManager<P> &other)#
copy constructor required to avoid internal dummy_patch pointer to be linked in between instances
- Parameters:
other – [in] Instance copied to created one new instance
-
patchManager(patchManager<P> &&other)#
move constructor required to capture properly internal dummy_patch pointer. On output the instance provided as argument must not be used anymore
- Parameters:
other – [inout] Instance moved to created one new instance
-
void generateProblems(Mat Aff, Vec bf)#
Generates all patches system from fine scale system
- Parameters:
Aff – [in] Fine scale matrix
bf – [in] Fine scale rhs
-
void solveProblems(Vec xf)#
Solve all patches system using a fine scale field approximation
- Parameters:
xf[in] – Vector providing field values to be imposed as Dirichlet boundary condition for all patches during their resolution
Update PEt operator with current patches solution.The PEt matrix is the operator to transfers fine scale field into coarse enriched field.
During its first call the rows (corresponding to a coarse node enriched) of PEt are transferred to their associated patch instance. The process owner of the row (PETSc sens) hold in the associated patch the initial values of PEt witch are simple interpolation coefficients. These coefficients are multiplied by the associated enriched function values to generate the new PEt terms. After the first call PEt do not hold anymore the interpolation coefficients but their product with the last enrichment function computed so far. In next call interpolation coefficients comes from patches instances themselves.
The enriched function in each patch is obtained by applying the func functor to the current patch solutions stored in the patch instance.
- Todo:
F functor should be defined in a concept
Note
This function could be split in 2 steps/function. More clear in term of API/user point of view. This may be mandatory in the future if scaleJumps is updated and PEt is modified on the fly.
Warning
Has the patches solution are used in this function it implies that function patchManager::solveProblems had to be called before this function to have a correct enriched function.
- Template Parameters:
F – Functor type
- Parameters:
PEt – [inout] operator to transfers fine scale field into coarse enriched field.
enriched_dof_id – [in] A map that connect indexes. The key of the map is the FEniCSx global topological index of the coarse node enriched (it correspond to one of the ids of a patch). The data associated to the key is a twoscale_dolfinx::enrichedDofIDs corresponding to block row and column of PEt. The row is related to enriched coarse dof and the column to the dof at fine scale equivalent to enriched coarse dof. This map is only used during the first call.
func – [in] functor applied to patch solution to obtain enriched function from it
-
int numberOfSequence()#
Function that gives the number of sequence needed for patches treatement
Warning
Should not be called before at least one call to generateProblem to provide correct information
- Returns:
number of sequence used to compute all distributed patches
-
std::int32_t grabPatchSolution(std::int32_t seq, Vec xf)#
Function to copy in xf all patches solutions related to sequence seq
Warning
Only for debugging.
- Parameters:
seq – [in] Sequence id to collect. It correspond to a set of independant patche computed at the same time.
xf – [inout] A PETSc vector provided by user and filled with solutions of patches computed by sequence seq.
-
patchManager(std::vector<twoscale::patchMPIDirect*> &&patches_, std::int32_t nbdl_, std::int32_t bs_d_, MPI_Comm comm_)#
-
~patchManager()
-
patchManager(const patchManager<twoscale::patchMPIDirect> &other)#
-
patchManager(patchManager<twoscale::patchMPIDirect> &&other)#
-
void generateProblems(Mat Aff, Vec bf)
-
void solveProblems(Vec xf)
-
std::int32_t grabPatchSolution(std::int32_t seq, Vec xf)
-
patchManager(const patchManager<twoscale::patchMPIDirect> &other)
-
patchManager(patchManager<twoscale::patchMPIDirect> &&other)
-
~patchManager()
-
patchManager(std::vector<twoscale::patchMPIDirect*> &&patches_, std::int32_t nbdl_, std::int32_t bs_d_, MPI_Comm comm_)
-
void generateProblems(Mat Aff, Vec bf)
-
void solveProblems(Vec xf)
-
std::int32_t grabPatchSolution(std::int32_t seq, Vec xf)
-
template<typename T, std::size_t D>