Partition tool
The partition tool performs two functions:
Convert json (
.mesh
) format meshes to binary HDF5 (.h5
)Partition an HDF5 mesh format for n MPI ranks.
To use CHM in MPI mode requires using an HDF5 mesh file. When using CHM with MPI, the mesh is partitioned at run-time for the appropriate number of MPI ranks. In this configuration CHM will load the entire base mesh (despite only operating on a small portion of it) into memory and load the corresponding subset from the parameter file. As a result, the memory usage for each MPI rank is high.
The partition tool allows for pre-partitioning the mesh and parameter file into n chunks, one for each MPI rank. Therefore only the mesh elements for this rank are loaded, dramatically reducing memory overhead.
Warning
The mesh should be have been permuted prior to hdf5 conversion for maximum MPI performance. Failure to do so will result in significantly degraded runtime performance!
Please see Mesh permutation for more details.
Usage
Options:
--help
--mesh-file
,-m
,
--param-file
,-p
,
--max-ghost-distance
,-g
--standalone
,-s
--mpi-ranks
--valid-ranks
,-v
--write-vtu
--help
Outputs the help message
--mesh-file
The mesh file to operate on. Either json or h5 format. However, formats cannot be mixed between mesh-file
and param-file
.
--mesh-file <filename>
--mesh-file my_mesh.mesh
--mesh-file my_mesh.h5
--param-file
A list of the parameter files. Either json or h5 format. However, formats cannot be mixed between mesh-file
and param-file
.
--param-file <filename>
--param-file my_mesh.param
--param-file my_mesh_param.h5
--max-ghost-distance
The maximum distance at which to include ghost triangles. Certain modules need triangles at a distance, for example
to compute shadowing or fetch. --max-ghost-distance
defaults to 100 m however this should be changed to match the
configuration option for various modules.
--max-ghost-distance <distance in meters>
--max-ghost-distance 100
--standalone
Produces a specific ranks’ mesh in a way that allows it to be loaded by itself. This allows debugging on a an individual rank. In order to be able to be loaded, the mesh is written without ghost regions. Zero-indexed.
--standalone <rank to output>
--standalone 5
--mpi-ranks
Number of MPI ranks to split the mesh for. Ranks must be >1.
--mpi-ranks <n ranks>
--mpi-ranks 32
--valid-ranks
Only writes a subset of the partition out. This option will ensure the global IDs and MPI rank owners have been rewritten to support loading this subset into CHM. This is useful if a specific set of ranks is causing problems that should be debugged in isolation.
For example, suppose there is a 448 rank partition that has a problematic interaction on ranks 199 and 208. Using
./partition -m my.partitioned.mesh.h5 -p params.h5 --mpi-ranks 448 -v 199 -v 208
will produce an output of only those two ranks, using the domain partition for the 448 ranks.
--write-vtu
Output each partition set as a separate vtu file for debugging.
Output
Note
If --mpi-ranks
is specified and the input mesh is in json format (.mesh),
the tool will, in two steps, convert the mesh to h5 and then partition the h5 mesh.
json to hdf5
Output will be two .h5
files called basename_mesh.h5
and basename_param.h5
.
For example if the input mesh is granger1m.mesh
then basename is granger1m
resulting in the files
granger1m_mesh.h5
granger1m_param.h5
partition
A json file is written basename_mesh.np<MPI_RANKS>.partition
. For example, the above file with 2 mpi ranks is
granger1m_mesh.np2.partition
. The contents of this file describe the partition with various metadata and allow CHM
the ensure the file, when loaded, matches the runtime MPI configuration.
{
"ranks": "2",
"max_ghost_distance": "100",
"num_global_faces": "37645",
"meshes": [
"granger1m_mesh.np2.partition.meshes\/granger1m_mesh.partition.0_mesh.h5",
"granger1m_mesh.np2.partition.meshes\/granger1m_mesh.partition.1_mesh.h5"
],
"parameters": [
"granger1m_mesh.np2.partition.meshes\/granger1m_mesh.partition.0_param.h5",
"granger1m_mesh.np2.partition.meshes\/granger1m_mesh.partition.1_param.h5"
]
}
Secondly a new folder is created basename_mesh.np<MPI_RANKS>.partition.meshes
that holds the underlying h5 mesh and param files.
Partition is MPI aware and can be run with multiple processors. This will not speed up the json -> h5 creation, but it will allow parallel decomposition. Each MPI rank must be able to hold the entire mesh in memory. The ranks used to run parition need not be the same number of ranks used in the domain decomp. For example,
# use 8 mpi ranks to decompose granger1m_mesh into 20 sub-domains
mpirun -np 8 ./partition --mesh-file granger1m_mesh.h5 --param-file granger1m_param.h5 --mpi-ranks 20
Example Usage
./partition --mesh-file granger1m.mesh --param-file granger1m.param # json to hdf5
./partition --mesh-file granger1m_mesh.h5 --param-file granger1m_param.h5 --mpi-ranks 2 # 2 MPI ranks
./partition --mesh-file granger1m_mesh.h5 --param-file granger1m_param.h5 --mpi-ranks 2 --standalone 1 # Make the 2nd rank standalone