GitHub

DMDA

The DMDA (Distributed Array) module provides functionality for creating and managing structured grids in 1D, 2D, or 3D.

Overview

DMDA is ideal for problems on regular structured grids where:

  • The grid is logically rectangular
  • Each grid point has the same number of degrees of freedom
  • Stencil operations follow a regular pattern (star or box stencils)

Creating a DMDA

# 2D grid example
da = DMDA(
    petsclib,
    MPI.COMM_WORLD,
    (PETSc.DM_BOUNDARY_NONE, PETSc.DM_BOUNDARY_NONE),  # boundary types
    (nx, ny),                                          # global dimensions
    1;                                                 # degrees of freedom per node
    stencil_width = 1,
    stencil_type = PETSc.DMDA_STENCIL_STAR,
)

da isa PETSc.DMDA{typeof(petsclib), 2}   # true: the flavour and the dimension

DMDA is a Julia type rather than a factory function, so the dimension is in the type and the returns are dimension-correct: PETSc.corners(da).size is a 2-tuple for the grid above, and PETSc.info(da) reports ndofs and procs (naming conventions, §5.3 and §12).

Reading the local extent

c  = PETSc.corners(da)         # (; lower, upper, size), 1-based and inclusive
gc = PETSc.ghost_corners(da)   # the same, with the ghost points
i  = PETSc.info(da)            # (; dim, global_size, procs, ndofs, …)
ndims(da)                      # the dimension, as Base spells it

Colouring for a finite-difference Jacobian

# Takes no `petsclib`, and every index field says its base (naming.md §12.1)
c = PETSc.star_fd_coloring(da)
c.row_coo_local_0b     # straight to MatSetPreallocationCOOLocal
c.perturb_cols_1b      # indexes Julia arrays

Functions

PETSc.DMDA — Method
DMDA(
    petsclib::PetscLib,
    comm::MPI.Comm,
    boundary_type::NTuple{D, DMBoundaryType},
    global_dim::NTuple{D, Integer},
    dof_per_node::Integer;
    stencil_width::Integer = 1,
    stencil_type = :box,
    points_per_proc::Tuple,
    processors::Tuple,
    set_from_options::Bool = true,
    setup::Bool = true,
    prefix::AbstractString = "",
    options...
)

Creates a D-dimensional distributed array with dof_per_node degrees of freedom per grid point.

stencil_width is the number of ghost points on each side. stencil_type is :box or :star (or a DMDAStencilType); a 1D DMDA ignores it.

If keyword argument points_per_proc[k] isa Vector{petsclib.PetscInt} then this specifies the points per processor in dimension k.

If keyword argument processors[k] isa Integer then this specifies the number of processors used in dimension k; ignored when D == 1.

With set_from_options = true, set_from_options! is applied with the remaining keywords options... as the options database, under prefix. With setup = true, setup! is called.

External Links

source
Base.size — Method
size(dm::DMDA{PetscLib, N})
size(dm::DMStag{PetscLib, N})

Return the global size of the DM as an NTuple{N,Int}.

v0.4 returned (M, N, P) with the unused dimensions padded to 1; the result is now dimension-correct (§12), which is a break with no shim (§16).

External Links

source
PETSc.corners — Method
lower, upper, size = corners(da::DMDA{PetscLib, N})

Returns a NamedTuple with the global indices (excluding ghost points) of the lower and upper corners as well as the size.

The result is dimension-correct (§12): lower and upper are CartesianIndex{N} and size is an NTuple{N,Int}, with no padding to three entries. This is a break with no shim (§16).

External Links

source
PETSc.ghost_corners — Method
lower, upper, size = ghost_corners(da::DMDA{PetscLib, N})

Returns a NamedTuple with the global indices (including ghost points) of the lower and upper corners as well as the size of the local part of the domain.

Dimension-correct like corners: CartesianIndex{N} and NTuple{N,Int}.

External Links

source
PETSc.info — Method
info(dm::DMDA)

Get information about a DMDA.

Returns

A NamedTuple with the following fields, all of them N-dimensional for an N-dimensional DMDA (docs/src/man/naming.md §12):

  • dim: Dimension of the DMDA (1, 2, or 3)
  • global_size: NTuple{N,Int} with the global dimensions in each direction
  • procs: NTuple{N,Int} with the number of MPI processes in each direction
  • ndofs: Degrees of freedom per node
  • stencil_width: Width of the stencil
  • boundary_type: NTuple{N,DMBoundaryType} with the boundary type per direction
  • stencil_type: Stencil type, either DMDA_STENCIL_STAR or DMDA_STENCIL_BOX

v0.4 reported the stencil width twice, as s and as stencil_width, spelled the dof count dof and the process grid mpi_proc_size, and padded every tuple to three entries. All four are gone; this is a break with no shim (§16).

External Links

source
PETSc.local_interior_linear_index — Method
ind = local_interior_linear_index(dmda::DMDA)

Returns the linear indices associated with the degrees of freedom own by this MPI rank embedded in the ghost index space for the dmda

source
PETSc.reshape_local_array — Method
reshape_local_array(Arr, da::Union{DMDA, DMStag}, ndof = ndofs(da))

Returns an array with the same data as Arr but reshaped as an array that can be addressed with global indexing.

source
PETSc.star_fd_coloring — Method
star_fd_coloring(da::DMDA{PetscLib, 2})

Build all data needed for manual FD coloring of a 2-D DMDA with a STAR stencil, using IS_COLORING_LOCAL and ghost-local COO indexing.

petsclib is not an argument: the DMDA carries it as a type parameter (docs/src/man/naming.md §8). A DMDA of another dimension, or with a BOX stencil, throws an ArgumentError: the coloring would leave out entries.

Specifically, this function:

  1. Creates an IS_COLORING_LOCAL ISColoring via DMCreateColoring and extracts the per-DOF color vector (ghost-local layout).
  2. Enumerates all STAR-stencil (row, col) pairs for every owned node and records their ghost-local 0-based indices and colors.
  3. Builds per-color index arrays (perturb_cols_1b, coo_idxs_1b, local_rows_1b) ready for use in an FD coloring Newton loop.

Returns a NamedTuple. The index vectors carry the base they use in their name (§12.1), because this function returns both: the COO arrays go straight to LibPETSc.MatSetPreallocationCOOLocal and keep PETSc's base, while the rest index Julia arrays and are 1-based.

  • n_colors — number of colors
  • n_local_dofs — number of locally owned DOFs (owned nodes × dof/node)
  • nnz_coo — total number of COO entries
  • row_coo_local_0b — ghost-local 0-based row indices (Vector{PetscInt})
  • col_coo_local_0b — ghost-local 0-based column indices (Vector{PetscInt})
  • perturb_cols_1b — perturb_cols_1b[c]: 1-based owned-local column indices with color c-1; used to scatter +h perturbations.
  • coo_idxs_1b — coo_idxs_1b[c]: 1-based COO entry indices for color c-1
  • local_rows_1b — local_rows_1b[c]: corresponding 1-based owned-local residual-row indices; used to read (f1-f0)/h.
2-D DMDA STAR stencil only

Neighbor enumeration covers only ±x and ±y directions. The ghost-local flat-index formula is d + ix*dof + iy*dof*nx_g. For a 3-D DMDA:

  • add ±z neighbors guarded by kk > 1 / kk < mz,
  • extend the flat-index formula with + iz*dof*nx_g*ny_g,
  • reshape col_colors_mat to (dof, nx_g, ny_g, nz_g),
  • decode z_owned in the perturb_cols_1b loop.
source