1 of 1

Surrogate Modeling via Functional Tensor Networks in Arbitrary Formats

Evolution of optimization variables projected on select 2D slices corresponding to the tensor-network model below. White traces correspond to the gradient-based algorithm while blue traces are computed via alternating least squares.

Scientific Achievement

This paper describes an efficient reverse-mode differentiation algorithm for contraction operations on arbitrary and unconventional network topologies.
We demonstrate improved performance over alternating least-squares optimization approaches and the capability to handle heterogeneous and arbitrary tensor network formats.

Significance and Impact

The gradient-based approach requires a smaller oversampling ratio (number of samples compared to number model parameters) for model recovery compared to alternating minimization algorithms.
This increased efficiency extends to fitting unstructured data of varying dimensionality and when employing a variety of tensor network formats. �

Research Details

Designed a tensor contraction tree that can be efficiently leveraged for differentiation of a full tensor network contraction using a recursive scheme that exploits (1) the bilinear property of contraction and (2) the property that trees have a single path from root to leaves.

Work under FASTMath SciDAC. For more information, contact Cosmin Safta (csafta@sandia.gov) or John Jakeman (jdjakem@sandia.gov)

Functional tensor network in hierarchical Tucker format. Figure depicts a 4D model.

A.A. Gorodetsky, C. Safta, J.D. Jakeman, “Reverse-mode differentiation in arbitrary tensor network format: with application to supervised learning”, Journal of Machine Learning Research, May 2022.

UQ of high-fidelity models typically requires large numbers of simulations. Building an approximation or surrogate for the model input-output map is an effective and popular approach to reducing the computational burden UQ and numerous techniques have been developed to build such approximations.

Existing surrogate methods can be very efficient when building approximations of models subject to independent random variables. However, most computational tools are built assuming the input random variables are independent, and there is a dearth of algorithmic options when the variables are dependent.

We use Gram-Schmidt orthogonalization (GSO) to numerically compute orthonormal polynomials for the dependent random variables. Then build an approximation by interpolating a set of samples. Interpolating random samples is ill-conditioned. To mitigate this issue generate weighted Leja sequences, which are nested sample sets, to build accurate polynomial interpolants. These sequences greedily maximize the determinant of a weighted interpolation matrix thereby balancing the need to sample in regions of high-probability with the need to keep the condition number of the interpolation matrix small.

Top figure depicts the zonotope resulting from projecting 20D i.i.d uniform variables onto a 2D subspace. The contours represent the joint density of the resulting two variables. This subspace is a linear combination of the 20 variables and represents the two directions that significantly influence the variability of the 20D function.

We build an approximation on the 2D subspace using a PCE that assumes independence and using our new approach. The l2 error in the resulting approximations for increasing number of samples is shown in the bottom figure.