Advances in High Performance Computing and Deep Learning: Data Engineering and Data Science

Big Data Systems and HPC

1

December 16 2020

Geoffrey Fox, Digital Science Center, Indiana University,

gcf@indiana.edu,

​

The IEEE DSS-2020 Virtual Conference

http://www.ieee-cybermatics.org/2020/dss/

​

​

Digital Science Center

Big Data Systems and HPC

Abstract

• We discuss Big Data Systems noting different requirements for data science with compute-intensive execution of deep/machine learning and data engineering with many pre-processing, post-processing, and data management tasks.
• High performance computing HPC will be critical especially in the data science component.
• We need to integrate the Java (data engineering) C++ (deep learning) and Python ecosystems.
• This architecture suggests a cloud of modest-sized AI-accelerated systems in a sea of commodity servers. We discuss how Twister2 and Cylon linked to a wealth of other capabilities help to address this.
• Deep Learning (DL) is rapidly replacing other AI (ML) and we give 3 examples
• MDS Multidimensional scaling for dimension reduction
• Surrogates for simulations and data analytics;
• Spatial time series where we give application examples from COVID-19 daily data, solutions of ordinary differential equations, and other fields of science generating geospatial time series.
• Most computing could be done on the HPC hybrid clouds described in the first paragraph with the largest scale simulations performed on (exascale) supercomputers deriving the surrogates that allow many simulations to be performed on smaller machines.
• Recently 2 Gordon Bell and the CASP awards went to molecular science simulations enhanced by deep learning surrogates which highlights new modes of science discovery.
• We review the core AI issues needed to advance the study of surrogates including neural and hyperparameter search, mapping of patterns to meta-surrogates, uncertainty quantification, and minimization of training set size.

2

Digital Science Center

Big Data Systems and HPC

What we will discuss today

• Deep Learning is here and will continue to grow in importance. It will dramatically accelerate both large scale data analytics (surrogates of data model) and large scale simulations (surrogates of numerical model).
• Large machines will exhibit even more accelerators (GPU’s)
• This motivates new software systems to link data science, computational science, data engineering�
• The hardware vision and software abstractions underlying future systems
• Twister2 and Cylon supporting data science and data engineering: 140,000 lines of Java, Python, C++
• Dimension reduction as example of ascendancy of deep learning
• Deep learning again: Universal (meta)surrogates as heart of computational science; the many CS challenges for surrogates
• Time series as another example of deep learning
• Conclusions: MLCommons and interactions with Industry

3

Digital Science Center

Big Data Systems and HPC

Data Engineering and Deep Learning

What do we need to support?

​

​

4

12/7/2019

Digital Science Center

Big Data Systems and HPC

Deep Learning Workflow

5

Workflow often divide into two:

Data => Information preprocessing -- Hadoop, Spark, Twister2, Scikit-Learn

Information => Knowledge Compute intensive step Cylon enhanced Spark Twister2, PyTorch and Tensorflow

Post-Processing

Data

Digital Science Center

Big Data Systems and HPC

6

NIPS 2015 http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

This well-known paper points out that parallel high-performance machine learning is perhaps most fun but just a part of system. We need to integrate in the other data and orchestration components.

​

This integration is not very good or easy partly because data management systems like Spark are JVM-based which doesn’t cleanly link to C++, Python world of high-performance ML

​

ML code module is itself built up hierarchically from Numpy and Pandas operations (if Python)

Need to assemble 10 large modules into full workflow and efficiently execute Numpy/Pandas inside modules

​

​

​

Integrating Data Engineering and Data Science

Digital Science Center

Big Data Systems and HPC

Some Important Trends I

• Leadership systems (Supercomputers, HPC Clouds) all have high speed networking, NVMe/SSD on the nodes and fast access to an object store
• HDFS less important as object store data networked
• Computing is changing as CPU performance is not increasing as fast as before
• So ratio GPU: CPU (or Accelerator:CPU) is increasing in 3 measures: number of GPU’s/CPU’s, contribution to purchase cost and contribution to performance
• Both simulations and data analytics learn to use GPU
• Long ago GPU:CPU was numerically 0:1 and then became 2:2 (each node with 2 CPU’s and 2 GPU’s) with some (not all) jobs able to get factors of ~5 performance from GPU’s
• Stress on performance led to machines like Summit at ORNL with 6:2 GPU:CPU and growing reliance on GPU’s
• NVIDIA DGX with 8 A100 GPU’s and 2 AMD 64 core CPU’s
• Next generation HPC systems could be near 50:1 GPU:CPU contribution to performance as this is soundest way to get high performance as this certainly fits deep learning which will give important flagship benchmarks

7

Digital Science Center

Big Data Systems and HPC

Some Important Trends II

• 50:1 GPU:CPU contribution to performance has major impact on application codes
• Ratio even higher for low precision as is possible to use in deep learning
• Implications for Data Systems
• Deep learning work loads “trivially” excel on GPU’s and emphasizes accelerator heavy machines
• What fraction of job is the deep learning rather than the data engineering around this?
• Is data engineering compute or I/O dominated?
• Modin + Pandas and NVIDIA RAPIDS (CuDF) suggest data engineering built around operators (including SQL as in BlazingSQL) that can be accelerated on GPU’s
• Implications for Simulations
• Deep Learning surrogates will increase in use and not need large numbers of nodes
• Will need training sets run as native simulations but maybe fewer of them than at present
• Will this shift happen?
• Need to understand how individual surrogates will support a rich set of simulations; use transfer learning to modify surrogates?

8

Digital Science Center

Big Data Systems and HPC

Data Engineering and Deep Learning

A solution with Cylon and Twister2

​

​

9

12/7/2019

Digital Science Center

Big Data Systems and HPC

Digital Science Center

Big Data Systems and HPC

Data Engineering

• Data engineering includes producing structured data from raw data with ETL Extract-Transform-Load operations.
• Data engineering enables Deep Learning(DL) and Machine Learning (ML) workflows.
• No clear requirements but needs
• Java ecosystem important with networking focus
• Python ecosystem for user-facing capabilities
• C++ ecosystem for performance

Digital Science Center

Big Data Systems and HPC

Twister2

Big Data Processing�Eco-System

Dataflow

API

Dataflow

API

Twister2

Linear/Relational Algebra Operators

Distributed Linear/Relational Algebra Operators [C++]

Distributed Relational Communication Operations [C++]

Communication Kernels

Twister2 one of 5 possible engines for Apache Beam, which implements a rich data processing (dataflow) workflow environment – other engines are Spark, Flink, Samza, Google Cloud Dataflow

​

​

Cylon

Lines of Code�Twister2 125000 (Java, Python)

Cylon 15000 (C++, Python, Java)

​

Digital Science Center

Big Data Systems and HPC

244 Pandas Operators: Core Data Functions

• Some are very simple but others like SQL hide a jungle of complexity

pandas.DataFrame, index, columns, dtypes, info, select_dtypes, values, axes, ndim, size, shape, memory_usage, empty, astype, convert_dtypes, infer_objects, copy, bool, head, at, iat, loc, iloc, insert, __iter__, items, iteritems, keys, iterrows, itertuples, lookup, pop, tail, xs, get, isin, where, mask, query, add, sub, mul, div, truediv, floordiv, mod, pow, dot, radd, rsub, rmul, rdiv, rtruediv, rfloordiv, rmod, rpow, lt, gt, le, ge, ne, eq, combine, combine_first, apply, applymap, pipe, agg, aggregate, transform, groupby, rolling, expanding, ewm, abs, all, any, clip, corr, corrwith, count, cov, cummax, cummin, cumprod, cumsum, describe, diff, eval, kurt, kurtosis, mad, max, mean, median, min, mode, pct_change, prod, product, quantile, rank, round, sem, skew, sum, std, var, nunique, value_counts, add_prefix, add_suffix, align, at_time, between_time, drop, drop_duplicates, duplicated, equals, filter, first, head, idxmax, idxmin, last, reindex, reindex_like, rename, rename_axis, reset_index, sample, set_axis, set_index, tail, take, truncate, backfill, bfill, dropna, ffill, fillna, interpolate, isna, isnull, notna, notnull, pad, replace, droplevel, pivot, pivot_table, reorder_levels, sort_values, sort_index, nlargest, nsmallest, swaplevel, stack, unstack, swapaxes, melt, explode, squeeze, to_xarray, T, transpose, append, assign, compare, join, merge, update, asfreq, asof, shift, slice_shift, tshift, first_valid_index, last_valid_index, resample, to_period, to_timestamp, tz_convert, tz_localize, attrs, plot, plot.area, plot.bar, plot.barh, plot.box, plot.density, plot.hexbin, plot.hist, plot.kde, plot.line, plot.pie, plot.scatter, boxplot, hist, sparse.density, sparse.from_spmatrix, sparse.to_coo, sparse.to_dense, from_dict, from_records, to_parquet, to_pickle, to_csv, to_hdf, to_sql, to_dict, to_excel, to_json, to_html, to_feather, to_latex, to_stata, to_gbq, to_records, to_string, to_clipboard, to_markdown, style,

• 224 Pandas Operators
• 140 CuDF (NVIDIA RAPIDS) Operators
• Currently 36 of these in Cylon
• 40 Spark operators Plus 29 in Pandas collection and 23 SQL operations similar to Pandas
• BlazingSQL has GPU enabled SQL operators
• These are “small” (fine-grain) atomic operations with which you build a microservice or coarser grain capability - as fine grain, you run “in-place” exploiting in memory Arrow storage
• Cylon runs these fine grain operators
• Spark or Twister2 links multiple Cylon calls to build a (micro)service or node of a dataflow graph ( operations at one node are “in-place”)
• Need to ensure coarse and fine-grain operators run with high performance on all platforms

13

Digital Science Center

Big Data Systems and HPC

Two Ecosystems

Enterprise: Java

Pre and Post Data Engineering

Research Labs, Universities: Python

Central Data Science (Deep Learning)

GOAL: High Performance in each ecosystem �and high-performance integration between ecosystems!

Digital Science Center

Big Data Systems and HPC

Twister2 Benchmarks with Big Data Processing Ecosystem

This benchmarks full (micro)services

Digital Science Center

Big Data Systems and HPC

Cylon: A High Performance Distributed Data Table

• Cylon is a high performance C++ kernel and a distributed runtime for data pre-processing
• Apache Parquet and Arrow based storage and in-memory data structure
• Supports seamless integration with Deep Learning workloads, Pandas and Numpy
• Zero-Copy data transfer between heterogeneous systems and languages.
• Table API, an abstraction for ETL (extract, transform, load) for scientific computing and deep learning workloads including Pandas, HDF5
• Join, Union, Intersect, Difference, Product, Project … 36 operators
• Currently we support Joins (all formats) and �other components (see all those 140(244) operators) are currently in development.
• Written in C++, APIs available in Java and Python (via Cython).
• Cylon is the high performance kernel of Twister2.
• Future link to RAPIDS (for NVIDIA GPU) BlazingSQL (SQL operators) Other accelerators such as AMD and Intel GPU’s

​

16

Digital Science Center

Big Data Systems and HPC

Cylon Architecture

Builds on Apache Arrow to��Link Python world: Jupyter, Numpy, Pandas, Modin

(Cython links Python to C++ Cylon)

​

with Java world�Spark, Twister2

​

and C++/CUDA world�High performance deep learning on PyTorch and Tensorflow

​

Table/Dataframe interface for usability

Digital Science Center

Big Data Systems and HPC

Strong Scaling Comparison with Other Frameworks

• 200M records per table (left and right)
• 160 Processes accross 10 Intel® Xeon® Platinum 8160 with 255GB RAM in each mode and mounted SSD. InfiniBand with 40Gbps bandwidth

Inner join

Union

Digital Science Center

Big Data Systems and HPC

Large Scale Experiments with PySpark and PyCylon

• Inner join - 10B records per relation (left/right)
• 160 Processes accross 10 Intel® Xeon® Platinum 8160 with 255GB RAM in each mode and mounted SSD. InfiniBand with 40Gbps bandwidth

Digital Science Center

Big Data Systems and HPC

Strong Scaling Comparison with Other Frameworks

• 200 Million records per table (left and right)
• 160 Processes accross 10 Intel® Xeon® Platinum 8160 with 255GB RAM in each mode and mounted SSD. InfiniBand with 40Gbps bandwidth

Aggregations

Group-by + aggregations

Digital Science Center

Big Data Systems and HPC

Cylon Performance with Language Bindings

• 200M records per relation (left/right)
• 160 Processes accross 10 Intel® Xeon® Platinum 8160 with 255GB RAM in each mode and mounted SSD. InfiniBand with 40Gbps bandwidth

Digital Science Center

Big Data Systems and HPC

Interoperability Goals for Cylon

• Keep functionality focus: High performance for in-place heterogeneous tables
• Include need for data spilled to disk
• Is performance of operators the best metric? � If so, relatively clear how to include GPU and measure success
• Hardware Devices: Intel CPU, Arm, NVIDIA GPU (via RAPIDS), AMD GPU (via OpenCL?), Intel GPU (via OneAPI)
• Language ecosystems: Python, Java, C++
• Data Environments: Spark, Twister2, Flink, Storm
• Operating Systems: Data or Communications or Compute or Container centric models
• Jupyter notebooks,
• Dask, Ray,
• Legion, MPI Execution environment, Radical Pilot
• Kubernetes, Argo or other workflow

​

Digital Science Center

Big Data Systems and HPC

Dimension Reduction from Classic MDS (improved by DA) to Deep Learning

Example of DL replacing previous methods

23

12/7/2019

Digital Science Center

Big Data Systems and HPC

MDS Approaches: Deep Learning v Previous Best Practice

1. DAMDS with Smith-Waterman for distance calculation -- sophisticated deterministic annealing -- parallelized
2. Autoencoder with One Hot Encoding to represent input vectors
3. Autoencoder with K reference based approach to represent input vectors

Digital Science Center

Big Data Systems and HPC

Silhouette Coefficient SC

Silhouette Coefficient for different approaches and differing K

​

​

• The silhouette value is a measure of how similar an point is to its own cluster (cohesion) compared to other clusters (separation). SC is the mean of all s(i) values
• ​

​

a(i) - sum of distance to points in same cluster

b(i) - min sum of distance to points in other clusters

​

• The complex nature of the clusters in 3D contributes to the negative SC value even with cluster separation visible in 3 dimensional plots
• For larger K the increased size (more trainable parameters) of the Autoencoder contributes to lower SC values

Digital Science Center

Big Data Systems and HPC

170K fungi Gene sequence HeatMaps

Heatmaps of Smith-Waterman distance vs projected distance in 3D

(a)

(b)

(c)

(a)

(b)

(c)

Digital Science Center

Big Data Systems and HPC

Out of Sample Data

Some Lessons

• 30K gene sequence dataset
• Set of genes left out during autoencoder training as out of sample data.
• Reference sequence based autoencoder with K=400

​

• It worked but Smith-Waterman distances subtle and need to feed in
• by representing sequences by the vector of distances to reference vectors

Digital Science Center

Big Data Systems and HPC

Deep Learning Surrogates

28

12/7/2019

Digital Science Center

Big Data Systems and HPC

Some Important Trends II

• 50:1 GPU:CPU contribution to performance has major impact on application codes
• Ratio even higher for low precision as is possible to use in deep learning
• Implications for Data Systems
• Deep learning work loads “trivially” excel on GPU’s and emphasizes accelerator heavy machines
• What fraction of job is the deep learning rather than the data engineering around this?
• Is data engineering compute or I/O dominated?
• Modin + Pandas and NVIDIA RAPIDS (CuDF) suggest data engineering described by operators (including SQL as in BlazingSQL) that can be accelerated on GPU’s
• Implications for Simulations
• Deep Learning surrogates will increase in use and not need large numbers of nodes
• Will need training sets run as native simulations but maybe fewer of them than at present
• Will this shift happen?
• Need to understand how individual surrogates will support a rich set of simulations; use transfer learning to modify surrogates?

29

Digital Science Center

Big Data Systems and HPC

Let’s look at ML for HPC

• Traditionally we did HPC for ML -- run deep learning on a GPU BUT ML for HPC is probably more interesting (stems from Jeff Dean 2017 NeurIPS)
• Currently in science the ML is used to enhance simulations (not data analytics) and dominantly the ML used is Deep Learning
• Introduced 3 major categories and 8 subcategories
1. Improving Simulation with ML-controlled Configurations and Integration of Data
2. Use ML to Learn Structure, Theory and Model for Simulation
• SC20 Gordon Bell winners a) Learn multi-particle potential b) Learn to explore phase space
3. Use ML to Learn Surrogates for Simulation
• Work with Shantenu Jha
• https://arxiv.org/abs/1909.13340
• https://arxiv.org/abs/1909.02363
• 100 refs classified but already out of date
• Will produce surrogate exemplars and benchmarks

​

​

​

30

Digital Science Center

Big Data Systems and HPC

Up to two billion times acceleration of scientific simulations with deep neural architecture search

• January 23 2020 https://arxiv.org/pdf/2001.08055.pdf
• 10 scientific cases including astrophysics, climate science, biogeochemistry, high energy density physics, fusion energy, and seismology, using the same super-architecture, algorithm, and hyperparameters.
• Approach also dynamically choses deep network and provides uncertainty estimation, adding further confidence in their use.

31

Digital Science Center

Big Data Systems and HPC

Examples of ML for HPC (work with JCS Kadupitiya, Vikram Jadhao)

• Uses quite small Multi-layer perceptron MLP to predict 150 observables from 5 input parameters (~5000 in training set)
• MLP outperforms other ML choices
• Deployed on nanoHUB for education (an attractive use of surrogates so students get answers fast)
• General Electric uses similar approach to give interactive Engine design options (200 in training set)

32

→ 10⁶ as N_lookup → ∞

The Learning Net

Direct simulation compared to Surrogates

• Extraction of ionic structure in electrolyte solutions confined by planar and spherical surfaces.
• Classic HPC code written with C++ and accelerated with hybrid MPI-OpenMP.

Digital Science Center

Big Data Systems and HPC

INSILICO MEDICINE USED CREATIVE AI TO DESIGN POTENTIAL DRUGS IN JUST 21 DAYS

• Map Drug (Material) Structure to Drug (Material) Properties
• Hong Kong-based Insilico Medicine sent shockwaves through the pharma industry after publishing research in Nature Biotechnology that proves its AI-powered drug discovery system was capable of producing at least one potential treatment for fibrosis in less than a month's time.
• The system uses a Deep Reinforcement Learning algorithm that can imagine potential protein structures based on existing research and certain preprogrammed design criteria.
• Insilico's system initially produced 30,000 possible designs, which the research team whittled down to six that were synthesized in the lab, with one design eventually tested on mice to promising results.
• Insilico's AI-powered research process could offer a massive push forward for the pharmaceutical industry, which faces increasingly high drug development costs. In just a handful of weeks and for approximately $150,000, Insilico delivered what typically takes pharmaceutical companies $2.6 billion over seven years.

33

1/7/2020

September 4 2019 News Item

Digital Science Center

Big Data Systems and HPC

Operator Formulation of Deep Learning Inference

• Suppose we are solving PDE’s or sets of coupled ODE’s
• Typically we solve iteratively New Values = (Differential Operator O) Previous Values
• Classic applied math tells you nifty difference equations and spectral methods to represent Operator numerically
• Deep Learning learns the operator from classic numerics or observational data or their combination
• Inference is New Values = (DL Operator O) Previous Values
• This new nonlinear trained DL operator can allow much larger time steps, incorporate variations in parameters, learn potentials etc.
• DL Operator O is the new theory (Newton’s laws) of science
• High order approximations are traditionally very sensitive to noise and one was taught to avoid but Deep NNs are the opposite – both verbose and robust
• Note DL operator O with multiple LSTM layers has 100s-100,000 parameters
• Newton’s laws for this have 2-4 parameters

34

Digital Science Center

Big Data Systems and HPC

Learn Newton’s laws with Recurrent Neural Networks

• Deep Learning is revolutionizing (spatial) Time series Analysis
• Good example is integrating sets of differential equations
• Train the network on traditional 5 time step series from (Verlet) difference equations
• Verlet needs time step .001 for reliable integration but
• Learnt LSTM network is reliable for time steps which

35

are 4000 times longer � and also learn potential.

• Speedup is 30000 on 16 particles interacting with Lennard-Jones potentials
• 2 layer-64 units per layer LSTM network: 65,072 trainable parameters
• 5000 training simulations

JCS Kadupitiya, Vikram Jadhao

Digital Science Center

Big Data Systems and HPC

Status of Surrogates and ML for HPC

• ML for HPC broadly but current use is nonuniform across domains
• Use of modest DL network to map material/potential drug structure to properties (generalized QSAR) with simulation and observation: Advanced Progress
• Learn surrogates for large scale simulations: good results with major speedup but not yet developed in many fields
• Use of MLforHPC in agent-based systems (learn agents replacing by surrogates): Very promising but few results
• Use in Sociotechnical simulations and in virtual tissues (agents are people or cells)
• Macroscopic Structure as in learn complex multi-particle potentials scaling to N⁷: many great successes
• Learn Collective coordinates and guide ensemble computations: dramatic progress with speedups up to 10⁸
• Microscale; learn dynamics of small scale such as NCAR with clouds, turbulence: Interesting results but much more to do
• Use of Recurrent NN’s to represent dynamics (learn numerical differential operators): Promising but only studied in small problems

36

Digital Science Center

Big Data Systems and HPC

Futures of Surrogates and Computational Science

• Is Surrogate-based Computational Science the future?
• Traditional simulations used to train very flexible surrogates
• Some production examples (as in GE’s use in engine design) clear
• Value of surrogates in education clear
• Challenge is cases where a simulation set (mapping to a single surrogate) is small
• Surrogates run well on GPU’s; traditional simulations may run well on GPU’s

37

• Need a neural architecture (meta-surrogate) which can both describe individually many different simulation sets and for each set be as broad as possible
• So a computational scientists can base their work around one (or a few) surrogates

​

Digital Science Center

Big Data Systems and HPC

Research Issues

Neural Search

• Meta-surrogates need a flexible architecture that can be optimized over range of simulation sets. NAS (Neural Architecture Search) limited use so far
• related to hyperparameter search

Uncertainty Quantification

• statistics: dropout, hyper-parameter variation, Bayesian probability
• physics: ergodicity of distribution of training points across phase space

Minimizing Size of Needed Training Set

• Surrogates have challenge of expensive simulations but advantage they can choose simulation to perform
• Need approaches that can find the “best” next training point or set of points -- perhaps by minimizing overall uncertainty�
• I expect major progress over next 3 years

​

38

Digital Science Center

Big Data Systems and HPC

Diversion on Time Series

39

12/7/2019

Digital Science Center

Big Data Systems and HPC

Times Series represented by Deep Learning

• Molecular Dynamics Solution is “just” a time series and we saw DL derived an operator O to describe this
• Classic applied math also gives a numeric operator O based on Newton’s Laws
• m a(t) = F(x,t)
• v(t+δt) = v(t) + δt a(t)
• x(t+δt) = x(t) + δt v(t)
• An operator that produces a time series x(t), v(t), a(t) initialized by values at t= 0 and 1 (W window length = 2)
• Time Series or rather sequences have two important cases
• Forecast the future of time series observables Seq2Forecast Seq2Future
• Predict new observables in time interval of time series - sequence to sequence map Seq2Seq
• Geospatial time series are very common with space x as one variable and can be of either Seq2Seq or Seq2Forecast
• Consider spatial bags where x is just a label and
• Spatial geometries where “euclidean geometry” useful and data often looks like a sequence of images

​

40

Digital Science Center

Big Data Systems and HPC

Basic Spatial (bag) Time Series

41

Data Analysis Unit

Time sequence at one space point

For Natural Language Processing, space points are different paragraphs or books. A few sentences at each point.

Digital Science Center

Big Data Systems and HPC

General Deep Learning Strategies II

• Attention means that you “learn” from other related data (the past)
• Transformer clearly looking for patterns as Q and K match
• Transformer uses matrix features by comparing structure (scalar product) across other time and space points
• LSTM uses history passed through time sequence
• Just a few studies of Transformer for forecasting problem
• We combined transformer encoder to find patterns and followed with an LSTM decoder - called hybrid transformer

42

Hybrid Transformer (for encoder) and LSTM (for decoder)

Pure LSTM

Two different models used

Final

Initial

LSTM Layer

LSTM Layer

Outputs

Inputs

optional but best results if you do this

Digital Science Center

Big Data Systems and HPC

Search Strategies in Hybrid (Science) Transformer

• Choose group of items (space-time collections) to be considered together
• This attention group considered as batch in FFFFWNPF
• Choose how attention will be looked for
• a) Temporal: At each location look over time window O(NW²)����
• b) Spatial: Look across items at fixed position in time window O(N²W)����
• c) Full: Look over all space time windows in batch O(N²W²)

43

Each color is a different location with 5 time values�N=10 W=5 in exampler

Digital Science Center

Big Data Systems and HPC

44

Time Series

Benchmarks

​

We are looking at examples of time series including hydrology (rainfall) and epidemiology as in COVID infection and fatality data.

Latter dataset is currently 314 cities and 205 days

​

Models are LSTM and Transformer�W=9 N=159

Digital Science Center

Big Data Systems and HPC

Hybrid Transformer (Attention+LSTM) with intrinsic error for 314 cities/counties and 159 days

45

Digital Science Center

Big Data Systems and HPC

CAMELS Hydrology Data Sequence Length 21. Simple search along axes. 671 locations, 7000 days

46

Runoff with some locations missing data so location sum observation not plotted - only ~2000 missing points over 671 locations, 7000 days

Digital Science Center

Big Data Systems and HPC

Implementation Issues

• Need custom training and custom loss function in Tensorflow
• Cannot construct windowed sequences ahead of time; rather dynamically create windows at time batch members chosen; no loss in efficiency -- gain huge factors in memory
• Attention search space can also run out of memory -- use parallelism or divide search space into subspaces
• Real-time response; methods need to be reviewed for real-time inference scenarios when the large search space that can be used by the transformer can significantly increase prediction time latency
• Time Series software FFFFWNPF divides work into five stages:
• pre-notebook data engineering
• specialized notebook input; convert particular application to generic structure
• generic data pre-processing; e.g. scale data, add positional encoding, add futures, process missing data, generate windows
• training; recover from jumps in loss
• visualization.

47

Digital Science Center

Big Data Systems and HPC

Collection of Time Series Machine Learning Algorithms (MLPerf)

48

Xinyuan Huang from Cisco and MLPerf

Digital Science Center

Big Data Systems and HPC

Conclusions

• Twister2 and Cylon ready to go
• High Performance Computing
• MLforHPC
• MLPerf Benchmarks and Tutorials

49

12/7/2019

Digital Science Center

Big Data Systems and HPC

MLCommons (MLPerf) Consortium Deep Learning Benchmarks

Some Relevant Working Groups

• Training
• Inference (Batch and Streaming)
• TinyML (embedded)
• Power
• Datasets
• HPC (DoE Labs)
• Research
• Science Data
• Best Practice, Infrastructure

​

​

​

​

​

50

Benchmarking, Datasets, Best Practices

Total Effort ~50 FTE

• Major effort to produce benchmarks with ongoing challenges
• Training at V0.7 (third release)
• Set up Science Data Working Group with co-chair Tony Hey who has a significant benchmarking group SciML
• Identified ~12 science benchmarks including light source, satellite, surrogate and time series

Digital Science Center

Big Data Systems and HPC

Conclusions

• Parallel Computing growing in importance with Deep Learning, Surrogates and operator-based big data giving new challenges
• Everything should aim at (GPU) acceleration
• Surrogates can remake computational science
• Opportunities in ML for HPC as well as HPC for ML
• Switch to Deep Learning for Big Data is making good progress
• Many new algorithms to be developed including for geospatial time series
• Consider Science Research Benchmarks in MLCommons
• Include surrogates
• Use to produce tutorials
• Enhance collaboration between Industry and Research; HPC and MLCommons/MLSys communities
• Twister2/Cylon offer much of advantages of Spark with attention to both HPC and deep learning; streaming and batch; Jupyter notebooks; Java Python and C++
• Cylon tracking CuDF, BlazingSQL, and RAPIDS

​

51

Digital Science Center

Big Data Systems and HPC

Call for Action!

• Use 140,000 lines of open source Cylon, Twister2 https://cylondata.org/ https://twister2.org/
• Use open source software FFFFWNPF Deep Learning for Time Series https://colab.research.google.com/drive/1dUwrxlUq8G8HHTzImJXmJ9qiUFabdIKf?usp=sharing ��
• Develop Best Practice, Datasets, Tutorial material, Benchmarking Technology
• Join MLCommons Science Data working group by joining MLCommons Forum (automatic at web site https://groups.google.com/u/2/g/mlcommons ) and then request to join working groups at https://mlcommons.org/en/get-involved/
• We would have a special relationship with the HPC MLCommons working group as much scientific data is analyzed by HPC systems.
• Join these other working groups!

52

Digital Science Center

Big Data Systems and HPC

Thank you!

External Collaborators at Argonne National Lab, Arizona State University, Kansas, Rutgers, Stony Brook, UT Knoxville, University of Virginia and MLPerf

Indiana University Digital Science Center:

Faculty: David Crandall, James Glazier, Vikram Jadhao, Judy Qiu and others

Staff: Josh Ballard, Gary Miksik, Fugang Wang, Chathura Widanage,

Researchers: Gurhan Gunduz, Supun Kamburugamuve, Ahmet Uyar, Gregor von Laszewski

Students: Vibhatha Abeykoon, Bo Feng, JCS Kadupitiya, Niranda Perera, Pulasthi Wickramasinghe,

​

53

Digital Science Center

Big Data Systems and HPC