Data Platform of the Future

November 12, 2021

​

Rupert Berk, UWash

Satya Kunta, NYU

Ken Taylor, UIUC

Ashish Pandit, UCSD

​

Agenda

• Introductions
• Shifting Technical Capabilities for Data Platforms
• Towards a Unified Data Platform
• Next Gen Data Analytics Platforms
• UCSD
• NYU
• Data Governance (@ UIUC)
• Discussion

Shifting Technical Capabilities for Data Platforms

Event streams support real and near real-time reporting and alerting, as well as incremental ETL scenarios (small batches).

HTTP APIs speed development, provide fast access, and enable efficient governance.

The rapid growth in AI/ML offerings by cloud service providers promise predictive and even prescriptive analytics, in addition to traditional descriptive or diagnostic analytics.

​

​

Data lakes promise faster experimentation and innovation by expanding analysis of raw data by analysts and data scientists.

Data Warehouse

RDBMS

Data Lake

Object Storage, Document, graph, in-memory

ETL

ELT

Batch

Events & Streams

Modeling

Persistence

Transformation

Increments

Traditional Programming

Machine Learning

Coding

Files & SQL

APIs

Interfaces

Towards a Unified Data Platform

Event Broker

Data Sources

Batch-Driven Apps

Event-Driven Apps

Data Persistence Service

Raw

Zone

Curated Zone

Usage Optimized Zone

Stream Processing

Batch Processing

ML

ML Inference

Query API

Stream Query API

User Interfaces

Data Science Workbench

Data Visualization

Dashboards

CDC

Database

File

Event

IoT

Adapted from Trivadis Blueprint for Modern Data Platform (v4), Guido Schmutz

UCSD’s Data Analytics Platform

<- Hierarchy slot ID + [attributes]

Hierarchy slot attributes ->

Curated views (CVs)

• Built off of activity records only
• No base tables
• CVs are built on top of viewlets
• CVs can also be built on top of other CVs
• Viewlet reuse should be high
• Reuse should be at the highest level
• CVs eliminate the need for user to do joins
• CVs are normally materialized
• Viewlets can also be materialized
• CVs handle duplicate activities (idempotency)

Stream in ->

<- Message out

<- Model development ->

<- Message out

Stream in ->

Activity Hub architecture

NYU’s Data Analytics Platform

Industry Tipping points...

Infinite Compute & Storage

Machine Learning/AI

FaaS(Function as a Service)/Serverless

Real-Time/Streaming

The 3 V’s of Future Data Platforms….

Conceptual Architecture

High Level Technical Architecture

Data Governance

Data Classification and Account Organizational Units

• Data classifications
• Public - Data that can be freely shared with the public
• Internal - Data that could result in reputational damage for the university, its operations or assets; unpublished research data; intellectual property; data exempted from FOIA requests
• Sensitive - Data that requires authorization to be accessed; FERPA student records; Employee information; information covered by NDAs; System and network configuration information
• High risk - Health information HIPAA, credit cards, bank accounts, Social Security numbers, driver’s license numbers, genetic/biometric information, government classified information
• Legal considerations
• AWS Regions are located worldwide under different laws and regulations
• AWS Organizational Units (OU)
• Allows accounts to be centrally managed by applying policies uniformly to all accounts within an OU

Data Classification and Account Organizational Units

Data Classification and Account Organizational Units

Data Classification and Account Organizational Units

Discussion