Learning Record Store�Architecture

Jason Paluck and Damian Doyle

UMBC

www.umbc.edu

Data Warehouse Connections:

- PS Student (+ FinAid and Student Billing)

- PS Finance

Incoming

- PS HR

- OUE (Academic Policy Exceptions)

- First Year Experience Alerts

- LRC (Tutor)

- LRC (Supplemental Instruction)

- Symplicity (Accommodate)

- Global Viewer (Projectors)

- RT (Request Tracker)

- PT (Project Tracker)

- ImageNow

- Blackboard Learn

- NuPark (Parking Services)

- Symplicity (Residence)

- Campus Card

- GradesFirst

- Point and Click (Health Services)

- UMBC Bookstore

- myUMBC Check My Activity (CMA)

- myUMBC Course Catalog

- Central Payroll Bureau

- Division of Professional Studies

Bi-directional

Outgoing

Legend

- Daily

- Bi-weekly

- Quarter hour

- Semesterly

- On demand

- Visual Arts (AV equipment; card swipe)

- Symplicity (Advocate [Judicial])

- UMBCWorks (Career Center)

- myUMBC

- ARMS (Athletes)

- Pharos (Printing Kiosks)

- PowerLogic (Electric Meters)

- UniversityTickets

www.umbc.edu

UMBC Learning Record Store

• ~300 million Blackboard Caliper events
• ~10 million VitalSource Caliper events
• Rex (data warehouse) contextual data
• MyUMBC student portal contextual data
• Blackboard Analytics for Learn contextual data

www.umbc.edu

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 1: Ingest Events

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 1: Ingest Events

• HTTP POST requests from LMS or tool vendor
• Accepts arrays of Caliper events as JSON-LD
• AWS API Gateway
• Fully-managed API endpoint at scale
• AWS Cloudfront
• High performance API edge-caching

www.umbc.edu

Step 2: Unpack Events

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 2: Unpack Events

• AWS Lambda
• Serverless Python environment for transforming payload JSON array into individual Caliper events
• Invoked by API Gateway upon POST
• Individual events passed into AWS Kinesis Firehose as data stream
• Fetch vendor contextual data from GCP/internet

www.umbc.edu

Step 2: Unpack Events

​

def lambda_handler(event, context):

firehose = boto3.client('firehose')

try:

for loadedEvent in event['data']:

dumpedEvent = json.dumps(loadedEvent)

response = firehose.put_record(DeliveryStreamName='blackboard_caliper_events', Record={'Data': dumpedEvent})

logger.info('Event received!')

except Exception as e:

logger.info('Event {0}'.format(event))

logger.error(e)

www.umbc.edu

Step 3: Data Lake Storage

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 3: Data Lake Storage

• AWS Kenisis Firehose
• “Circuit capacitor”
• Accepts data stream of individual Caliper events from AWS Lambda
• Writes newline-delimited JSON into text files in AWS S3 data lake for long-term source data storage

www.umbc.edu

Step 4: ETL

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 4: Extract, Transform & Load

• AWS Glue
• Fully-managed ETL service
• Scheduled to run every 6 hours
• Crawls AWS S3 data lake to discover changes in JSON schema
• Transforms data from Caliper event hierarchy into relational schema in AWS Redshift

www.umbc.edu

Step 4: Extract, Transform & Load

www.umbc.edu

Step 5: Data Warehouse

Amazon S3

Amazon Kinesis Firehose

Caliper Event Stream

www.umbc.edu

Step 5: Relational Data Warehouse

• Relational schema (LRS) for analytics/end-user consumption
• Periodic ingestion of contextual data sources
• Student portal (myUMBC)
• SIS (Through Data warehouse)
• Vendor-specific dimensional data

www.umbc.edu

www.umbc.edu

Ingesting Streaming Campus Activity Data

“Student 123 declared major X”

• JMS messages, web services, transaction logs, etc
• Event data enriched and converted to standardized, semi-structured profiles
• Event payloads streamed to AWS S3 data lake

• Data securely stored in 5MB line- delimited text files
• Organized by profile type (source)
• SNS initiates data load into Snowflake in-memory analytics warehouse

PeopleSoft SIS

Apache NiFi

AWS S3 Data Lake

“WiFi client ABC dwelled 17 minutes in building 1, floor 2, room X”

Cisco CMX WiFi

JMS

REST API

www.umbc.edu

Tomorrow: Student Analytics Hub

19

Streaming Event Data

Blackboard Caliper Events

VitalSource Caliper Events

Ingest Event Streams

• Ingest streaming vendor Caliper events
• Ingest streaming contextual events (grades, enrollment, bio/demo, etc)

Enrichen and Load

• Store events in AWS S3 data lake
• Lossless, transaction data remains very close to source

Transform in Memory

• Curated data sets (views) matched to consumer need
• Optimized for analytics discovery

SIS Events

Student Portal Events

WiFi Location Events

Student Activity Events

AWS S3 Data Lake

Blackboard Caliper Events

VitalSource Caliper Events

SIS Events

Student Portal Events

WiFi Localtion Events

Student Activity Events

Snowflake Warehouse

Semi-structured data

Curated Views

www.umbc.edu