Ninjacart Data Strategy

Vijaykumar Shenbagamoorthy

Ninjacart is India’s largest Fresh Produce Supply Chain Company.

​

We source fresh vegetables and fruits from farmers and deliver them to kiranas and businesses across the country within 12 hours by leveraging innovative technology.

OUR VISION

​

We are pioneers in solving some of the hardest supply chain problems of the world.

​

We are focused on using our innovation to build a more efficient supply chain to improve the lives of farmers, businesses, and consumers in a meaningful manner.

Who are We?

Our Solution

Ninjacart Product Portfolio

Content

• Data Domains
• Data Platform Architecture
• Data Platform Reference Architecture
• Lego Blocks of Data Platform
• Lakehouse Adoption
• Unified Data Platform – Overview
• Defensive Data Strategy towards Governance
• What's Next?

Data Strategy

Data Domain Overview

Farmers

Organization

Analytical Platform Reference Architecture

Layers of Data Platform

Lakehouse Adoption

COW vs MOR

• Copy-on-Write: With COW, when a change is made to delete or update a particular row or rows, the datafiles with those rows are duplicated, but the new version has the updated rows.

COW vs MOR

• Merge-On-Read : With merge-on-read, the file is not rewritten, instead the changes are written to a new file. Then when the data is read, the changes are applied or merged to the original data file to form the new state of the data during processing.

TPC-DS (Transaction Processing Performance Council Decision Support)

Iceberg

Folder

Structure

Multi-Dimensional Sort

ZOrder - Deltalake

// read as delta format

DeltaTable deltaTable = DeltaTable.forPath(inputMapper.getSparkSession(),"/tmp/ds-partition/");

​

// Do Zorder Partition

dataset = deltaTable.optimize().executeZOrderBy(cols.stream().toArray(String[]::new));

Data Platform Architecture

Yesterday’s data architecture can’t meet today’s need for speed, flexibility, and innovation

Focus Areas

• Reference Architecture – Stick to Booze Allen Reference Architecture
• Data Domain based Tech stack selection
• 3^rd or 4^th Generation Data Platform
• Automated Data Ingestion & Mapping - Rapidly load data from anywhere, in any format, at scale using Apache Spark.
• Data Catalog - Secure, organize and govern all your data.
• Enterprise Knowledge Graphs - A semantic layer to link and contextualize your data
• Discovery & Analytics
• Easy of use – Data Engineering and Data Analytical Team

Ninjacart Data Platform | Architecture

Ninjacart Data Platform | Technical Architecture

Data Platform - Components

Application and Platform Integration with Data Platform

GEMS – Generic Entity Micro Service

Codeless Microservice Platform

Asserted Data

Workflow Events to Data Platform

Observed Data

Unified Data Management

Unified Pipeline Architecture

Industry First Unified Data Pipeline Framework

• ETL/ELT
• Deduplication
• Reconciliation
• Data Quality/Profiling

Data Governance Strategy

Pillars of Data Governance

Data governance focuses on the daily tasks that keep information usable, understandable, and protected

Defensive vs Offensive Approach

• Defensive data strategy focuses on minimizing risk. The activities associated with this include:
• Regulatory compliance mandates like data privacy and financial reporting laws
• Detecting and mitigating risk of fraud and theft
• Identifying, standardizing, and governing authoritative data sources
• Offensive data strategy supports business objectives. These activities include:
• Gaining insight about customer needs
• Integrating customer and market data for planning future business goals
• Supporting the sales and marketing pipelines
• Operational efficiency and process improvement

“When it comes to choosing your data strategy, it doesn’t have to be an either-or decision between offense and defence,” says Willem Koenders, a renowned consultant in the field of data strategy.

What’s Next?

What’s Next?

• Domain Centric Data Mesh Architecture – New Consumption Access pattern based on GEMS with support for Ask APIs 🡪 Where we define clear bounded context and expose Data Products for consumption
• Data Exchange Platforms
• Attribute Based Access Control (ABAC)

​

Ninjacart Data Platform Evolution – Quick Recap

Data Platform Architects

Surya Prabhakar

Karthik K

Vijay

• Data Domain
• Reference Architecture – Data Platform
• Different Terminologies used in Data World
• How to Design Data Platform
• Implementation alternatives available
• Data and Governance Strategy

​

​