1 of 38

Making sense of Data-Driven Architecture

Ben Wu

Executive Architect APAC

NetApp

Eddy Ku

Co-Founder

Innovations Accelerated

06 Aug 2020

Disclaimer: This content contains our personal views, not necessarily those of our employers

2 of 38

Purpose

Sharing our insights in Innovation, Data-Driven Architecture and Human Centered Design

Sharing

Showing how Data-Driven Architecture alone is not enough, it must be tied back to human purpose

Showing

Motivating our audience’s thinking and act to create value

Motivating

3 of 38

Ben Wu

Personal

Worked in IT for over 30 years, 22 as an architect
Fact-based insights driving meaningful business transformation
Highlights�- Banking�- Airline�- Government�- Education�- MADIs
Plays bass guitar OK

Executive Architect

Maximise value realisation from data
Connect desired business outcome to foundational enablers
Drive value realisation through transformational architectures that use data to drive value in diverse and interesting ways

NetApp

Global leader in cloud data services
Empowering global organisations to change their world with a unique data fabric
Simplify hybrid multi-cloud
Securely deliver the right data, services, and applications to the right people at the right time

4 of 38

Cofounder of�Innovations Accelerated Eddy Ku

My specialty and passion is in the area of human centred design

5 of 38

The virtuous cycle for data

Innovation

Human-Centred Design

Data-Driven Architecture

6 of 38

6

Innovation and creating new value isn’t easy…

7 of 38

Innovation is hard…

8 of 38

Innovation

What is innovation?
Why do people innovate?
What are they expecting to achieve?
How successful is innovation?

Innovation requires data driven architecture to produce repeatable, optimised results

9 of 38

The virtuous cycle for data

Innovation

Human-Centred Design

Data-Driven Architecture

10 of 38

A day in the life of a�data-driven architect Ben Wu

Why do they call me “Pringle”?

What I do each day
How it relates to data-driven architecture
How it relates to you

11 of 38

Westpac Internet Banking 1998��Innovation required a data-driven architecture

“I wanted to create a customer solution beyond sending out floppy disks and build an internet banking system which applied the practice of usability and human centered design which we used to develop Westpac’s website. This technique allowed us to create a fully functional internet banking system which was revolutionary at the time,” Ian Muir
Realising Ian’s innovative vision in 1998 required a data-driven architecture
We scaled from 500 pilot users to 500,000 users within months

https://www.arnnet.com.au/article/591137/westpac-group-celebrates-20-years-internet-banking/

Westpac head of customer experience for digital, Ian Muir, was the brainchild behind Westpac’s Internet banking platform which launched in 1998.

“This may be hard to believe, but the usual process back then was to mail out floppy disks with software to our customers to install onto their PC so they could access their bank account details.

“I wanted to create a customer solution beyond sending out floppy disks and build an internet banking system which applied the practice of usability and human centered design which we used to develop Westpac’s website. This technique allowed us to create a fully functional internet banking system which was revolutionary at the time,” he said.

“The visual design was simple and text-based and it held the test of time. We paired back all the graphics so it was fast enough for people to use as there was no such thing as high speed Internet at the time.

“This more simplified Internet banking system stayed around for 15 years and was the precursor to today’s more sophisticated Westpac Live. I was proud it lasted so long – in Web year’s that’s a life time!” Muir added.

12 of 38

Data-driven architecture - Its purpose and value – why do we need it?

Balancing the need for optimisation with the need for individualisation

Efficiency, quality, repeatability,

velocity, leverage, sustainability

Individualised service, insights,

convenience, choice, simplicity

13 of 38

Development and change of architectures as a result of increasing reliance on data

14 of 38

Bad data

15 of 38

Data Fabric - an approach that enables data to drive and validate architectural value

Data is the new oil
Speed to insights and agility are the new money
Data Fabric is an approach and an architecture
Data Fabric makes data available everywhere to drive insight and decisions faster, more efficiently, greater scale and with less disruption

16 of 38

“The key indicator of failed digital projects is a poor data foundation” �ACL

https://www.cioinsight.com/it-strategy/big-data/slideshows/why-bad-data-quality-is-bad-for-business.html

“Among the top challenges in IoT adoption, data quality is the one that really needs special attention. After all, poor data is what makes wrong decisions. By developing a proof of concept, you’ll be able to assess what kind of data exactly you need and how to gather it effectively. Also, skimping on device costs by deploying the cheapest sensors is not recommended, as sub-par data quality that they provide may bring down the whole initiative.”

“Even more than 50% of smart deployments become stalled at the conceptual phase, which sadly makes some really promising IoT projects fail.”

“Among the top challenges in IoT adoption, data quality is the one that really needs special attention. After all, poor data is what makes wrong decisions.”

“…as sub-par data quality that they provide may bring down the whole initiative.”

https://www.avsystem.com/blog/why-iot-projects-fail/

17 of 38

Three data-driven architecture journeys

ActiveIQ by NetApp
DreamWorks
A Very Large Service Organisation

18 of 38

Data Driven Architecture�NetApp ActiveIQ

Protect client data
Improve client experience
Save money

Objective

Siloed per-client data
Reactive alert approach
Static data format
Growing portfolio

Challenge

Create data lake
Leverage AI
Predictive and adaptive analysis and alerting

Solution

Reduce cycle times from months to hours
Lower cost of ops by 70%
Personalised: Every client has own visibility
Shift to proactive support
Happier clients

Outcomes and Benefits

Improved customer experience and insights

Proactive responsive and remediation

Improved resource management

NetApp ActiveIQ is a prime example of a data–driven architecture.

In data science, volume, velocity, and variety matter. Each day, Active IQ receives telemetry data from hundreds of thousands of assets around the globe, adding to a multi-petabyte data lake that processes trillions of data points each month. With NetApp systems supporting diverse use cases and customers, Active IQ has a rich set of data for machine learning and predictive analytics, leading to deeper insights.

Active IQ continuously assesses telemetry data, drawing on trillions of real-time and historical diagnostic records and known risk signatures to spot potential problems before they affect businesses. Active IQ can also look across the community to identify trends and suggest ways to enhance the security and efficiency of your infrastructure.

Wisdom without action is useless. Active IQ makes it easy for you to take the “best next action” to ensure the overall health, availability, and security of your data platform. Its actionable insights and recommendations are available in the format that best suits each client. There’s a web UI for full-featured visibility and guidance, a mobile app for on-the-go alerts and advice, email reports for convenience, and new API support that allows programmatic integration of Active IQ recommendations directly into IT dashboards and change-management workflows. Active IQ also provides individualised insights and recommendations in two other related platforms: Active IQ Unified Manager and Cloud Insights.

ActiveIQ is one of my go-to information sources I use almost daily to enrich my understanding of my clients’ data needs.

19 of 38

NetApp ActiveIQ

ActiveIQ’s architecture is evolving. Recently ActiveIQ was replatformed from Hadoop to Iguazio on hybrid cloud for even more productivity gains

10 trillion data points per month collected from storage controllers globally, providing actionable intelligence for predictive maintenance and optimal data management.

20 of 38

400-500TB�per feature, 100% uptime demand

Extreme data needs
Data-driven
Finely tuned business and technology architecture

Data-driven Architecture

https://customer-pdf.netapp.com/69ee1e5571a1ca7311510e3892a2de798863929d-cs_dreamworks_animation.pdf

Compelling feature films
Create huge amounts of intricate content
Protect critical workflows

Objective

Handle many 500TB projects
Rapid multimedia expansion
Maximise artist productivity

Challenge

Scalable high-performance data platform
Unified management
Data views boost IT efficiency

Solution

2-3x performance and speed increase, power and cooling savings and reduced wait time

Benefits

When the first Shrek movie was produced in 2001, it required 6TB of data in total. In contrast, a single animated sequence from the upcoming release of How to Train Your Dragon: The Hidden World might need to draw on 50TB of data.

Many people don’t know how complex it is to create a feature-length animated film. Hundreds of artists work on any given DreamWorks feature, each of which takes three to four years to create. The process requires an average of 400TB to 500TB of data spread across 500 million or more files. It turns out that creating characters such as Hiccup, Toothless, and Astrid, and environments such as the Hidden World in the third How to Train Your Dragon film, is pretty heavy lifting, from a data and computation perspective.

A Global Entertainment Leader that Brings Dragons and Pandas to Life with Data

DreamWorks Animation has long captivated audiences with such beloved franchises as Shrek, Kung Fu Panda, Madagascar, Trolls and How to Train Your Dragon. Global audiences are invited into imaginative worlds where they fall in love with DreamWorks’ charismatic animated characters, while behind the scenes, petabytes of data come together to create magic. With NetApp technology, DreamWorks quickly and efficiently manages the massive amounts of data needed to make its animated films come to life.

NetApp has managed data for every CG animated film that’s been produced at DreamWorks.

Skottie Miller Technology Fellow, DreamWorks

Since its inception, DreamWorks has been at the forefront of technology innovation as a way to unleash the creativity of its artists. NetApp solutions are at the center of that innovation which has helped the studio create those fantastic stories and visuals that captivate audiences worldwide.

"Not only do we create data in the form of digital asset components for our films, but we use data about how those assets are created and combined to optimize our environment."

Jeff Wike | CTO, DreamWorks

21 of 38

�Bad architecture

22 of 38

Data-driven Architecture�A large service organisation with data challenges

Sustain services to the community at large
Improve service safety, quality & experience
Avoid IT disruptions and lengthy fulfillment times, cost blowouts and confusing offerings
Redistribute spending away from firefighting to innovation, adapt to new modalities
Comply with data governance mandates

Objective

Siloed data across and within divisions
Poor governance
No master data model or CMDB
Complex and strained relationships with service providers
Low IT maturity

Challenge

Discover, analyse and service-align over 10,000 workloads within weeks
Identify governance challenges and best practices to address them
Construct a data fabric strategy to offer ubiquitous, flexible simple and reliable services
Deliver automated, elastic data technology aligned to strategy and integrated into services

Solution

Common data consumption services available to all divisions with predictable business SLAs
Simplified provisioning, accounting and billing; fulfilment time down from months to minutes
Starting to break down data silos between traditional and cloud data workloads and assets
Improved data governance and compliance, automated data lifecycle

Benefits

23 of 38

Data-driven Architecture – Who owns the data?

Detailed discovery

Intelligent analysis

Identify data risks

Take action!

We found 220TB of orphaned data on 700 VMs!

Recommendations:

Identify owners and tag data
Implement processes to tag data on creation
Align tagging with CMDB
Extend tagging for research and analysis

Outcomes:

Improved operational risk management
Improved asset utilisation and ROI
Better, faster decisions supported by fact
Avoid unnecessary data change

Machine learning augments human insight to accelerate analysis

Reducing years of uncertainty to 2-4 weeks of granular discovery

24 of 38

Data-driven Architecture – Why is performance so poor?

Symptoms

Detailed

Analysis

Cause identified – demand and supply misaligned

25 of 38

Data-Driven Architecture - What does the End State Look Like?

Begin with the End in Mind – Multi Cloud Service Level Consistency and Contestability for all data assets
Simplicity and visibility eliminates orphaned data, improves service agility, quality and value

Data Asset Tagging

Policy-driven lifecycle

Tenant Securiity & Visibiliy

26 of 38

Data-Driven Architecture - Eliminate fragmented manual processes

Operational Intelligence for IT systems and services
A single data governance dashboard
Aggregates factual data from multiple sources
Meaningful data and actionable insights
REST API integrates with enterprise BI toolsets

Fragmented, incomplete, out of date data sets

No master data model

Hundreds of manual hours to assemble basic reports

Data hub correlates operational data from domain-based intelligence tools to enrich insights and inform complex decision

27 of 38

Data-Driven Architecture – Proactive data governance

Deliver exactly the data services needed - identify and execute efficiencies - improve value extraction

28 of 38

Data-Driven Architecture – Service Catalogue

Making the complex simple is complex

A data-driven architecture makes it possible!

29 of 38

Data-driven architecture

Consistency
Longevity
Awareness
Responsiveness
Faster decisions

Image: https://www.linkedin.com/pulse/step-up-your-organizational-performance-5-benefits-using-hasseltine/

But without human-centred design, it’s just a stairway to nowhere…

30 of 38

�Bad design

31 of 38

The virtuous cycle for data

Innovation

Human-Centred Design

Data-Driven Architecture

32 of 38

What is Human Centred Design (HCD)?

Human Centred Design is a creative approach to problem solving. It starts with a deep understanding of customer needs with the goal to create desirable customer solutions that challenge and improve the status quo

33 of 38

Exploring the right question to answer

Exploring the right answer

The HCD process

34 of 38

Benefits of HCD

Reduces the risk associated with launching new ideas.
Helps organisations learn faster.
Generates solutions that are innovative, not just incremental.
Drives creative confidence
Helps people navigate ambiguity and solve complex problems

David Kelly (IDEO)

“The most valuable companies in the world places design at the centre of everything it does... prototype, iterate, and build more nimbly with a human-centred approach” (Harvard Business Review)

35 of 38

HCD and Data Driven Architecture

PERSON

PROBLEM

SOLUTION

EMPATHY

36 of 38

Conclusion

Good data informs good architecture
Good architecture results in better experiences for all stakeholders

Link your data and architecture to human-centred design and innovation outcomes
Evaluate the quality and value of the data you have at your disposal
Put in place appropriate measurements to validate the results of your data-driven architecture

Invest in architectures that accelerate innovation to improve experience for your clients, users and you!

Innovation

Human-Centred Design

Data-Driven Architecture

37 of 38

Further Takeaways

Is ROT (Redundant Obsolete Trivial) data clogging up your architecture?
Is there good data that you have not yet uncovered or used to inform your architecture development?
Does your data and related architecture improve value, relevance and experience?
Think outside the box: sometimes our greatest value contribution is to simply create trust through predictability and interest through innovation – we need data !

Invest in architectures that accelerate innovation to improve experience for your customers, users and you!

38 of 38

38

Eddy Ku

+61 411 986070

eddy@iaccelerated.com

www.iaccelerated.com

Contact us to speak further about:

HCD / Innovation

Data-Driven Architecture

Ben Wu

+61 425 570 042

wub@netapp.com

www.netapp.com