Up in the Cloud: ��Service-Oriented Architecture �and the Future of Computing

A Data-Centric Approach

A presentation to DAMA

By David Lawrence

June 22, 2010

SOA Components

����Page

BPM

Virtualization

Virtual Data Centers

SaaS

Reusability

Service

Stateless

Loose Coupling

HTTP/SOAP

Web Service

Internet

Service�Contract

Actions

On Demand�Computing

Govern

Choreograph

Enqueue

Listen

Event Driven Architecture

Interoperability

Robustness

EAI

Legacy Migration

XML

Abstraction

Transactions

Workflows

Object-Oriented

Open Standards

Open Architecture

Enterprise Service Bus

Distributed�Computing

Metadata

WSDL

Registry

Components

Encapsulation

Discoverability

Provide Service

Consume Service

UDDI

RPC

JAVA

.Net

Open Source

JMS

Authenticate

Open Interfaces

Web 2.0

Messaging Systems

Technologies

Architectures

Patterns

Publish

Integration

Service-Oriented Architecture

Principles

Communicate Asynchronously

Subscribe

Communicate Synchronously

Dequeue

WS-Reliability

Introduction

• Event-Driven Architecture, Service-oriented Architecture and the Data Cloud
• Design principles for the Data Cloud

​

Key Insights:

• New paradigm represents a radical change (for the better) in the nature of software development
• The data professional will be the key person in designing and implementing future data clouds

​

����Page

Key Terms

• Service-oriented architecture
• Event-driven architecture
• Messaging
• Loose coupling
• Statelessness
• Canonical data model
• Transaction/Workflow

​

​

����Page

Presentation Outline

Part I: What is the data cloud?

​

Part II: What are key concepts in the data cloud?

​

Part III: The data cloud: An example

����Page

Part I: What is the data cloud?�

• From Service to Service-Oriented Architecture
• Event-driven Architectures
• The Data Cloud

����Page

What is a service?

• File Transfer One application writes a file that another later reads
• Shared Database Multiple applications share the same database schema, located in a single physical database
• Remote Procedure Invocation  One application exposes some of its functionality so that it can be accessed remotely by other applications as a remote procedure. The communication occurs real-time and synchronously.
• Messaging One applications publishes a message to a common message channel. Other applications can read the message from the channel at a later time. The communication is asynchronous.

​

����Page

What is a Web service?

Different Camps – focused on technical terminology

• It's all about SOAP messages …
• It's all about WSDL
• It's all about XML/HTTP

​

My view – based on an architectural approach

• Web services are application components which interact based on several design principles, the most important of which is that the interactions are loosely coupled

����Page

Aside: Frameworks

Framework: consists of a software framework used by software developersconsists of a software framework used by software developers to implement the standard structure of an application for a specific development environment

Spring Framework: Spring is a leading open source, POJO framework providing a one stop shop for all your enterprise application development needs. The flexibility of configuring the service assembly and its ease of use in test-driven development is, perhaps, one of its biggest selling features

​

• Eases development of any of the four service-types by reducing high complexity of many java components (e.g., messaging bus, shared database)

����Page

Spring Framework

����Page

Industry Trends: Frameworks

����Page

What is a service-oriented architecture

• Service
• Well-defined, Self-contained
• Context independent
• Universally accessible
• Service-Oriented Architecture
• An architectural style
• A document-oriented interaction model
• Interface contracts, registry

​

SOA = Pub/Sub Message bus?

����Page

How does it compare to a Distributed Component Architecture?

• Main driver: transparency to developer
• Remote code looks like local code
• The Distributed Object approach ignores:
• Latency
• Unreliable networks
• Lack of shared memory
• Handling partial failure and uses of coupling

����Page

N-tier vs service-oriented applications

Why is an n-tier architecture considered application distribution and not application integration?

• The communicating parts are tightly coupled—one tier cannot function without the others
• Communication between tiers tends to be synchronous
• An application tends to require rapid system response

In contrast, service-oriented applications are independent applications, but coordinate in a loosely coupled way

• Don’t have to wait for a response
• Can work on other tasks until a result becomes available

����Page

Fundamental Problems of SOA

• Networks are unreliable Sending data across a network is multiple orders of magnitude slower than making a local method call.
• Any two applications are different Integration solutions need to transmit information between systems that use different programming languages, operating platforms, and data formats.
• Change is inevitable Applications change over time. An integration solution needs to minimize the dependencies from one system to another by using loose coupling between applications

����Page

What is an event-driven architecture?

• Distributed processing, no central control
• Nodes respond to incoming events and publish events in response.
• Event the next, usually asynchronously (sender does not wait)
• Channels transport events from one node to another

����Page

Event model

����Page

Event-sourced system: Capture all changes to application state as a sequence of events

����Page

(messages)

What are defining qualities of an EDA?

• Timeliness Publish events as they occur instead of waiting for the next batch cycle
• Asynchrony The publishing system does not wait for the receiving system(s) to process the event.
• Fine Grained Publish single events as opposed to large aggregated event
• Canonical A nomenclature to classify and express interest in certain groups of events
• Complex Event Processing Understanding the relationships between events

����Page

What is a cloud?

Data Driven Cloud:

• Exchange of data using Internet Standards whereby shared resources, software and information are provided to computers and other devices on-demand, like the electricity grid.
• Independent of underlying technologies
• Open standards, reusable business objects
• Exchange of data, not of application control

​

Compare to other types of clouds (not pure data exchanges)

• Amazon
• Google
• Software As A Service (SalesForce.com, Flickr)

����Page

Why will the data cloud change the nature of software development

• Open standards
• No hardware/ software lock in, e.g.,EAI
• Reusable objects defined in business terms
• Quick integration with external providers and consumers
• Ease of legacy migration

​

����Page

Part II: What are key concepts in the data cloud? �

• Key principles
• Deep dive on important principles

����Page

What are key principles for the data cloud?

• Reusability
• Loosely coupled
• Abstraction
• Transaction
• Durability
• Business focus

• Messaging
• Asynchronous
• Canonical Data Model
• Stateless
• Encapsulation
• Discoverability

​

����Page

Deep dive on important principles

• Transaction
• Workflow
• Messaging (Canonical Models)
• Asynchronous behavior
• Loose Coupling
• Abstraction

����Page

What is a transaction?

Transaction: The key concept of the data cloud

​

ACID

• Atomicity states that database modifications must follow an “all or nothing” rule based on transactions defined at a business
• Consistency states that only valid data will be written to the database. If a transaction successfully executes, it will take the database from one state that is consistent with the rules to another state that is also consistent with the rules.
• Isolation requires that multiple transactions occurring at the same time not impact each other’s execution. other.
• Durability ensures that any transaction committed to the database will not be lost. Durability facilitates the restoration of committed transactions in spite of any subsequent software or hardware failures.

​

����Page

How transactions work with queues

����Page

Server

Reply queue

Request queue

Transaction 2:

�Start

Dequeue

process request

Enqueue

Commit

Client

​

Transaction 1:

�Start

get input

construct request

Enqueue

Commit

Transaction 3:

�Start

Dequeue

decode reply

process output

Commit

Stateful

Device

Log

Transaction 1

Transaction 2

Transaction 3

Transaction

(Business level)

Workflow (Conversation)

����Page

​

​

Check Travel Costs

Book Reservation

Find Reservation

Find

Make Tentative Reservation

Compute Cost

Still

Available ?

Cost > Budget

No

Cancel Previous Selection

Cost ≤ Budget

Complete Reservation

Compensating Transaction

Yes

Independent Systems

Possible

Delay

Messaging: The Poster Child of Loose Coupling

• Systems communicate via Channels
• Canonical messages
• Location-independent channel names
• “Fire-and-forget”
• Simplified interaction

​

• Level of indirection

​

• Universal data models
• Location decoupling

​

• Temporal decoupling

​

• Conversational decoupling

​

����Page

Asynchronous messages

����Page

Tightly Coupled Architecture

����Page

Loosely Coupled

Avoids:

​

• Technology Dependency
• Location Dependency
• Data Format Dependency
• Semantic Dependency
• Temporal Dependency
• Interaction Style Dependency

����Page

Loosely Coupled

����Page

Web Services: Tightly coupled to loosely coupled (Stateful messages)

����Page

Advantages of Loose Coupling

����Page

Abstraction (Example 1)

����Page

Abstraction (Example 2)

����Page

Part III: The data cloud: An example �

• From events to a service-oriented data cloud
• Implications for future software development

����Page

Loan Quote with Broker

����Page

Event Cloud

����Page

Event:

Ask for Quote

Event:

Get Credit Score Select Best Rate Return Quote

Event:

Provide Rate Quote Bank 1

Consumer

​

Loan Broker

Credit Bureau

Bank 1

​

Bank 2

​

Bank 3

​

Event:

Return Credit Score

Event:

Provide Rate Quote Bank 2

Event:

Provide Rate Quote Bank 3

Event Cloud with Enterprise Service Bus

����Page

Event JMS/HTML/XSLT, etc:

GetQuote

Event Web Service:

GetCreditScore GetLoanQuote SelectBestRate

Event Web Service:

RateQuote

Consumer

​

Loan Broker

Credit Bureau

Bank 1

​

Bank 2

​

Bank 3

​

Event Web Service:

ReturnCreditScore

ESB

Event Web Service:

RateQuote

Event Web Service:

RateQuote

Workflow (aka Conversation) example

����Page

Compensating Transaction:

Re-verify Credit Score

​

Event JMS:

GetQuote

Event Web Service:

GetCreditScore GetLoanQuote SelectBestRate

Event Web Service:

RateQuote

Consumer

​

Loan Broker

Credit Bureau

Bank 1

​

Bank 2

​

Bank 3

​

Event Web Service:

ReturnCreditScore

ESB

Event Web Service:

RateQuote

Event Web Service:

RateQuote

Time Delay

Canonical Data Model

����Page

Consumer

​

Loan Broker

Credit Bureau

Bank 1

​

Bank 2

​

Bank 3

​

Canonical Data Model

Stateless Message

<QuotedCustomer>

<FirstName>Jim</FirstName>

<LastName>Phillips</LastName>

<Address>151 Main Street</Address>

<City>Cambridge</City>

<State>MA</State>

<ZipCode>02139</ZipCode>

<PhoneNumber>617-555-1212</PhoneNumber>

<SSN>xxx-xx-xxxx</SSN>

<QuotedCustomer>

<FirstName>Jim</FirstName>

<LastName>Phillips</LastName>

<Address>151 Main Street</Address>

<City>Cambridge</City>

<State>MA</State>

<ZipCode>02139</ZipCode>

<PhoneNumber>617-555-1212</PhoneNumber>

<SSN>xxx-xx-xxxx</SSN>

<AppState>???</AppState>

Stateful Message (Type1)

Initial After Loan Broker After Credit Bureau After Bank 1 After Bank 2 etc

Possible States:

<QuotedCustomer>

<FirstName>Jim</FirstName>

<LastName>Phillips</LastName>

<Address>151 Main Street</Address>

<City>Cambridge</City>

<State>MA</State>

<ZipCode>02139</ZipCode>

<PhoneNumber>617-555-1212</PhoneNumber>

<SSN>xxx-xx-xxxx</SSN>

<RateQuote1>0.0525</RateQuote1>

etc.

Stateful Message (Type2)

Asynchronous Messaging

����Page

Consumer

​

Loan Broker

Credit Bureau

Bank 1

​

Bank 2

​

Bank 3

​

GetQuote

GetCreditScore

GetLoanQuote

RateQuote

ComputeRate

ReturnCreditScore

ComputeCreditScore

RateQuote

ComputeRate

RateQuote

ComputeRate

SelectBestRate

RateQuote

Synchronous

GetQuote

GetCreditScore

GetLoanQuote

RateQuote

ReturnCreditScore

ComputeCreditScore

GetLoanQuote

GetLoanQuote

ComputeRate

ComputeRate

ComputeRate

RateQuote

RateQuote

SelectBestRate

RateQuote

Asynchronous

Lessons Learned

• Must keep architectural big picture in mind
• Integration testing more critical –less compile time validation (the price of loose coupling)
• Shared state not always avoidable. Can lead to hidden dependencies
• Make minimum necessary assumptions about sequence of events
• Loosely coupled systems harder to diagnose

����Page

The Future of the Cloud:�Business Operations Platform

����Page

My Company’s Business Processes

D

Development

Team

Data Professional

Questions?

����Page

What is a Cloud

����Page

What is a Service

����Page

What is a Web Service

����Page

What does SOA Mean

����Page

Why SOA Changes the nature of application development

����Page

Synchronous vs Asynchronous

����Page

State: Active

State: Waiting

State: Active

Processing Response

Request for Data

Response

State: Active

State: Active with

Other Processes

State: Active

Processing Response

Request for Data

Response

����Page

����Page

����Page

����Page

����Page

����Page

1-1

����Page

1-2

����Page

1-3

����Page

1-4

����Page

2

����Page

2

����Page

2

����Page

2

����Page

2

����Page

2

����Page

3-2

����Page

3-1

����Page

3-4

����Page

3-4

����Page

3-5

����Page

3-8

����Page

3-14

����Page

����Page

����Page

����Page

Messaging: The Poster Child of Loose Coupling

• Systems communicate via Channels
• Location-independent channel names
• “Fire-and-forget”
• Simplified interaction

​

• Level of indirection

​

• Location decoupling

​

• Temporal decoupling
• Conversational decoupling

​

����Page