- the way forward

STAFFSOFT

09/29/2023

Michelle Sollicito

Solution Architect, Data Platforms

• 30 years in software
• Managed data migrations on many cloud projects
• Many Microsoft certifications
• Many Cloud certifications
• Our whole team is similarly qualified

Together we are awesome

https://www.credly.com/users/michelle-sollicito and https://google.accredible.com/c7dc8ed2-ac44-4a5d-9e3b-6e1e78f28b4b

Evergreen

Staffing Solutions, Managed Services

• We build and manage teams
• Part of Insight Global, founded in 2001
• 70+ offices across the US, Canada and the UK
• Strong teams, strong expertise, strong values
• Empowering people and Supercharging Diversity is who we are

https://insightglobal.com/about/

https://evergreen.insightglobal.com/

Everyone Matters

Understanding of the Problem

​

Current State

• 80+ versions of product
• Different schemas
• Different application architectures*

​

• Decentralized legacy data*
• Some in SQL databases
• Some in NOSQL databases*
• Some in data lakes/blob storage*

​

• Data on-premise, other clouds*

​

• Single tenant siloed architectures*

*assumptions

Future State

• 1 SAAS (Software as a Service) product
• One schema
• One application architecture

​

• Centralized data
• Internal view
• External view for clients
• One view of data

​

• All systems and data on one cloud – Azure

​

• Multi-tenant architecture

High-level Business Requirements

• Solution must be hosted in Microsoft Azure
• Standardization and consolidation of all legacy data for reporting and analytics
• Data from legacy products should be streamed until all clients have migrated to a new SAAS based product
• Real time analytics for fraud detection
• Back-end and front-end users should be able to query any import attribute and any data record
• Back-end users should be able to see Source data alongside Transformed data

Assumptions

• Large data volumes
• Large data variety (SQL and NOSQL, json, csv files, parquet files)
• Future state data does not need to be siloed on dedicated hardware
• Current data is on-premise and on other clouds (AWS, Google Cloud)
• Application detailed requirements are out of scope
• All clients will move to the latest version of the product
• Current version of the product is well specified
• Current version of the product data model is the future state data model
• Multi-national /region solution is needed
• Compliance requirements are SOC2 and GDPR.

​

High-level Technical Solution

​

Transitional Data Architecture

Ingestion from multiple data sources and types:

• SQL
• NOSQL
• Json
• Parquet
• CSV
• XML

Data Pipeline:

• Remove bad data
• Clean up data
• Move data from old schema to new
• “ACID” data

Data Warehouse:

• Metrics
• Dimensions
• Relationships
• Sums/Averages

​

Transitional Data Architecture

Azure Tool Options

• Tenant segregation and integration – Azure tenant services
• Migration – Azure Database Migration Service, Azure Data Studio or Azure Data Factory
• Data governance – Purview
• Data streaming – Azure Stream Analytics, SQL replication
• Data ingestion, clean up and integration - Azure Data Factory
• Data warehousing - SQL pools (DW/Synapse Analytics)
• Data storage (unstructured) – ADLS2 (Data lake) or Blob / File Storage
• Business intelligence - PowerBi
• ACID data, tools for data manipulation - Databricks
• Data engineering - Synapse
• Data science - Synapse
• Real time analytics - Synapse
• No code insights and actions - Data activator
• Data lakehouse – Databricks plus Delta Lake

Azure New Tool Options (Future State)

• Unified view of data - OneLake
• Unified environment for all data work - Fabric

Transitional Data Architecture

1 Customer stream may use Stream Analytics and may require a buffer/throttling layer – it is possible to use a pub sub model (Event Hub) or ServiceBus

2 Customer batch files represents any data for ingestion - csv files/json files, data from AWS S3, a host of data sources to Data Factory

3 On-Premise data sources include SQL Server data sources and almost any data source to Data Factory. Azure Data Studio may also be used

Future State High-Level Data Architecture

• Microsoft Fabric
• On top of OneLake and OneSecurity
• Unified view of the data internally, segregated views externally
• Built on top of Data Lakehouse architectures
• Customized User experiences
• SAAS Foundation
• Purview built-in

Future State Application Architecture

Meeting High-level Requirements with High-level Architecture

• All components are Azure-based
• Purview provides data governance, fraud detection
• The Data Pipeline (Azure Data Factory), Data Lakes (Raw, Processed, Analytics) and DataBricks allow the legacy data to be standardized and consolidated
• Customer analytics via a Customer 360 solution called Customer Insights solution
• PowerBi and Azure Data Factory fulfill these requirements:
• Reporting and analytics
• Back-end and Front-end Users should be able to query any import attribute and any data record
• Back-end Users should be able to see Source data alongside Transformed Data
• Back-end Users will be able to view and verify data (optionally in Staging environment before data is imported into Production environment)
• Users need a UI to perform verification and mark records for import

​

​

• Note that data viewed must be controlled by region* (cannot be accessed outside of the region the data belongs to), Tenant Specific (tenants can only see their own data; Staffsoft personnel can see all tenants’ data) and then based upon tenant/user role.

​

​

*assumptions

Team Composition

​

Teams

• We, Evergreen, have learned from our experience how to structure teams for best results
• Center of Excellence / Control Tower
• StaffSoft
• Business Sponsor
• Key Client representative(s)
• Product Managers/SMEs
• Enterprise Architect*
• Security Architect*
• Evergreen
• Relationship Manager (part time)
• Program Manager
• Senior Cloud Architect
• Security Architect (part time)
• Change Management Architect (part time)
• Technology and Architecture Office
• StaffSoft
• Enterprise Architect*
• Security Architect*
• Application Management/Ops team (StaffSoft)
• PODs (SAFe Scrum Teams)

​

​

*probably the same people fulfill these roles

WE BUILD & MANAGE TEAMS�THAT HELP YOU GROW.

Teams/PODs

Our proposed makeup of each POD is:

​

​

​

​

​

Evergreen team members:

• 1 Scrum Master
• 1 or 2 Data Developers/Engineers
• 1 Quality Engineer
• 1 DevOps Engineer

​

StaffSoft team members:

• 1 Product Owner
• 1 or 2 Developers/Engineers
• 1 Quality Engineer
• Optional UX/UI Designer
• Optional DevOps Engineer

​

​

POD stands for "Product Oriented Delivery". In the Scaled agile framework (SAFe), a POD is a small, cross-functional team that works together to deliver a product. 

 The team members have the skills to design, develop, test, and operate the product.  They work in multiple sprints using the Scrum methodology.

​

WE BUILD & MANAGE TEAMS�THAT HELP YOU GROW.

Delivery Strategy

​

Our Methodology - Requirements

• Based upon SAFe Agile, so scales up well
• In-depth Requirements Analysis phase up front to determine detailed roadmap and prioritize for planning
• After Requirements phase, each Phase/theme will be broken down into Epics, Features (optional) and then to Stories/Tasks
• One roadmap Phase may consist of one or many Epics.
• An Epic is usually completed over a number of sprints.
• Usually, one Feature is completed in a sprint.
• Features, stories and tasks may be continually re-prioritized each sprint as the project progresses.
• Azure Boards will be used to house these, as well as the backlog

​

Our Methodology - Sprints

• Before each sprint the work will be prioritized (backlog refinement)
• The Pod (team) will work on those stories assigned to the sprint
• A scrum board will be visible each day during a stand-up meeting so that the Pod knows which stories to work on and the status of stories within the sprint.
• Continuous improvement is built into our methodology with sprint reviews and retrospectives
• A sprint’s work must be either approved by the client (StaffSoft’s Product owner, Product manager or Business Sponsor) or changes needed to make the work approvable must be added to the backlog for future sprint(s).
• If the sprint’s work is approved, it will be released to production environment.

​

Our Development Methodology and CI/CD

• Development work will be carried out in the “Dev” environment
• Code will be checked in to feature branches regularly (minimum daily) within Azure Repos
• A pull request will be made against a feature branch
• A code review will be carried out before code changes can be merged into the main branch.
• Before all code is released to “Staging” environment*:
• The code changes must be approved
• Code must pass all unit tests and quality/lint tests
• Build is deployed as an artifact or container
• The above is automated using Azure Pipelines
• Infrastructure and Data schema version control:
• Terraform code for infrastructure build will be subject to the same source code control, development, pull request/code review cycle as application code to ensure that infrastructure and code is kept at corresponding versions.
• Data model/schema changes will also be subject to version control in a similar way.

*Note: In this case, Transformed data may be kept in Staging environment until approved before being imported into Production environment.

Our Development Methodology and CI/CD

• Before all code is released to “Production” environment, these must pass in Staging:
• Unit tests
• Integration tests, contract/pact tests
• System tests, security tests and user acceptance tests/manual tests
• (and Blue/Green deployment /Smoke testing in “Production” environment)
• Monitoring and Operations
• Automated monitoring of all Azure services and application components
• Where possible automated self-healing is put in place:
• Healthchecks can be used to determine problems in microservices and in infrastructure
• Heartbeats can be used to detect software or infrastructure problems
• Self-healing scripts / azure functions can rectify issues where possible
• Automated alerts inform the Service / Support staff of issues in an appropriate manner
• Track SLA data / KPIs especially

​

Delivery Strategy

1. In-depth Requirements Analysis Phase

​

First phase – Requirements Analysis

• If you decide to use Evergreen for this work…
• What will the Requirements Analysis Phase look like?
• What Deliverables might look like?

​

​

​

​

Deep Dive on Non-functional Requirements

• Security/Privacy/Compliance issues
• Zero trust should be the baseline*
• Authentication and authorization
• Groups/Users/Roles
• How will we separate the data of different clients?
• How will we ensure all internal users can access all data for their use case (but conform to least privilege)
• How will we ensure that no bad actors/unauthorized users access data (IPS/IDS)
• How will we ensure GDPR compliance?*
• Encryption of data at rest and in transit
• SLA-related requirements
• Availability
• Business continuity/disaster recovery
• Scalability
• Performance/speed/response times
• IPS/IDS (Intrusion Prevention and Detection)
• Monitoring and Operations / ITIL requirements (example. ServiceNow integration/JIRA integration)
• Internationalization and Accessibility?
• Languages/translations supported now or in future?
• Currencies, dates, left to right text/right to left text?
• Browser support/mobile support?
• Aria / accessibility (for the blind and deaf)

​

​

​

​

​

​

​

​

​

​

​

*assumptions

Deep Dive on Multi-tenant requirements

• Change Management
• How will clients transition from legacy product to new product?
• Onboarding and Provisioning
• Technical support / training for clients
• Training and support for Staffsoft employees
• Billing /Contract - How will clients be charged for use of the new system?
• By number of users?
• By number of queries on data?
• By data volume?
• SLAs? Refunds for poor availability?
• Metrics/KPIs
• How will we track usage (and report on that usage) so that we can charge clients accurately?
• How will we monitor system availability for SLA purposes
• Data Isolation
• Separate infrastructure?
• Separate data lakes?
• Separate workspaces within One Lake?

​

​

​

​

​

Multi-tenant requirements

• All tenants will use the same software application and infrastructure *

​

​

​

​

​

​

​

Onboarding of clients to new SAAS product

• Subscription model (license/subscription per company, per BU, who pays and how
• Offboarding from old product

Provisioning of new users/clients in new SAAS product

Deep Dive on Requirements - Application Architecture

• Data Migration detailed requirements
• Different customers will have different data / schemas
• Need to design the flow of data from current state to future state
• Data Pipeline with numerous data lakes?
• Application Architecture and Application detailed requirements
• Deeper dive into Application (or Application Architecture) Requirements themselves
• Consolidation of all 80+ versions of the product:
• Different source code bases
• Future state codebase?
• Different coding languages
• Future state coding language?
• Different data schemas?
• Future state data schema?
• Consolidation of all 80+ versions of the product:
• Microservices /REST architecture design (Web apps and Web jobs)
• Scalable (in and out, up and down)
• Flexible for future changes

​

​

​

Delivery Timelines

​

Delivery Timelines

• Non-functional requirements document

​

• Multi-tenant requirements document

​

• Detailed Application / architecture requirements document

​

• Detailed Future State Data Model diagram

​

• Detailed Application Architecture diagram

​

• Roadmap diagram

​

• Epics and Features

​

• Stories and sprints for Proof-of-Concept Phase

​

• Create/ prioritization of Stories and Tasks

​

• Build out the architecture/ infrastructure

​

• Migration of one client’s data

​

• Transform their data

​

• Verification, queries and reports

​

• Review project, learn lessons, tune our methodology

​

• Analytics, KPIs, test plans, test results

​

​

• Group of clients in one region

​

• Multiple pods (3-4)

​

• Adjust architecture/infrastructure

​

• Migrate each client’s data

​

• Transform their data

​

• Verification, queries and reports

​

• Review and learn, tune methodology

​

• Analytics, KPIs, test plans, test results

​

​

​

• Multiple clients, multiple regions

​

• Multiple parallel pods working together (3-8)

​

• Adjust architecture/infrastructure

​

• Migrate each client’s data

​

• Transform their data

​

• Verification, queries and reports

​

• Review and learn, tune methodology

​

• Analytics, KPIs, test plans, test results

​

​

January 2024

October

2023

April - May 2024

July - Sept

2024

December 2024

Deliverables

Next Steps

• Contact me on 678 357 3661
• Cost estimates can be provided
• Questions?

​

INTRODUCING INSIGHT GLOBAL’S MANAGED SERVICES CAPABILITIES

Extra slides included here may be used to help explain/answer questions

Evergreen Consultancy Costs

Monthly Azure Costs

- Pricing Calculator used

- Not taking into account special deals with Microsoft etc.

- Gives a ballpark, not exact figures

- Based upon average data loads anticipated during the Transitional period

- Based upon very high-level requirements

- Some services will reduce and some data storage will increase once in future state

- Approximately $10,000 per month

Azure Estimates for Evergreen/StaffSoft

Useful resources

Microsoft Fabric

https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview

Databricks with Fabric and Onelake

https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake

​

Azure Fabric

Azure DataBricks run data pipeline

Azure Databricks for prep and train

Why DataBricks plus Fabric?

• Fabric - Separation of compute and storage, underpinned by a Data Lake
• Many existing Data Lakes are created using Databricks
• Physically stored in ADLS2 but prepped and cleaned using Databricks
• Data Lakehouse supported by a centralized unified catalog, governance and security
• Multiple ways of building and consuming data products: Spark, SQL, ML, PowerBI, …
• Widespread adoption of DataBricks
• Power BI in Microsoft Fabric now works in direct lake mode, allowing data to be queried with blazing fast performance directly over data in OneLake
• Unified set of tools for deploying, sharing, and maintaining enterprise-grade data and AI solutions at scale
• OneLake – one vision of data, shortcuts to referred to data (works well with Databricks)
• https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake

Azure Data Factory