Velocity @ Scale | Platform Engineering
Platform Engineering Meetup #1
17:00 - 17:30 Gathering
17:30 - 18:20 Building a Platform for Real Engineers / Yuval Perry
18:20 - 19:10 A New Microservice in 1 hour / Savva Khalaman & Oded Apel
19:00 - 19:40 Open discussion
Velocity @ Scale | Platform Engineering
Yuval Perry
yuvalp@wix.com
Velocity @ Scale | Platform Engineering
Platform Engineering
The discipline of designing and building toolchains and workflows that enable fast self-service capabilities for software engineering organizations
Velocity @ Scale | Platform Engineering
The Maslow pyramid of engineering needs
Business �logic
Frameworks
Containers
CI/CD
Production
Velocity @ Scale | Platform Engineering
To become the best Web creation platform �for any type of user and any type of business, anywhere in the world.
Wix’s north star goal
~1,500 Web Developers
@aviranm
EU | Ukraine | Israel | America |
Vilnius | Kyiv | Tel-Aviv | USA |
Krakow | Dnipro | Be’er Sheva | Canada |
Berlin | Lviv | Haifa | |
Amsterdam | | | |
Velocity @ Scale | Platform Engineering
Deployed Services
~3K
Lambda Functions
1K
Builds a day
~50K
Merges to master a day
~7K
Deployments�Per Minute (work hours)
>1
Deployment Per Day
>600
Total registered users
~238M
Premium Subscriptions
>6m
Wix in numbers (as of January 2023)
Velocity @ Scale | Platform Engineering
Iterate as we scale
We can always improve
Enabling the business needs
Focus on DevEx
Velocity @ Scale | Platform Engineering
An example
Velocity @ Scale | Platform Engineering
No staging environment
No integration environment
Wix’s Development Strategy, in a Nutshell
At the end of it, it’s your laptop and production.
TDD for high code coverage
A/B testing and feature toggles
Velocity @ Scale | Platform Engineering
No staging environment
No integration environment
Wix’s Development Strategy, in a Nutshell
At the end of it, it’s your laptop and production.
Gradual rollout (Automated canary)
Velocity @ Scale | Platform Engineering
Our next leap
Open Platform
Velocity @ Scale | Platform Engineering
Quality
Uniformity
Extensibility
Open Platform
Velocity @ Scale | Platform Engineering
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
RPC
Secrets
Data Access
Domain Events
Error Handling
A new microservice
needs ...
GDPR
PII
Plat. Query
Advance Search
Caching
Webhooks
Scheduled Tasks
BI
Logging
Testing Strategy & Tools
Scaffold
Build
Deploy
A/B Testing Support
Gradual Rollout
Production Ownership
Monitoring
Debugging
Framework
Framework
Identity
Framework
Visibility
Web Hooks
GDPR
Identity
Framework
Input validation
Visibility
Web Hooks
GDPR
Identity
Testing
API Design
Brokers & Events
Error Handling
Database
Framework
Events
Visibility
Web Hooks
GDPR
Identity
Testing
API Design
Brokers & Events
Error Handling
Database
Data integrity
gRPC
Webhooks
Death by a thousand papercuts
Velocity @ Scale | Platform Engineering
The Un-platform Red Flags
→
→
Lots of docs
Un-discoverability
Unmanaged concerns
Duplicate solutions
→
→
→
→
→
→
Lots of communication
Long waits (DB, routing, etc)
No emergency rollout
Low productivity
Velocity @ Scale | Platform Engineering
We need a platform
Before
Velocity @ Scale | Platform Engineering
Exactly
After
Way After
Day 1
Day 1
Velocity @ Scale | Platform Engineering
״A product company’s success has a direct correlation with the speed of development - �If we don’t move faster than our competition, we’ll lose.”
Velocity @ Scale | Platform Engineering
NILE
Streamlines platformized �microservice development
Introducing:
Nile Layers and tools
Tools
Libraries
Guidelines
Documentation
Production
Consulting & Support
Velocity @ Scale | Platform Engineering
CI processes
Samples
Internal Development Platform�enables users to focus on delivering real business value
Velocity @ Scale | Platform Engineering
Factor out the common areas from multiple well designed applications.
Build a platform prior to any application.
Once complete, build applications on top of it.
Platform building strategies
Velocity @ Scale | Platform Engineering
Infrastructure Engineer
Requirements from an
+
Product
Velocity @ Scale | Platform Engineering
Platform is a Product
Engineers are clients
Velocity @ Scale | Platform Engineering
Our Principles
Velocity @ Scale | Platform Engineering
Pave the road
Measure to improve
Our Principles
→
→
→
Velocity @ Scale | Platform Engineering
Avoiding Shelfware
Boots on the ground
→
→
→
→
Velocity @ Scale | Platform Engineering
Boots on the ground
The art of escaping ivory towers
Velocity @ Scale | Platform Engineering
Boots on the ground
→
→
→
→
→
→
→
→
The art of escaping ivory towers
Velocity @ Scale | Platform Engineering
Find the road that most engineers take - �and make it nice and cozy.
Minimal cognitive load
Just enough documentation
Zero boilerplate
Prefer defaults and conventions
Pave the road
Velocity @ Scale | Platform Engineering
Yet you can't improve what you don't measure.
→
→
Merge requests per day?
→
MTTRs?
Lines of code?
→
Deployment Frequency?
→
Onboarding time?
No KPI is Perfect
Velocity @ Scale | Platform Engineering
DON’T OVER COMPLICATE.
Find the most trivial KPI for your situation and evolve in it.
Velocity @ Scale | Platform Engineering
Pave the road
Measure to improve
Main Principles
→
→
→
Velocity @ Scale | Platform Engineering
The speed paradox
Adding one person to a team of two �adds not only 50% more workforce, �but also 50% more communication
Platform Engineering Meetup #1
17:00 - 17:30 Gathering
17:30 - 18:20 Building a Platform for Real Engineers / Yuval Perry
18:20 - 19:10 A New Microservice in 1 hour / Savva Khalaman & Oded Apel
19:00 - 19:40 Open discussion
Savva Khalaman
Oded Apel
odeda@wix.com
savvak@wix.com
Velocity @ Scale | Platform Engineering
A New Microservice in One Hour
Velocity @ Scale | Platform Engineering
Velocity @ Scale | Platform Engineering
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
Secrets
RPC
Data Access
Domain Events
This takes more than an hour…
How can we help?
Advance Search
Error Handling
GDPR
PII
Plat. Query
A/B Testing Support
Testing Strategy & Tools
Caching
Webhooks
Scheduled Tasks
Scaffold
Build
Deploy
Gradual Rollout
Production Ownership
Logging
Monitoring
BI
Debugging
Velocity @ Scale | Platform Engineering
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
Secrets
RPC
Data Access
Domain Events
Advance Search
Error Handling
GDPR
PII
Plat. Query
A/B Testing Support
Testing Strategy & Tools
Caching
Webhooks
Scheduled Tasks
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
Secrets
RPC
Data Access
Domain Events
Advance Search
Error Handling
GDPR
PII
Plat. Query
A/B Testing Support
Testing Strategy & Tools
Caching
Webhooks
Scheduled Tasks
Scaffold
Build
Deploy
Gradual Rollout
Production Ownership
Logging
Monitoring
BI
Debugging
Today we will focus on these:
Velocity @ Scale | Platform Engineering
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
Secrets
RPC
Data Access
Domain Events
Advance Search
Error Handling
GDPR
PII
Plat. Query
A/B Testing Support
Testing Strategy & Tools
Caching
Webhooks
Scheduled Tasks
Domain modeling�API Design
Authentication
Authorization
Input Validation
Object Mapping
Secrets
RPC
Data Access
Domain Events
Advance Search
Error Handling
GDPR
PII
Plat. Query
A/B Testing Support
Testing Strategy & Tools
Caching
Webhooks
Scheduled Tasks
Scaffold
Build
Deploy
Gradual Rollout
Production Ownership
Logging
Monitoring
BI
Debugging
But in reality we already have good solutions for all of these :)
Some basic background about protobuf
Velocity @ Scale | Platform Engineering
Protobuf is ...
rpc GetProduct (GetProductRequest) returns (GetProductResponse) {
option (google.api.http) = { get: "/v1/products/{id}" };
option (wix.api.permission).name = "WIX_STORES.READ_PRODUCTS";
}
An IDL (interface definition language) used to describe APIs
message Product {
string id = 1 [(wix.api.readOnly) = true];
google.protobuf.StringValue name = 2 [(wix.api.minLength) = 1, (wix.api.maxLength) = 80];
}
Velocity @ Scale | Platform Engineering
message Product {
string id = 1 [(wix.api.readOnly) = true];
google.protobuf.StringValue name = 2 [(wix.api.minLength) = 1, (wix.api.maxLength) = 80];
}
Additional Wix semantics, e.g., validations and permissions
my.proto file
Velocity @ Scale | Platform Engineering
This was the experience of developing a new service in Wix two years ago
Velocity @ Scale | Platform Engineering
Start a new service
New Service
Application().start(args)
Write just one line ❤ of code to get a working Web server in production.
Developer Needs To
...
Velocity @ Scale | Platform Engineering
Application(myApiImplementation).start(args)
(defined in the Proto file)
New Service
+ APIs
Expose APIs
Implement myAPI �(generated from proto)
Developer Needs To
...
Velocity @ Scale | Platform Engineering
New Service
+ RPC
+ APIs
Build RPC Clients
val currencyConverter = CurrencyConverter.stub(grpcChannel,
_.withAuthority("com.wixpress.platform.currency-converter")
)
Application(myApiImplementation).start(args)
Know which library to use (out of 3), depending on which API we’re connecting to
Developer Needs To
...
Velocity @ Scale | Platform Engineering
+ Messaging
New Service
+ RPC
+ APIs
Work with our Messaging Queue
Create it�
Create producers \ consumers�
Call produce when you want to send a message�
Subscribe to some topic and wire up a handler
Developer Needs To
...
val client = AdminClient.create()
val result1: Future[Unit] = client.createTopic("my-topic-name")
val producer: GreyhoundProducer = GreyhoundBuilder.producer()
val greyhound = GreyhoundBuilder().withProducers(producer).build
producer.produce("some-topic", SomeMessage("message"))
Velocity @ Scale | Platform Engineering
laboratory.conductExperiment(classOf[SomeFeatureToggle], false, new BooleanConverter)
+ A/B Tests
+ Messaging
New Service
+ RPC
+ APIs
Conduct Experiments
Velocity @ Scale | Platform Engineering
val db = MySql.newHikariDataSource(dbName = "my-db-name").build()
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
Create and Manage DB Connections
Velocity @ Scale | Platform Engineering
Many more things to take care of
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
Velocity @ Scale | Platform Engineering
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
If each library is well written, simple, and easy to use,
What is the Problem?
Velocity @ Scale | Platform Engineering
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
What it really means to wire RPC
Developer Needs To
...
Velocity @ Scale | Platform Engineering
for every RPC
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
Developer Needs To
...
Velocity @ Scale | Platform Engineering
+ A/B Tests
+ Messaging
+ DB
New Service
+ RPC
+ APIs
for all other examples
Velocity @ Scale | Platform Engineering
Here’s what it looks like
object SomeService {
val rpc = <wiring code>
val laboratory = <wiring code>
val kafkaProducer = <wiring code>
val kafkaConsumer = <wiring code>
val service = SomeService(rpc, laboratory, kafkaProducer, kafkaConsumer)
}
Starting up the service
class SomeService� (rpc, laboratory, kafkaProducer, kafkaConsumer) {
<Business logic>
}
Actual service code
Velocity @ Scale | Platform Engineering
class Test {
<Test some business logic>
def buildTestEnv = {
val rpc = <create mock>
val laboratory = <create mock>
val kafkaProducer = <create mock>
val kafkaConsumer = <create mock>
val service = SomeService(rpc, laboratory, kafkaProducer, kafkaConsumer)
}
}
class E2ETest {
<Test a flow that checks if the wiring code works>
def buildTestEnv = {
val rpc = <start mock service>
val laboratory = <start mock laboratory>
val kafka = <start mock Kafka>
val service = <start actual service>
}
}
Here’s what it looks like
Tests
Velocity @ Scale | Platform Engineering
Define the Problem
We want developers to �focus on business value.
Instead, time is wasted on wiring things and writing the same things over and over again
Velocity @ Scale | Platform Engineering
How Platform Engineers Can Improve DevEx
Talk to users
Velocity @ Scale | Platform Engineering
Don’t test the infra
Recommended way of testing things
Single entity per service
Make infra opinionated
How Platform Engineers Can Improve DevEx
Talk to users
Velocity @ Scale | Platform Engineering
Now that we understand it we decide to
How Platform Engineers Can Improve DevEx
Velocity @ Scale | Platform Engineering
Show Me Some Code Please!
Velocity @ Scale | Platform Engineering
Every service would have the following, which tells the framework 2 interesting things
Single entity per service
prime_app(
name = "some-service",
artifact = "com.wixpress.some-test-service",
rpcs = [
"com.wixpress.platform.currency-converter",
],
service = "com.wixpress.some.service.api",
)
Velocity @ Scale | Platform Engineering
prime_app(
name = "some-service",
artifact = "com.wixpress.some-test-service",
rpcs = [
"com.wixpress.platform.currency-converter",
],
service = "com.wixpress.some.service.api",
)
Every service would have the following, which tells the framework 2 interesting things
Velocity @ Scale | Platform Engineering
Single entity per service
Now, look what we can do :)
We can generate SomeServiceAPI
Once created, it gets automatically wired
class SomeServiceAPI(currencyConverter: CurrencyConverter) extends SomeServiceAPI { … }
New Service
Velocity @ Scale | Platform Engineering
+ RPC
object SomeServiceAPI {
def apply(contextBuilder: SomeServiceAPIContextBuilder) = {
val context = contextBuilder.build()
val service = new SomeServiceAPI(context.rpc.currencyConverter)
}
When you start up the service, you get a ready-to-use RPC client
New Service
Velocity @ Scale | Platform Engineering
+ Tests
+ RPC
New Service
For the tests, we generate a contextBuilder which you can pass to your service
val contextBuilder = SomeServiceAPITestContextBuilder()
val service = contextBuilder.blockingPlatformized(SomeServiceAPI(contextBuilder))
You can then easily mock the RPC results and test your flow end to end
context.rpc.currencyConverter(“EUR”, “USD”) returns 3.55
service.callSomeAPI
Don’t test the infra
Recommended way of testing things
Velocity @ Scale | Platform Engineering
class Test {
<Test some business logic>
def buildTestEnv = {
val rpc = <create mock>
val laboratory = <create mock>
val kafkaProducer = <create mock>
val kafkaConsumer = <create mock>
val service = SomeService(rpc, laboratory, kafkaProducer, kafkaConsumer)
}
}
class E2ETest {
<Test a flow that checks if the wiring code works>
def buildTestEnv = {
val rpc = <start mock service>
val laboratory = <start mock laboratory>
val kafka = <start mock Kafka>
val service = <start actual service>
}
}
Reminder - this is what it used to look like
Velocity @ Scale | Platform Engineering
+ Tests
+ RPC
New Service
You write your business logic
class SomeServiceAPI(currencyConverter: CurrencyConverter) extends SomeServiceAPI { {
override def SomeAPI(request: SomeAPIRequest): Future[SomeAPIResponse] = {
// Business logic goes here
}
}
+ Tests
+ RPC
New Service
Velocity @ Scale | Platform Engineering
Request data is enriched (with more data e.g., caller identity and site context) & Input validation & permission checks based on the proto declaration
+ request data
+ Input validation
+ Permission checks
You get:
+ Tests
+ RPC
New Service
You write your business logic and before your code is even reached
class SomeServiceAPI(currencyConverter: CurrencyConverter) extends SomeServiceAPI { {
override def SomeAPI(request: SomeAPIRequest): Future[SomeAPIResponse] = {
// Business logic goes here
}
}
Velocity @ Scale | Platform Engineering
+ Producing
Producing is easy
context.domainEvents.sendCreatedEvent(<id>, <entity>)
Since we decided that every service is about a single main entity, producing is just as simple as
contextBuilder.domainEventsCollector.allSentEvents <assertion>
Testing it is also easy
+ request data
+ Input validation
+ Permission checks
+ Tests
+ RPC
New Service
Velocity @ Scale | Platform Engineering
+ Consuming
+ Producing
+ request data
+ Input validation
+ Permission checks
+ Tests
+ RPC
New Service
rpc ConsumePaymentEvents (.wix.common.domainevents.DomainEvent) returns (google.protobuf.Empty) {
option (.wix.api.subscription) = {
entity: "com.wix.ecom.orders.payments.v1.OrderTransactions"
};
}
In your proto file, you declare that you want to consume messages from another entity.
No need to know which Kafka topic it is
Consuming
Velocity @ Scale | Platform Engineering
+ Consuming
+ Producing
+ request data
+ Input validation
+ Permission checks
+ Tests
+ RPC
New Service
As a result, you get this stub in your service to implement. It looks exactly like a regular endpoint in your service, so testing it is straightforward.
override def consumePaymentEvents (event: DomainEvent[OrderTransactions])
Consuming
Velocity @ Scale | Platform Engineering
+ Experiments
+ Consuming
+ Producing
+ request data
+ Input validation
+ Permission checks
+ Tests
+ RPC
New Service
context.petri.conductSpecsSomeSpec.contains(true)
And then you can just do:
prime_app(
…
petri_specs = [
"specs.someSpec",
],
)
Experiments
Every service would have the following, which tells the framework 2 interesting things:
How did we do it?
+ Experiments
+ Consuming
+ Producing
+ request data
+ Input validation
+ Permission checks
+ Tests
+ RPC
New Service
Velocity @ Scale | Platform Engineering
+ Input validation
+ Tests
+ Permission checks
+ Experiments
+ Consuming
+ Producing
+ request data
+ RPC
New Service
How did we do it?
Velocity @ Scale | Platform Engineering
Test env
and then...
+ Input validation
+ Tests
+ Permission checks
+ Experiments
+ Consuming
+ Producing
+ request data
+ RPC
New Service
How did we do it?
Velocity @ Scale | Platform Engineering
+ Input validation
+ Permission checks
+ Tests
+ Experiments
+ Consuming
+ Producing
+ request data
+ RPC
New Service
Platform
Velocity @ Scale | Platform Engineering
The Result
Happy developers :)
Focused on providing value, not writing boilerplate code
New Service
Velocity @ Scale | Platform Engineering
Savva Khalaman
Oded Apel
odeda@wix.com
savvak@wix.com
Velocity @ Scale | Platform Engineering
Data Layer
Can it be simple?
... at least for users
Velocity @ Scale | Platform Engineering
Micro Service Layers
Controller
Business Logic
Data Layer
Velocity @ Scale | Platform Engineering
Micro Service Layers
Controller
Business Logic
Data Layer
Velocity @ Scale | Platform Engineering
“…is a pattern that provides an abstract interface to some type of database or other persistence mechanism”
Micro Service Layers
Controller
Business Logic
Data Access Object
Velocity @ Scale | Platform Engineering
BUT WHY?
Micro Service Layers
Controller
Business Logic
Data Access Object
“…is a pattern that provides an abstract interface to some type of database or other persistence mechanism”
Take 1
“…to ease moving to another database!”
A quote from numerous hot discussions
Data Access Object
Well… NO
Take 2
Data Access Object
OK...
Take 2
Data Access Object
Wait.. is it completely true?
Common Concerns
Data Access Object
Velocity @ Scale | Platform Engineering
Common Concerns
Data Access Object
Velocity @ Scale | Platform Engineering
Technical
Common Concerns
Data Access Object
Velocity @ Scale | Platform Engineering
Business
Common Concerns
Data Access Object
Velocity @ Scale | Platform Engineering
Simple
Data Layer
(SDL)
Velocity @ Scale | Platform Engineering
Core Ideas
Velocity @ Scale | Platform Engineering
So… What have we got?
Velocity @ Scale | Platform Engineering
prime_app(
...
sdl = {
"my_cluster": {
"my_db": {
"persons": {
"entity": "com.example.Person",
"optimistic_locking": True,
"trash_bin": True,
},
},
},
}
Starting with the App macro, which tells the framework 2 interesting things
Velocity @ Scale | Platform Engineering
case class Person(@id id: Option[UUID], name: String, revision: Option[Long], updatedDate: Timestamp)
So… what have we got?
Definition of the persisted class
case class Person(@id id: Option[UUID], name: String, revision: Option[Long], updatedDate: Timestamp)
So… what have we got?
Definition of the persisted class
System Fields
val person: Person = …�context.sdl.update(person)
And then you just:
case class Person(@id id: Option[UUID], name: String, revision: Option[Long], updatedDate: Timestamp)
So… what have we got?
Definition of the persisted class
val person: Person = …�context.sdl.update(person)
And then you just:
case class Person(@id id: Option[UUID], name: String, revision: Option[Long], updatedDate: Timestamp)
Example: Updating an entity
Definition of the persisted class
Velocity @ Scale | Platform Engineering
How does it work?
Persistence
Common Table Layout
Tenant �(implicit, not a part of the method signatures)
tenant_id | entity_id | entity |
215a3b09-0858-.... | 136baec8-560e-... | { "name": "John", "revision": 31, … } |
Data as JSON
Velocity @ Scale | Platform Engineering
DDL deduction. Enables:
@indices(Set(
Index(Property("name")),
Index(Property("age", IndexDirection.Ascending)),
))
case class Person(
@id id: Option[UUID],
name: String,
age: Int,
revision: Option[Long] = None
)
What if I need an index?
Velocity @ Scale | Platform Engineering
PII/GDPR Support
@indices(Set(
Index(Property("name")),
Index(Property("age", IndexDirection.Ascending)),
))
case class Person(
@id id: Option[UUID],
@pii name: String,
age: Int,
revision: Option[Long] = None
)
Velocity @ Scale | Platform Engineering
And many more
features that we are happy to brag about…
Velocity @ Scale | Platform Engineering
Functionality CAN NOT come at the expense
of Dev Experience…
Velocity @ Scale | Platform Engineering
Build-time feedback whenever possible
Clear API, no ambiguity
Example:
Dev Experience
Velocity @ Scale | Platform Engineering
Example:
Dev Experience
With Optimistic Locking
Velocity @ Scale | Platform Engineering
Without Optimistic Locking
Query DSL
Velocity @ Scale | Platform Engineering
val query = QueryV2()
.withFilter( and(equal("name", "John"), gt("age", 16)) )
.withSort( Sorting("age", SortOrder.ASC) )
.withCursorPaging()
context.sdl.query.fromPlatformizedQuery(query).execute()
#1 Treat your platform as a product; your engineers as users
#2 Solve real problems
#3 Hyper Velocity - lies in solving common business problems via higher level abstractions
Summary
Velocity @ Scale | Platform Engineering
Thank You!
Any questions?
odeda@wix.com
savvak@wix.com
Platform Engineering Meetup #1
17:00 - 17:30 Gathering
17:30 - 18:20 Building a Platform for Real Engineers / Yuval Perry
18:20 - 19:10 A New Microservice in 1 hour / Savva Khalaman & Oded Apel
19:00 - 19:40 Open discussion