1 of 77

Optimize 2.0

Gesammelter Content

2 of 77

Optimize 2.0 Minimal Feature Set / Roadmap

3 of 77

Optimize 2.0 Minimal Feature set, further Roadmap

3

Rolling Start Date Filter	“All Process Instances that started in past 7 days”
Minimal Report Builder	As defined in next Slides
Save & Share Reports	Save Report on Server, find and open it via simple list
Minimal Dashboard Builder	Arrange Reports in resizable Tiles
Save & Share Dashboards	Save Dashboard on Server, find and open it via simple list (landing page)
Filter by Process Instance Duration	“All Process Instances that were/are running for more than 7 days”
Define Alerts	As defined in next Slides
Embed Dashboards	Via iFrame
Schedule Email Reports	As defined in next Slides
CSV Export	As defined in next Slides
Dashboards “Niceties”	Full View, Auto Refresh, Night-time Mode

Needs to be in 2.0

4 of 77

2.0 Report Builder Dimensions

4

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node Rolling start date Execution Duration Process Instances	Raw Data Process Instances Count Execution Frequency Process Instances Flow Nodes Avg Execution Duration Process Instances Flow Nodes Vs. Target Duration	None Timeline Start Date Process Instance Day \| Month \| Year Flow Nodes	Single Number Table (only x rows, no pagination) Chart Bar Line BPMN Heatmap Configure Goal (Number)

5 of 77

Possible Combinations

5

View	Grouped by	Visualize as	Configure Visualization
Raw Data Process Instances	None	Table	Pick (Variable) Columns
Count / Frequency / Process Instances	None	Number	Number <>= => BG-Color
	Start Date	Table	-
	Start Date	Bar / Line	Set “Goal” => Horizontal Line
Count / Frequency / Flow Node Instances	Flow Node	BPMN Heatmap/Table/Bar/Area	-
Avg / Duration / Process Instances	None	Number	Number <>= X => BG-Color
	Start Date	Table	-
	Start Date	Bar / Line	Set “Goal” => Horizontal Line
Avg / Duration / Flow Node Instances	Flow Node	BPMN Heatmap/Table/Bar/Area	Set “Goal” => Heat

6 of 77

Emails

Email Report

Define Report with rolling date filter, e.g. all process instances that were started yesterday
Schedule Report Email, e.g. every day at 8am,send Report to x@y.z
Time: dayly, weekly, monthly -> specialize exact time e.g. 12 pm
Send Email w/ Report attached (e.g. PDF) or inside (e.g. HTML)

Not only link to public report

(Email) Alerts

Define Report with rolling date filter and type “single number”
Define Email Alert for certain Report result: Number <>= X
Schedule Check for Alert: x Seconds | x Hours (Question: Relevant?)
An Email will be sent, e.g. when Report “All Process Instances that were started in the last hour, avg. duration > 1 minute, no grouping, single number” results in > 10.

6

7 of 77

Export as CSV

Report Results can be downloaded as CSV

Only available for Visualization “Table”

Limited amount of rows that can be downloaded (for tech reasons) would be ok

7

8 of 77

/ END Minimal Feature Set

9 of 77

UI/UX O 1 => O 2

10 of 77

Core Feature Clusters in O 2.0

Monitor

Report Builder

Filter
View
Group by
Visualize as

Dashboard Builder
Alert Configurator

Analyze

Branch Analysis

10

Covering O1 Views:

“Frequency”
“Duration”
“Target Value Comparison”

11 of 77

Let’s get rid of this

11

12 of 77

Possible Approach

12

13 of 77

Possible Approach

13

14 of 77

Next Steps

Define MVP Feature Set for Optimize 2

Define Roadmap and Milestones

Work out UX / UI based on these definitions

14

15 of 77

2.0 Vision: Reports

16 of 77

Why should we build this? Aren’t we re-inventing Kibana, Metabase, ...?

[Features Reasoning]

Generic BI products are not aware of process logic which makes it harder to get specific reports out of the box (e.g., target value comparisons, grouped by flow nodes, visualized in BPMN)

[Packaging Reasoning]

No additional product needs to be set up, operated and licensed
Integration with existing BI products can be time-consuming and expensive
Existing BI products are in different ownership than Camunda BPM, often limiting accessibility for integration, customization or simply usage

-

Still, we must be careful to draw a line since re-inventing Kibana would not make sense for us.

To validate Features Reasoning, as part of prototype, also set up Kibana with Optimize Data

16

17 of 77

Feature “bricks” and dependencies

17

Report Builder

Dashboards

Save & Share

Share via API

Export as CSV

Export as PDF

UI

API

Email Reports

Email Alerting

Save & Share

Engine

Ready-to-use Reports

18 of 77

Feature “bricks” and dependencies

18

Report Builder

Dashboards

Save & Share

Share via API

Export as CSV

Export as PDF

UI

API

Email Reports

Email Alerting

Save & Share

Engine

Ready-to-use Reports

19 of 77

Current Reports

19

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node	Count Execution Frequency Flow Nodes Avg Execution Duration Flow Nodes Vs. Target Value	Flow Nodes	BPMN Heatmap

20 of 77

Process Configurations / Target Duration / Flow Nodes

20

21 of 77

Process Configurations / Target Duration / Process Instances

21

Process “Lead Qualification”

22 of 77

Vision for Reports

22

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node Execution Duration Process Instances Vs. Target Duration Flow Nodes Vs. Target Duration With/Without Incidents	Raw Data Process Instances Count Execution Frequency Process Instances Flow Nodes Avg Execution Duration Process Instances Vs. Target Duration Flow Nodes Vs. Target Duration Variable Value (number) Sum Variable Value (number)	None Timeline Start Date Process Instance Flow Node Variable Values String, <=20 Values Boolean, true\|false Flow Nodes	Single Number Table (only x rows, no pagination) Chart Bar Line Pie BPMN number badges BPMN Heatmap Configure

23 of 77

Possible Visualizations for Views: Count, Avg, Sum

23

Grouped by	Number	Table	Bar/Line	Pie	BPMN Badge/Heat
None	X	X
Start Date		X	X
Variable Values		X	X	X
Flow Nodes		X	X	X	X

For View “Raw Data”, only Visualization “Table” is possible.

24 of 77

Example Report

24

Filter	View	Grouped by	Visualize as
Past 30 Days	Avg Execution Duration / Flow Nodes	Flow Node	BPMN Heatmap

25 of 77

Example Report: Count Process Instances per Day

25

Filter	View	Grouped by	Visualize as
Past 30 Days	Count Execution Frequency / Process Instances	Start Date, Day	Bar Chart

26 of 77

Example Report

26

Filter	View	Grouped by	Visualize as
Past 30 Days	Count Execution Frequency / Process Instances	Start Date, Day	Table

27 of 77

Example Report

27

Filter	View	Grouped by	Visualize as
Past 30 Days	Avg Execution Duration / Process Instances	Start Date, Day	Line Chart

28 of 77

Example Report

28

Filter	View	Grouped by	Visualize as
Past 30 Days	Avg Execution Duration / Process Instances	Start Date, Day	Line Chart, plus config “display avg target”

29 of 77

Example Report

29

Filter	View	Grouped by	Visualize as
Past 30 Days; Execution Duration > 3 days	Count Execution Frequency / Process Instances	Start Date, Day	Bar Chart

30 of 77

Example Report

30

Filter	View	Grouped by	Visualize as
Past 30 Days; Execution Duration > 3 days	Count Execution Frequency / Process Instances	None	Number

31 of 77

Example Report

31

Filter	View	Grouped by	Visualize as
Past 24 hours; Execution Duration > 1 minute	Count Execution Frequency / Process Instances	None	Number, plus config: 0-10 GREEN 11-50 YELLOW > 50 RED

96

32 of 77

Example Report

32

Filter	View	Grouped by	Visualize as
Past 30 Days; Execution Duration vs. Target: > 100%	Count Execution Frequency / Process Instances	Variable Values (Variable “Industry”)	Pie Chart

33 of 77

Example Report

33

Filter	View	Grouped by	Visualize as
Past 30 Days; Execution Duration vs. Target: > 100%	Avg Execution Duration / Flow Node Vs. Target	Flow Node	BPMN Heatmap

34 of 77

“Process Parts”

35 of 77

Process Configurations / Target Duration / Process Part Instances

35

Process Part Name:

When applying this filter or view, process instances will be automatically filtered to “only those instances where Start and End Node were executed”

When Loops are involved, only the first execution of start and end node of process part will be considered.

36 of 77

Vision for Reports

36

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node Execution Duration Process Instances Vs. Target Duration Flow Nodes Vs. Target Duration Process Part Instances Vs. Target Duration With/Without Incidents	Raw Data Process Instances Count Execution Frequency Process Instances Flow Nodes Avg Execution Duration Process Instances Vs. Target Duration Process Part Instances Vs. Target Duration Flow Nodes Vs. Target Duration Variable Value (number) Sum Variable Value (number)	None Timeline Start Date Process Instance Flow Node Variable Values String, <=20 Values Boolean, true\|false Flow Nodes	Single Number Table (only x rows, no pagination) Chart Bar Line Pie BPMN number badges BPMN Heatmap

37 of 77

Example Report

37

Filter	View	Grouped by	Visualize as
Past 30 Days;	Count Execution Frequency / Process Instances	Timeline: every 10 minutes	Line Area Chart, plus config: Display instances with avg process part duration > target duration in red

38 of 77

Stand off!

38

I believe “process parts” is a 1&1 specific concept and not relevant for the majority of Optimize users.

With all due respect sir, I disagree. I believe more than 50% of the Optimize users would want to use “process parts”.

39 of 77

The Bet, running until Camunda Day, December 8th

Track the interactions with (potential) optimize users regarding their interest in “process parts”
If “process parts” is a desired feature on more than 50% of these interactions, Jakob wins. If not, Johannes wins.
The Winner will pick a super hero costume that the loser will be wearing during Camunda Day.

39

DEPRECATED

40 of 77

The *new* Bet

When Optimize 2.0-alpha1 has been released, we create a short screencast as a feature preview
We then set up a short survey, asking for feedback. One question will explain the idea of “process parts” and ask how important the participant finds that feature for their own use cases. Possible answers: very important | important | not so important | Irrelevant
If more than 50% say the feature is (very) important, Jakob has won the bet. If not, Johannes.
The Winner will pick a super hero costume that the loser will be wearing during Camunda Day (January or April, not yet sure).

40

41 of 77

Feature “bricks” and dependencies

41

Report Builder

Dashboards

Save & Share

Share via API

Export as CSV

Export as PDF

UI

API

Email Reports

Email Alerting

Save & Share

Engine

Ready-to-use Reports

42 of 77

Save & Share

Save

Pick a name

Delete, Rename

Edit and Save / Save As

Organize in Collections (1 Level Hierarchy)

Share

Publish under URL (anyone can view it, not edit)

Embed via iFrame

Share Collection with other user groups (organized via Admin)

42

43 of 77

Share via API

Report can be accessed via REST API

Client can query report (without parameters) and will receive results as JSON

43

44 of 77

Export as CSV

Report Results can be downloaded as CSV

Only available for Visualization “Table”

Limited amount of rows that can be downloaded (for tech reasons) would be ok

44

45 of 77

Emails

Email Report

Define Report with rolling date filter, e.g. all process instances that were started yesterday
Schedule Report Email, e.g. every day at 8am,send Report to x@y.z
Send Email w/ Report attached (e.g. PDF) or inside (e.g. HTML)

(Email) Alerts

Define Report with rolling date filter and type “single number”
Define Email Alert for certain Report result, e.g. “more than 10”
Schedule Check for Alert, e.g. every hour
An Email will be sent, e.g. when Report “All Process Instances that were started in the last hour, avg. duration > 200% of target duration, no grouping, single number” results in > 10.

45

46 of 77

Feature “bricks” and dependencies

46

Report Builder

Dashboards

Save & Share

Share via API

Export as CSV

Export as PDF

UI

API

Email Reports

Email Alerting

Save & Share

Engine

Ready-to-use Reports

47 of 77

Dashboard

Arrange Reports as Tiles on a Page

Save and Share

(Re)name, Delete

Extras:

Put in Folder

Publish for User Group

Publish for everyone (Deep Link)

Embed (iFrame)

Resize Reports Tiles

Dashboard Page Full View

Auto Refresh Dashboard (e.g. every 5 minutes)

Night Mode (Inverted)

47

48 of 77

Scoping the Prototype

49 of 77

Prototype Hackathon Objectives

We want to

Play with the idea of a generic report builder UI and engine
Map the idea on specific example reports (see next page)
produce tangible results, i.e. something visible on the UI => example reports
Optional: Play with the idea of a dashboard that can include multiple reports

We do not want to

Validate technical feasibility
Figure out the right application architecture
Build the actual generic report builder UI or engine
Use anything we’ve built in the actual product development

49

50 of 77

Example Reports, prioritized

50

#	Filter	View	Grouped by	Visualize as	Done	In Kibana
1	Past 30 Days	Count / Process Instances	Start Date: Day	Bar Chart		Done
2	“	“	“	Table		Need to ask
3	“	“	None	Number		Done
4	“	“	Variable “Industry”	Pie Chart	BE: yes, limitation - String variables only	Need to ask
4a	“	Avg / Variable “Salary”	Start Date: Day	Bar Chart	BE: yes, limitation Int variables
5	“	Avg / Duration / Process Instances	Start Date: Day	Line Chart		Need to ask
6	“	“	“	Plus config: Show Target Duration		Need to ask
7	Past 30 Days; Duration > 10 days	Count / Process Instances	None	Numbers, plus config: Red Background if result > 20		Not possible (no target value definable, did not yet figure out how to filter based on duration, did not yet figure out how to visualize the result based on value)

51 of 77

Implementation for this Prototype

51

Elastic Search w/ LQ Process

Quick & Dirty API

{”Filter”: ...,

“View” :

{

“Type” : “count”,

“What” : “frequency”

“Object” : “processInstance”

},

“Group”:

“What” : “Start Date”

“Unit” : “Day”

}

{

”2017-09-01”: 23,

”2017-09-02”: 7,

”2017-09-02”: 11,

…

}

Report Request

Report Result

Quick & Dirty UI

Except for “View: Raw Data”, Report Result will always be 2-dimensional: Group Element and Number

52 of 77

Grober Schlachtplan

Seb: GitHub Branch Projekt mit Tickets | DONE

**

Johannes: Hiring Prozess mit Beispieldaten generieren | DONE

Askar: Backend auf AWS packen für Frontend-Testing Zwecke | In Progress

Johannes: Beispiel-Reports im Backend bauen / Input/Output JSON definieren und Frontend geben

Felix: Beispiel Reports machen

Request konfigurieren können und
anzeigen können (Bar Chart etc.)
Visualisierung konfigurieren können (z.B. Target Duration im Bar Chart einblenden)

Sebastian: Dashboard Builder machen

***

Jakob: Wie weit komm ich mit Kibana => Beispiel-Reports machen | In Progress

52

53 of 77

Implementation #3/#7

53

Elastic Search w/ LQ Process

Quick & Dirty API

{

"processDefinitionId" : "...",

“name” : “...”,

”filter”: ...,

“groupBy” :

{

“startDate” : null,

"variable" : null

}

{

“totalCount” : 12512

}

Report Request

Report Result

Quick & Dirty UI

Except for “View: Raw Data”, Report Result will always be 2-dimensional: Group Element and Number

54 of 77

Implementation #1/#2

54

Elastic Search w/ LQ Process

Quick & Dirty API

{

"processDefinitionId" : "...",

“name” : “...”,

”filter”: ...,

“groupBy” :

{

“startDate” :

{

"operation" : "count",

"interval" : "day"/"month"/...

},

"variable" : null

}

{

”2017-09-01”: 23,

”2017-09-02”: 7,

”2017-09-02”: 11,

…

}

Report Request

Report Result

Quick & Dirty UI

Example:

“startDate” :

{

"operation" : "count",

"interval" : “day”

},

"variable" : null

}

Possible intervals are:

hour/day/week/month

55 of 77

Implementation #4

55

Elastic Search w/ LQ Process

Quick & Dirty API

{

"processDefinitionId" : "...",

“name” : “...”,

”filter”: ...,

“groupBy” :

{

“startDate” : null,

"variable" :

{

"operation" : "count" / "avg",

"name" : "...",

"type" : "String" / "Double" / "Long" / "Boolean" / "Date" ,

“Interval” : “”

}

{

”Consulting”: 23,

”Insurance”: 7,

”Finance”: 11,

…

}

Report Request

Report Result

Quick & Dirty UI

Example:

{

"operation" : "count" ,

"name" : "Industry",

"type" : "String"

"interval" : ""

}

Possible intervals are:

hour/day/week/month�Only available for date variables

56 of 77

Implementation #4a

56

Elastic Search w/ LQ Process

Quick & Dirty API

{

"processDefinitionId" : "...",

“name” : “...”,

”filter”: ...,

“groupBy” :

{

“startDate” : {

“Interval”: “day”

},

"variable" :

{

"operation" : "avg",

"name" : "Salary",

"type" : "Long"

}

{

”Consulting”: 23,

”Insurance”: 7,

”Finance”: 11,

…

}

Report Request

Report Result

Quick & Dirty UI

Example:

{

"operation" : "count" ,

"name" : "Industry",

"type" : "String"

"interval" : ""

}

Possible intervals are:

hour/day/week/month�Only available for date variables

57 of 77

Example Queries #4

Valid Queries (Representations : pie\bar\table)

Count process instances in each industry
Count process instances with string variables
Average number of activities executed in each industry
Average duration of activities executed in every month of this year

Invalid Queries

Count process instances in each industry for every month of the year

Invalid since grouping by month in year is added (extra dimension)

57

58 of 77

Implementation #5/#6

58

Elastic Search w/ LQ Process

Quick & Dirty API

{

"processDefinitionId" : "...",

“name” : “...”,

”filter”: ...,

“groupBy” :

{

“startDate” :

{

"operation" : "avg",

"interval" : "day"/"month"/...

},

"variable" : null

}

{

”2017-09-01”: 23.5,

”2017-09-02”: 7.93,

”2017-09-02”: 11.52,

…

}

Report Request

Report Result

Quick & Dirty UI

Example:

“startDate” :

{

"operation" : "avg",

"interval" : “day”

},

"variable" : null

}

Possible intervals are: hour/day/week/month

59 of 77

Implementation #7

59

Elastic Search w/ LQ Process

Quick & Dirty API

{

"filter" : {

"durations" : [ {

"operator" : ">",

"unit" : "day",

"value" : 5

}]

},

"processDefinitionId" : "hiring-demo:1:bb2dc4e5-a8e7-11e7-b860-024210b8ff1e",

"name" : "bla",

"groupBy" :

{

"startDate" : null,

"variable" : null

}

{

"totalCount": 8534

}

Report Request

Report Result

Quick & Dirty UI

Example:

Possible units are: hour/day/week/month

60 of 77

Timeline

4 Days: Wednesday, Thursday, Friday, Monday

Location: Complex Gateway

Goal: Results Presentation to all on Monday 4pm

60

61 of 77

Backlog

Bis Freitag EOB

Example Reports fertig machen End-to-End - Askar, Johannes, Felix
Dashboard schick machen - Sebastian
Process Parts - Felix, Askar
Alerting - Sebastian, Johannes

Montag vormittag

Letzte Feature-Feinschliff-Themen - Alle
Demodaten Generierung auf Demo-Themen optimiert - Johannes

61

62 of 77

Process Parts

Count average duration of paths in process definition that contain Activity A and Activity B
I\O

Input
Output

Representation

62

63 of 77

Open Issues

Report Builder (Felix)

X Filter by Process Instance Duration does not seem to work

X Default Bar Chart Color pink is disgusting

X Remove Data Points for Target Value Line in Bar Chart

X Target Value Line should be thinner, dashed and in red and in front of chart

X Hover still says “Test Data” as Legend

Process Parts not yet working

Dashboard (Sebastian)

Embed Dashboard does not work (src=”file” ?)

Alert

Rolling Date Filter not yet there (Johannes)

Generate Process Instances takes too long (Johannes)

63

64 of 77

Demo Script Summary

Jakob - Intro - 5

Felix - Reports - 15

Sebastian - Dashboard - 10

Johannes - Alerts - 10

Jakob - Wrap up - 5

64

65 of 77

Results Demo 1

Intro: 2.0 Vision, Prototype Objectives

Live Demo

Create Reports

Count all Process Instances, then filter by start date => Number

Group by Start Date => Table => Bar Chart => Color => Target Value => Save

Avg Process Instance Duration => Area Chart => Save

Avg Variable Value “Salary Expectation” => Save

Count PI, group by Variable “Department” => Pie Chart => Save

Count PI, filter by PI duration “too long” => Bar Chart => Save

BPMN Heatmap flow node duration => Save

Process Parts based Demo Report?

65

66 of 77

Results Demo 2

Live Demo

Create Dashboard

Put Reports together

Embed in Confluence

Full Screen Mode

Auto Refresh

Night Time Mode

66

67 of 77

Results Demo 3

Live Demo

Create Alert

Create Alert on Count > 1, receive Email

The “Real Thing”

Create Report on Count, goes red when > now + 10

Add to Dashboard, Auto Refresh

Create Alert on Count > now+10

Start 100 Process Instances

Observe Dashboard, Wait for Email

Next Steps

67

68 of 77

Optimize 2.0 Prototype Hackathon

Results Demo

69 of 77

Prologue

Optimize 1.0 released in June
Huge interest from customers and prospects
We keep getting similar feature requests, e.g.

I want to know how long the process takes in total, not just single steps
I want to get a daily summary of what’s going on
I want to be notified when something is “bad”

Meanwhile, we’re getting requests for Cockpit that are clearly in the domain of Optimize (e.g. SUPPORT-3581)

69

70 of 77

Optimize Product Vision

70

Process in Camunda BPM

See

Improve

Understand

Business Activity Monitoring

Process Intelligence

Pattern Recognition

Machine Learning

Artificial Intelligence

Structural changes on the fly, without developers and app deployment cycles

71 of 77

Optimize 2.0: Reports

71

Report Builder

Dashboards

Save & Share

Share via API

Export as CSV

Export as PDF

UI

API

Email Reports

Email Alerting

Save & Share

Engine

Ready-to-use Reports

72 of 77

Current Reports

72

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node	Count Execution Frequency Flow Nodes Avg Execution Duration Flow Nodes Vs. Target Value	Flow Nodes	BPMN Heatmap

73 of 77

Vision for Reports

73

Filter Process Instances	View	Grouped by	Visualize as
Start Date Variable Value Executed Flow Node Execution Duration Process Instances Vs. Target Duration Flow Nodes Vs. Target Duration With/Without Incidents	Raw Data Process Instances Count Execution Frequency Process Instances Flow Nodes Avg Execution Duration Process Instances Vs. Target Duration Flow Nodes Vs. Target Duration Variable Value (number) Sum Variable Value (number)	None Timeline Start Date Process Instance Flow Node Variable Values String, <=20 Values Boolean, true\|false Flow Nodes	Single Number Table (only x rows, no pagination) Chart Bar Line Pie BPMN number badges BPMN Heatmap Configure

74 of 77

Hackathon Objective: Quick & Dirty!

We want to

Play with the idea of a generic report builder UI and engine
Map the idea on specific example reports
produce tangible results, i.e. something visible on the UI
Play with the idea of a dashboard that can include multiple reports

We do not want to

Validate technical feasibility
Figure out the right application architecture
Build the actual generic report builder UI or engine
Use anything we’ve built in the actual product development

74

75 of 77

Demo Time!

75

76 of 77

Why would we build this? Aren’t we re-inventing Kibana, Metabase, ...?

[Features Reasoning]

Generic BI products are not aware of process logic which makes it harder to get specific reports out of the box (e.g., target value comparisons, grouped by flow nodes, visualized in BPMN)

[Packaging Reasoning]

No additional product needs to be set up, operated and licensed
Integration with existing BI products can be time-consuming and expensive
Existing BI products are in different ownership than Camunda BPM, often limiting accessibility for integration, customization or simply usage

76

77 of 77

Next Steps

Finish Optimize 1 UI Refactoring

Talk / Demo 2.0 Prototype to interested customers (e.g., 1&1)

Figure out concrete Roadmap for 2.0

Start Developing 2.0

77