1 of 38

WebRTC�

How we built ITMS2014+

...and did not pay for software licences

​

Martin Kováčik

@martinkovacik

2 of 38

#01

Why open source ?

#02

ITMS2014+ architecture

#03

Some interesting numbers

#04

Lessons learned & Final thoughts

3 of 38

Wrong

“Let’s build it on open source, it’s FREE”

4 of 38

Better

What are the requirements ?

​

Are they met by existing open source frameworks, libraries, tools... ?

​

Are they mature enough and alive ?

​

… and then, build it on open source :)

​

5 of 38

Requirements

In no particular order ...

​

  • existing codebase
  • large team
  • frequent changes
  • maintainability
  • performance
  • developer productivity
  • testability
  • scalability
  • no vendor lock-in
  • UI, UX
  • price

Compiled language

6 of 38

Requirements

In no particular order ...

​

  • existing codebase
  • large team
  • frequent changes
  • maintainability
  • performance
  • developer productivity
  • testability
  • scalability
  • no vendor lock-in
  • UI, UX
  • price

Localhost development with minimal external dependencies

Actual developer machine

7 of 38

Requirements

In no particular order ...

​

  • existing codebase
  • large team
  • frequent changes
  • maintainability
  • performance
  • developer productivity
  • testability
  • scalability
  • no vendor lock-in
  • UI, UX
  • price

No JSFs, Liferays, ...

8 of 38

Database requirements

  • Transactional DDL
  • Indexes on Expressions
  • Fast substring search
  • Partial indexes
  • Window functions
  • Json datatype
  • Array type
  • Materialized views
  • (Recursive) Common Table Expressions
  • SQL Server
  • DB2
  • Oracle DB
  • MySQL
  • PostgreSQL

9 of 38

Application server requirements

  • Fast code/deploy/test
  • Low memory footprint
  • We don’t need full JEE compatibility
  • Servlet 3.1 (websockets, NIO)
  • Same development and deployment env.
  • Weblogic
  • Websphere
  • JBoss
  • Glassfish
  • Tomcat
  • Jetty

10 of 38

#02

ITMS2014+ architecture

#01

Why open source

#03

Some interesting numbers

#04

Lessons learned & Final thoughts

11 of 38

ITMS2014+ architecture stack

  • Lang: Java, Javascript, Go, SQL, ... shell
  • DB: PostgreSQL
  • App Server: Jetty
  • OS: GNU/Linux (RHEL)
  • Web framework: Apache Wicket
  • DI: Spring Framework
  • ORM: (JPA2) Hibernate + Querydsl
  • DB Migrations: Flyway (Flyway + Transactional DDL rocks)
  • JMS: HornetQ
  • Auth: Apache Shiro, OpenLDAP
  • Build: Apache Maven
  • VCS: Git + BitBucket
  • CI: Jenkins
  • Load Balancer: HAProxy
  • Static site generator: Hugo
  • Reporting: Jasper Reports
  • Testing: JUnit, Selenium, Gatling
  • Logging: SLF4J + Logback, ELK + Redis
  • UI: Twitter Bootstrap, Font Awesome
  • JS: D3, jQuery
  • Integrations: Apache CXF, Akka
  • OLAP: Mondrian + Saiku
  • Scheduling: Quartz
  • Infra: Puppet
  • Bug Tracking: Jira

Full list at: www.itms2014.sk/we-love-oss (version 6.3+)

12 of 38

13 of 38

14 of 38

15 of 38

16 of 38

Git branching model (http://bit.ly/197szRP)

  • Separate version branches
    • release/6.3, release/7.1, ...
  • Feature/bugfix branches
    • feature/ITMS-1234
    • bugfix/ITMS-2345
  • PRs: feature/bugfix -> version branch
    • Every PR is code reviewed
    • Rejected PR is sent to original developer

17 of 38

Audit subsystem - roll your own

Requirements:

​

  • everything needs to be recorded
  • even data changes...
  • ...integration messages
  • ...outgoing email messages
  • ...user activity
  • ...

​

A LOT of data

18 of 38

Audit subsystem - roll your own

“MongoDB is web scale”

www.mongodb-is-web-scale.com

Requirements:

​

  • everything needs to be recorded
  • even data changes...
  • ...integration messages
  • ...outgoing email messages
  • ...user activity
  • ...

A LOT of data

19 of 38

20 of 38

Audit subsystem - ELK + Redis Stack

Server 1

+

Logstash forwarder

Server n

+

Logstash forwarder

. . .

Redis

+

Logstash indexer

Elasticsearch

Kibana

21 of 38

Audit subsystem - ELK + Redis Stack

22 of 38

Online documentation server

Requirements:

​

  • versioned documentation
  • multiple environments support
  • easy to work with
  • parallel editing
  • separate content and style

23 of 38

Online documentation server

Requirements:

​

  • versioned documentation
  • multiple environments support
  • easy to work with
  • parallel editing
  • separate content and style

140 lines of GO code

24 of 38

Testing + App monitoring

  • Unit tests
  • Repeatable integration tests
  • UI Tests
    • WicketTester
    • Selenium
  • Manual tests
  • Performance tests
    • JMeter
    • Gatling
  • Zabbix
  • New Relic (let’s hope)

25 of 38

#03

Some interesting numbers

#01

Why open source

#02

ITMS2014+ architecture

#04

Lessons learned & Final thoughts

26 of 38

What people think...

… how it really is

27 of 38

Is ITMS2014+ big ?

java

?

sql

?

css

?

jrxml

?

js

?

utf8.properties

?

xml

?

html

?

go

?

LOC per filetype

Number of unique screens

?

Project build time ?

?

28 of 38

Is ITMS2014+ big ?

java

864160

sql

298228

css

56764

jrxml

53717

js

38448

utf8.properties

16228

xml

13212

html

9326

go

294

LOC per filetype

Number of unique screens

1224

Project build time ?

01:30 min excl. tests

12:58 min incl. tests

​

29 of 38

  • App can easily sustain 6k simultaneous users => ~630 reqs/s
  • 95th percentile response time @6k users => ~771 ms
  • Majority of requests served within 1000 ms
  • … and still a lot of room for improvements

30 of 38

#04

Lessons Learned & Final thoughts

#01

Why open source

#02

ITMS2014+ architecture

#03

Some interesting numbers

31 of 38

ORM

is DIFFICULT to tame

​

generates suboptimal queries

​

N+1 behind every corner

​

only good for CRUDs

​

​

​

32 of 38

Sometimes you don’t need

Enterprise Service Bus

33 of 38

When not sure prototype

DO NOT trust white collar architects & product materials

34 of 38

Things change. Adapt.

35 of 38

Developer productivity is crucial

36 of 38

Experiment !

you may be surprised what happens :)

37 of 38

Prefer simple solutions

​

&

​

Don’t be MEGALOMANIAC

38 of 38

Thanks!

Martin Kováčik

@martinkovacik