OpenDaylight technical utils

Sep 22/23, 2018 @ Neon Developer Design Forum (Amsterdam)

Michael Vorburger.ch, Red Hat

What's new in infrautils?

March 26th, 2018 @ Fluorine Developer Design Forum (ONS, LA)

Michael Vorburger.ch, Red Hat

$ history

​

Infrautils is an ODL project initiated by HPE (no longer active), which I’ve joined 2 years ago and since contributed significantly to.

​

This will present mostly #DONE as well as some #IDEAS.

​

#helpwanted = YOUR contributions most welcome!

​

https://wiki.opendaylight.org/view/Infrastructure_Utilities:Main

This OpenDaylight SDK presentation is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Infrautils’ project scope

• Offers technical utilities and infrastructure for other projects
• Used in genius and netvirt, ..., and soon YOUR project?

​

• Intentionally only depends on odlparent
• No dependencies to controller, mdsal, yangtools

5. Caches
6. Ready
7. DiagStatus
8. Origin & Trace (*)

• Metrics (Counters)
• @Inject
• [[I]Test]Utils
• JobCoordinator

This OpenDaylight SDK presentation is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Meanwhile there are some other interesting new free :) utilities elsewhere as well:

• infrautils.parent
• aaa.web API
• genius.tools
• odl.simple (*)

This OpenDaylight SDK presentation is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

1. Metrics

Metrics (Counters)

All about Metrics in this presentation.

​

Counters was the predecessor to Metrics.

Counters will be removed from infrautils git in Fluorine.

#helpwanted to switch any remaining counters usages to metrics.

2. @Inject

Wiring with Guice for Component Tests

infrautils.inject & .guice.testutils

Q: How to “wire” together a bunch of related Java classes (bundles’ inner @Bean in OSGi Blueprint BP plus external OSGi Services) for an End2End “component” JUnit Test?

A: Use @Singleton @Inject @PostConstruct @PreDestroy @OsgiService @OsgiServiceProvider annotations instead of BP XML, write Test*Module and use infrautils’ GuiceRule in *Test!

https://wiki.opendaylight.org/view/BestPractices/DI_Guidelines

https://wiki.opendaylight.org/view/BestPractices/Component_Tests

class AclServiceModule extends GuiceModule {

​

@Override protected void configureBindings() {

​

① bind(AclServiceManager.class)� .to(AclServiceManagerImpl.class);�� ② bind(AclInterfaceStateListener.class);

JUnit GuiceRule to @Inject into *Test

public class YourTest {

​

@Rule MethodRule guice =

new GuiceRule(

YourModule1.class,

YourModule2.class);

​

@Inject YourService s;

See many existing component tests in netvirt & genius and my previous presentation�for more details!

​

AbstractGuiceJsr250Module, AnnotationsModule, AbstractCheckedModule and Lifecycle interface.

3. Utils, TestUtils, ITestUtils

Utilities incl. for unit & component tests

org.opendaylight.infrautils.utils

Q: Where in ODL to put low level general utility classes à la Apache (Lang) Commons or Guava and similar, incl. test related helpers?

A: Here.

Door open for… yours?

org.opendaylight.infrautils.utils.concurrent

ListenableFutures toCompletionStage

CompletableFutures|Stages completedExceptionally

​

LoggingRejectedExecutionHandler, LoggingThreadUncaughtExceptionHandler, ThreadFactoryProvider, Executors newSingleThreadExecutor

org.opendaylight.infrautils.testutils

• LogRule
• LogCaptureRule
• RunUntilFailureRule
• ClasspathHellDuplicatesCheckRule

@Rule LogRule logRule = new LogRule();

@Rule LogCaptureRule logCaptureRule = new LogCaptureRule();

​

static @ClassRule RunUntilFailureClassRule� classRepeater = new RunUntilFailureClassRule(10);

@Rule RunUntilFailureRule � repeater = new RunUntilFailureRule(classRepeater);

​

static @ClassRule ClasspathHellDuplicatesCheckRule

jHades = new ClasspathHellDuplicatesCheckRule();

[main] INFO ...tests.ExampleTest - BEGIN @Test testA()

[main] INFO ...tests.ExampleTest - doin' stuff in testA...

[main] INFO ...tests.ExampleTest - END (51ms) @Test testA()

[main] INFO ...tests.ExampleTest - ============================

[main] INFO ...tests.ExampleTest - BEGIN @Test testB()

[main] INFO ...tests.ExampleTest - doin' stuff in testB...

[main] INFO ...tests.ExampleTest - END (100ms) @Test testB()

[main] INFO ...tests.ExampleTest - ============================

LogCaptureRule

• Hooks autom. into slf4j-simple (used in our tests behind slf4j API)
• Fails (typically component/end2end) test if there was a LOG.error()
• From any thread, not just main thread! Find problems in BG threads..

​

• Unless you tell it you are asserting to expect an LOG.error!
• If your test failed AND there was a LOG.error, then BOTH are shown!

org.opendaylight.infrautils.testutils.concurrent

• AwaitableExecutorService
• SlowExecutor
• CompletionStageTestAwaiter

org.opendaylight.infrautils.itestutils

Some glue to make using PAX Exam to simplify writing OSGi Integration Tests for ODL Apps.

​

Bit similar to something older in controller - minus CSS, plus easy parent POM.

AbstractIntegrationTest

• SampleIntegrationTest
• CachesIntegrationTest
• DiagStatusIT
• SystemReadyTest

org.opendaylight.infrautils.itestutils

Missing Karaf CLI Command testability. Totally possible...

​

#IDEA #helpwanted

4. JobCoordinator

by Ericsson, used (a lot) in genius/netvirt

JobCoordinator

Q: How to run async background jobs, which need ordering?

A: JobCoordinator enqueueJob.

​

Includes rollback compensation job, and retries.

Originally in ODL project genius, moved to infrautils to share.

Used very (!) heavily in netvirt & genius. (Too? RT w c/63471)

5. Caches

ConcurrentHashMap without overfill, with monitoring

Caches

Q: ConcurrentHashMap used [too] extensively in application code - at least in genius and netvirt. But it’s problematic, because unbounded (OOM), and its API is easy to misuse, and cannot be monitored (without re-writing similar CLI commands, everywhere). But what else is there?

A: infrautils.caches! It’s just a thin layer around existing caching FMK; e.g. Google Guava Cache (DONE)

#TODO integration with infrautils.metrics

org.opendaylight.infrautils.caches

Cache<String, String> hellosCache =� cacheProvider.newCache(� new CacheConfigBuilder<String, String>()� .anchor(this)� .cacheFunction(� key -> sayHelloThatIsExpensive(key))� .id("hellos") // optional� .description("Hellos Hallos") // opt� .build());

Caches CLI: list & clear

cache:list ==>

​

cache:clear

​

cache:policy� hellos � maxEntries� 100000

Cache ID: hellos� description: Hellos Hallos� anchored in: ...YourService@123�� Policies� * statsEnabled = true� * maxEntries = 1000� Stats� * entries: 478� * hitCount: 134� * missCount: 72� * ...

OpenDaylight Karaf OSGi caching infrautils

https://www.youtube.com/watch?v=h4HOSRN2aFc

6. Ready

infrautils.ready (Bug 8415)

Q: How can a human or code known when ODL is “ready” ?

A: infrautils.ready.SystemReady Monitor getSystemState():

BOOTING, ACTIVE, FAILURE.

​

registerListener(SystemReadyListener) onSystemBootReady()

It’s basically our build time SFT available at run-time!

Q: How to “order” service startup seq?

For example, make an application like netvirt/genius (IdManager) wait for boot-on- import “signal” event from DAEXIM...

A: Do this by publishing “marker” services in OSGi!

​

c/61769 has an example of a one-off solution; it works, but the code is not so clear.

c/61480 has a FunctionalityReady “sugar marker” parent, JUST to make this “pattern” explicit TODO Impl & use e.g. in Daexim #helpwanted

7. DiagStatus

DiagStatus

Q: How to clearly identify dynamic system health by higher-level view of functional service “operational” status? (“Service” used here not as in technical lower-level OSGi service, but functionality offered by 1..N OSGi bundles’ services.)��A: infrautils.diagstatus

Orig by faseela.k@ericsson.com

Cluster status, pull & push API, JMX, CLI, infrautils.ready integration…

�See also its Spec in c/51171

DiagStatus

ServiceStatusProvider

DiagStatusService

DatastoreServiceStatusProvider

feature:install odl-infrautils-diagstatus

feature:install odl-genius

GET http://localhost:8181/diagstatus (see INFRAUTILS-33)

8. Origin & Trace

org.opendaylight.infrautils.utils.mdc

Q: In an ocean of log messages, how do you correlate what belongs together?

A: We’d have to create IDs at “incoming”, store in a thread local context, pass it across thread boundaries… and include it in each LOG.

slf4j MDC is great for this!

Started on gerrit/#/q/topic:mdc

TODO #IDEA #helpwanted

• ExecutionOriginFilter (in a new TBD infrautils.web)
• MDSAL YANG RPC int.

​

Distributed Tracing?

Linux Foundation Cloud Native Computing Foundation (CNCF) Open Tracing seems a very useful foundation for this!

​

TODO: ODL infrautils integration!

We need prev. slide, first.

#IDEA #helpwanted

It would be COOL (and useful for performance analysis?) to be able to visualize & measure the flow of an “event” originating while still inside OpenStack northbound, through ODL’s various inner layers (app code, MDSAL, incl. all background threads!), ODL cluster nodes, and into OVS southbounds...

Other Utilities...

Not in infrautils

9. infrautils.parent

Parent POM, extending odlparent, with more code quality checks:

• PMD CPD enforcement fails build for copy/pasted code
• error-prone (incl. suitable configuration)
• Classpath Duplicate Finder enforced
• modernizer-maven-plugin (c/67838)
• NullAway? (c/67839)
• Detect slf4j API bugs
• CS & FB, of course

Used by all code in infrautils itself; available to other projects.

​

10. Web API

Programmatic registration of Servlets, Filters (and Context Listeners and Parameters): WebContext[Builder] and WebServer service. Alt. to declarative web.xml, for:

• Instances, not class names; avoids (very) ugly hacks
• non-OSGi, with implementation for component tests

In AAA but not dependant on AAA, should be in infrautils.

11. genius.tools (infra)

1. FutureRpcResults (c/63413)
2. ManagedNewTransactionRunner (c/63372 & c/63414)
3. RetryingMgdTransactionRunner (c/63471)
4. DataBrokerFailures (c/63120)
5. [InstanceId]DataObjectCache

∀ let’s see..

• Why?
• How?
• You?

FutureRpcResults (c/63413)

• Correct logging and error handling
• propagate root cause to RpcResult correctly
• LOG.error if exceptions (unless you don’t want to, rarely)
• LOG.debug successes, just in case
• include RPC input/output context in LOG
• Async encouragement, e.g. don’t Future get(), return!
• Correctly does the Future<O> to Future<RpcResult<O>> thing
• Avoid “boiler plate” code
• RPC impl everywhere become shorter and thus more readable
• Avoid bugs (e.g. propagation of cause, usually, incorrect)

FutureRpcResults (c/63413)

How to use? See example, but in each RPC method, replace:

​

try { … } catch (…) and …RpcResultBuilder…

​

with:

return FutureRpcResults.fromListenableFuture(LOG, input, () -> { … } ).build();

ManagedNewTransactionRunner

• MDSAL Transaction Management sometimes done wrong
• Always, EVEN if your code abort with any sort of Exception:
• [Read]WriteTransaction MUST either submit() or cancel()
• ReadTransaction MUST always close()
• Share a managed transaction, not SingleTransactionDataBroker
• Above becomes more interesting - where do you now do this?
• New (very small!) utility with more a “pattern” how to do this right
• Avoids “boiler plate” code
• All related code shorter and more readable by using abstraction
• Avoid bugs, and possibly memory leaks

ManagedNewTransactionRunner

How to use? See an example, and another, but basically:

​

txRunner

.callWithNewReadWriteTransactionAndSubmit(tx -> {

// do things with the transaction

}));

​

It will submit or cancel on Exception. It also has a *WriteOnly* instead of *ReadWrite* variant. It returns the Future, non blocking!

RetryingManagedNewTransactionRunner

Q: How do you handle OptimisticLockFailedException? The JavaDoc of WriteTransaction submit() tells you how - but do you do that everywhere? Correctly, with the async Future? Really?? ;-)

A: This new utility will do it for you - using THE SAME API as seen on the previous slide (but it’s separately implemented, you choose).

PS: Perhaps we don’t (always) need the [DataStore]JobCoordinator anymore if it’s only for retries, and some of its usages can be replaced by this?

DataBrokerFailures (c/63120)

Q: How do you simulate failures from the DataBroker in your JUnit (Component) tests? Because you ARE, or at least want to, test your components, both for best and worst cases, aren’t you? �

dbFailureSimulator.failSubmits(� new OptimisticLockFailedException("boum!"));

…

dbFailureSimulator.unfailSubmits();

DataObjectCache InstanceIdDataObjectCache

Use instead of your own Map of YANG gen. types

Makes sure you get cache invalidation done right (listener)

#TODO delegate to infrautils.caches, for built-in metrics

12. opendaylight.SIMPLE POC

Personal project exploring feasibility of running typical ODL application (such as netvirt) as a “simple” package assembled at build time. No Karaf. No OSGi.

​

https://github.com/vorburger/opendaylight-simple WIP

Findings gradually make their way into ODL (simple-dist*).

Q & A