ThoughtWorks Technology Radar Vol. 20
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

View only
CypressadopttoolsTRUE<p>We keep receiving positive feedback on "post-Selenium" web UI testing tools such as <strong><a href="">Cypress</a></strong>, <a href="/radar/tools/testcafe">TestCafe</a> and <a href="/radar/languages-and-frameworks/puppeteer">Puppeteer</a>. Running end-to-end tests can present challenges, such as the long duration of the running process, the flakiness of some tests and the challenges of fixing failures in CI when running tests in headless mode. Our teams have had very good experiences with Cypress by solving common issues such as lack of performance and long wait time for responses and resources to load. Cypress has become the tool of choice for end-to-end testing within our teams.</p>
JupyteradopttoolsTRUE<p>Over the past couple of years, we've noticed a steady rise in the popularity of analytics notebooks. These are Mathematica-inspired applications that combine text, visualization and code in a living, computational document. <strong><a href="">Jupyter</a></strong> Notebooks are widely used by our teams for prototyping and exploration in analytics and machine learning. We've moved Jupyter to Adopt for this issue of the Radar to reflect that it has emerged as the current default for Python notebooks. However, we caution to use <a href="/radar/techniques/productionizing-jupyter-notebooks">Jupyter Notebooks in production</a>.</p>
LocalStackadopttoolsTRUE<p>One of the challenges of using cloud services is being able to develop and test locally. <strong><a href="">LocalStack</a></strong> solves this problem for <a href="/radar/platforms/aws">AWS</a> by providing local <a href="">test double</a> implementations of a wide range of AWS services, including S3, Kinesis, DynamoDB and Lambda. It builds on top of best-of-breed tools such as <a href="">Kinesalite</a>, <a href="">dynalite</a> and <a href="">Moto</a> and adds isolated processes and error injection functionality. LocalStack is very easy to use, ships with a simple JUnit runner and a JUnit 5 extension and can also run inside a docker container. For many teams, it has become the default for testing services that are deployed on AWS.</p>
TerraformadopttoolsTRUE<p><strong><a href="">Terraform</a></strong>, is rapidly becoming a de facto choice for creating and managing cloud infrastructures by writing declarative definitions. The configuration of the servers instantiated by Terraform is usually left to Puppet, Chef or Ansible. We like Terraform because the syntax of its files is quite readable and because it supports a number of cloud providers while making no attempt to provide an artificial abstraction across those providers. The active community will add support for the latest features from most cloud providers. Following our first, more cautious, mention of Terraform almost two years ago, it has seen continued development and has evolved into a stable product with a good ecosystem that has proven its value in our projects. The issue with state file management can now be sidestepped by using what Terraform calls a "remote state backend." We've successfully used <a href="">AWS S3</a> for that purpose.</p>
UI dev environmentsadopttoolsTRUE<p>As more and more teams embrace <a href="/radar/techniques/designops">DesignOps</a>, practices and tooling in this space mature. <strong>UI dev environments</strong> provide a comprehensive environment for quickly iterating on UI components, focusing on collaboration between user experience designers and developers. We now have a few options in this space: <a href="">Storybook</a>, <a href="">React Styleguidist</a>, <a href="">Compositor</a> and <a href="">MDX</a>. You can use these tools standalone in component library or design system development as well as embedded in a web application project. Many teams were able to decrease their UI feedback cycles and improve timing of UI work in preparation for development work, which has made using UI dev environments a reasonable default for us.</p>
AnyStatustrialtoolsTRUE<p>As developers used to pushing many small commits daily, we rely on monitors to notify us when builds go green. <strong><a href="">AnyStatus</a></strong> is a lightweight Windows desktop app that rolls up metrics and events from various sources into one place. Examples include build results and releases, health checks for different services and OS metrics. Think of it as CCTray on steroids. It's also available as a Visual Studio plugin.</p>
AVAtrialtoolsTRUE<p><strong><a href="">AVA</a></strong> is a test runner for Node.js. Even though JavaScript is single-threaded, IO in Node.js can happen in parallel because of its asynchronous nature. AVA takes advantage of this and runs your tests concurrently, which is especially beneficial for IO-heavy tests. In addition, test files are run in parallel as separate processes, giving you even better performance and an isolated environment for each test file. AVA is a lightweight option, when compared to full-featured frameworks such as <a href="/radar/languages-and-frameworks/jest">Jest</a>. It is opinionated and forces you to write atomic test cases.</p>
batecttrialtoolsTRUE<p>So much energy and effort continue to be wasted on configuring local development environments and troubleshooting the "works on my machine" problem. For many years our teams have adopted the <a href="">"check out and go"</a> approach where we use a scripted approach to ensure the local development environment is configured consistently. <strong><a href="">batect</a></strong> is an open source tool developed by a ThoughtWorker that makes it easy to set up and share a build environment based on <a href="/radar/techniques/docker-for-builds">Docker</a>. batect becomes the entry point script for your build system, launching containers to perform build tasks that don't rely at all on local setup. Changes to build configuration and dependencies are simply shared through source control without requiring any changes or installations on local machines or CI agents. While we like <a href="/radar/tools/cage">Cage</a>, among other tools, in this space, we see batect quickly growing in favor with our teams.</p>
Elasticsearch LTRtrialtoolsTRUE<p>One of the challenges of search is ensuring the most relevant results for the user appear at the top of the list. This is where learning to rank (LTR) can help. LTR is the process of applying machine learning to rank documents retrieved by a search engine. If you're using <a href="/radar/platforms/elastic-search">Elasticsearch</a>, you can achieve search-relevant ranking with the <a href=""><strong>Elasticsearch LTR</strong></a> plugin. The plugin uses <a href="">RankLib</a> for generating the models during the training phase. Then, when querying Elasticsearch, you can use this plugin to "rescore" the top results. We've used it in a few projects and have been happy with the results. There's also an equivalent <a href="">LTR solution</a> for Solr users.</p>
HelmtrialtoolsTRUE<p><a href=""><strong>Helm</strong></a> is a package manager for <a href="/radar/platforms/kubernetes">Kubernetes</a>. It comes with a repository of curated Kubernetes applications that are maintained in the official <a href="">Charts repository</a>. Helm has two components: a command line utility called Helm and a cluster component called Tiller. Securing a Kubernetes cluster is a wide and nuanced topic, but we highly recommend setting up Tiller in a role-based access control (RBAC) environment. We've used Helm in a number of client projects and its dependency management, templating and hook mechanism has greatly simplified the application lifecycle management in Kubernetes. However, we recommend proceeding with caution — Helm's <a href="/radar/techniques/templating-in-yaml">YAML templating</a> can be difficult to understand, and Tiller still has some rough edges. Helm 3 is expected to address these issues.</p>
InSpectrialtoolsTRUE<p>How does an organization give autonomy to delivery teams while still making sure their deployed solutions are safe and compliant? How do you ensure that servers, once deployed, maintain a consistent configuration without drift? <strong><a href="">InSpec</a></strong> is positioned as a solution for continuous compliance and security, but you can also use it for general infrastructure testing. InSpec allows the creation of declarative infrastructure tests, which can then be continuously run against provisioned environments including production. Our teams particularly praise its extensible design with resources and matchers for multiple platforms. We recommend trialling InSpec as a solution to the problem of assuring compliance and security.</p>
LottietrialtoolsTRUE<p>Good UI animation could greatly improve user experience. However, to reproduce a designer's delicate animation on an app is usually a challenging task for developers. <strong><a href="">Lottie</a></strong> is a library for Android, iOS, web, and Windows that parses Adobe After Effects animations exported as JSON with Bodymovin and renders them natively on mobile and on the web. Both designers and developers can continue to use their familiar tools and have a fluent collaboration.</p>
StolontrialtoolsTRUE<p>Setting up highly available PostgreSQL instances can be tricky, which is why we like <a href="/radar/tools/patroni">Patroni</a> — it helps us speed up the setup of PostgreSQL clusters. <a href=""><strong>Stolon</strong></a> is another tool that we've used successfully to run high-availability (HA) clusters of PostgreSQL instances in production using Kubernetes. Although PostgreSQL supports streaming replication out of the box, the challenge in an HA setup is to assure that the clients always connect to the current master. We like that Stolon enforces the connection to the right PostgreSQL master by actively closing connections to unelected masters and routing requests to the active one.</p>
TestCafetrialtoolsTRUE<p>We have good experience using "post-Selenium" web UI testing tools such as <a href="/radar/tools/cypress">Cypress</a>, <strong><a href="">TestCafe</a></strong> and <a href="/radar/languages-and-frameworks/puppeteer">Puppeteer</a>. TestCafe lets you write tests in JavaScript or <a href="/radar/languages-and-frameworks/typescript">TypeScript</a> and runs in-browser tests. TestCafe has several useful features that include out-of-the-box parallel execution and HTTP request mocking. TestCafe uses an asynchronous execution model with no explicit wait times, which results in much more stable test suites. Its selector API makes it easier to implement <a href="">PageObject</a> patterns. TestCafe recently released version 1.0.<em>x</em>, which improved stability and functionality.</p>
TraefiktrialtoolsTRUE<p><strong><a href="">Traefik</a></strong> is an open-source reverse proxy and load balancer. If you're looking for an edge proxy that provides simple routing without all the features of <a href="">NGINX</a> and <a href="">HAProxy</a>, Traefik is a good choice. The router provides a reload-less reconfiguration, metrics, monitoring and circuit breakers that are essential when running microservices. It also integrates nicely with <a href="/radar/tools/let-s-encrypt">Let's Encrypt</a> to provide SSL termination as well as infrastructure components such as Kubernetes, Docker Swarm or Amazon ECS to automatically pick up new services or instances to include in its load balancing.</p>
AnkaassesstoolsTRUE<p><strong><a href="">Anka</a></strong> is a set of tools to create, manage and distribute build and test macOS reproducible virtual environments for iOS and macOS development. It brings Docker-like experience to macOS environments: instant start, CLI to manage virtual machines and registry to version and tag virtual machines for distribution. We discovered Anka when we proposed a macOS private cloud solution to a client. This tool is worth considering when applying DevOps workflow to iOS and macOS environments.</p>
CageassesstoolsTRUE<p><strong><a href="">Cage</a></strong> is an open-source wrapper around <a href="">Docker Compose</a> that lets you configure and run multiple dependent components as a larger application. It lets you orchestrate the execution of components such as Docker images, service source code from repo, scripts to load datastores and pods, which are containers that run together as a unit. Cage uses the Docker Compose v2 configuration file format. It addresses some of the Docker Compose gaps such as supporting multiple environments, including the dev environment for running a distributed application on the local developer machine and the test environment for running integration tests and production.</p>
CiliumassesstoolsTRUE<p>Traditional Linux network security approaches, such as iptables, filter on IP address and TCP/UDP ports. However, these IP addresses frequently churn in dynamic microservices environments. By leveraging Linux <a href="">eBPF</a>, <strong><a href="">Cilium</a></strong> provides API-aware networking and security by transparently inserting security in a way that is based on service, pod or container identity in contrast to IP address identification. By decoupling security from addressing, Cilium could play a significant role as a new network protection layer and we recommend you to check it out.</p>
DetektassesstoolsTRUE<p><strong><a href="">Detekt</a></strong> is a static code analysis tool for <a href="/radar/languages-and-frameworks/kotlin">Kotlin</a>. It finds code smells and code complexity. You can run it from the command line or use its plugins for integration with popular developer tools such as <a href="/radar/tools/gradle">Gradle</a> (to perform code analysis via builds) or <a href="">SonarQube</a> (to perform code coverage in addition to static code analysis), and IntelliJ. Detekt is a great addition to build pipelines of Kotlin applications.</p>
FlagrassesstoolsTRUE<p><a href="">Feature toggles</a> are an important technique in continuous deployment scenarios. We've come across a number of good home-grown solutions, but we do like the approach <strong><a href="">Flagr</a></strong> takes: a complete feature toggle as a service, distributed as a Docker container. It comes with SDKs for all major languages, has a simple and well-documented REST API and provides a convenient frontend.</p>
GremlinassesstoolsTRUE<p><a href=""><strong>Gremlin</strong></a> is a SaaS solution for organizations to conduct <a href="/radar/techniques/chaos-katas">chaos experiments</a> and help test the resilience of their systems. It comes with a series of failure attacks — including resource, network and state failures — that can be run ad hoc or on schedule and require minimal setup (especially for <a href="/radar/platforms/kubernetes">Kubernetes</a> users, who can run <a href="/radar/tools/helm">Helm</a> to install Gremlin). The Gremlin client also has a nice web-based user interface, which makes it easy to execute and manage chaos experiments.</p>
HoneycombassesstoolsTRUE<p><strong><a href="">Honeycomb</a></strong> is an observability tool that ingests rich data from production systems and makes it manageable through dynamic sampling. Developers can log large amounts of rich events and decide later how to slice and correlate them. This interactive approach is useful when working with today's large distributed systems because we've passed the point where we can reasonably anticipate which questions we might want to ask of production systems.</p>
HumioassesstoolsTRUE<p><strong><a href="">Humio</a></strong> is a fairly new player in the log management space. It's been built from the ground up to be super fast at both log ingestion and query using its built-in query language on top of a custom-designed time series database. It integrates with just about everything out there from an ingestion, visualization and alerting perspective. The log management space has been dominated by Splunk and the ELK Stack, so having alternatives is a good thing. We'll be watching Humio's development with interest.</p>
Kubernetes OperatorsassesstoolsTRUE<p>We're excited about the impact <a href="/radar/platforms/kubernetes">Kubernetes</a> has had on our industry but also concerned about the operational complexity that comes with it. Keeping a Kubernetes cluster up and running and then managing packages deployed on it requires special skills and time. Operational processes such as upgrades, migrations, backups, among others, can be a full-time job. We think that <strong><a href="">Kubernetes Operators</a></strong> will play a key role in reducing this complexity. The framework provides a standard mechanism to describe automated operational processes for packages running in a Kubernetes cluster. Although Operators were spearheaded and promoted by RedHat, several community-developed Operators for common open-source packages such as <a href="/radar/tools/jaeger">Jaeger</a>, <a href="/radar/platforms/mongodb">MongoDB</a> and <a href="/radar/platforms/redis">Redis</a> have begun to emerge.</p>
OpenAPMassesstoolsTRUE<p>One of the challenges in adopting an open-source alternative to popular commercial packages is sorting through the complicated landscape of projects to understand which components you need, which ones play nicely together and exactly which part of a total solution each component covers. This is particularly difficult in the world of observability, where the standard practice is to purchase one comprehensive but pricey package to do it all. <strong><a href="">OpenAPM</a></strong> makes the open-source selection process for observability tools easier. It displays the current crop of open-source packages classified by component roles, so you can interactively select compatible components. As long as you keep the tool up to date, it should help you navigate through the confusing array of possible tools.</p>
SystemsassesstoolsTRUE<p>It's easy to think of many of the processes we work within as linear chains of cause and effect. Most of the time we are working within more complex systems where positive and negative feedback loops influence outcomes. <strong><a href="">Systems</a></strong> is a set of tools for describing, executing and visualizing systems diagrams. Using a compact DSL and running either standalone or within a Jupyter Notebook, it's super easy to describe fairly complex processes and the flow of information through them. It's pretty much a niche tool; but an interesting and fun one.</p>
TaurusassesstoolsTRUE<p><strong><a href="">Taurus</a></strong> is a handy application and service performance testing tool written in Python. It wraps many performance testing executors, including <a href="/radar/tools/gatling">Gatling</a> and <a href="/radar/tools/locust">Locust</a>. You can run it from the command line and easily integrate it with continuous delivery pipelines to run performance tests at different stages of the pipeline. Taurus also has great reporting either as console text-based output or integrated with an interactive web UI. Our teams have found that configuring Taurus YAML files is easy because you can use multiple files to describe each test scenario and refer to underlying executer's scenario definitions.</p>
Terraform provider GoCDassesstoolsTRUE<p><strong><a href="">Terraform provider GoCD</a></strong> lets you build pipelines using <a href="/radar/tools/terraform">Terraform</a>, a mature and widely used tool in the <a href="/radar/tools/infrastructure-as-code">infrastructure as code</a> space. With this provider, you can write pipelines in the <a href="">HashiCorp Configuration Language (HCL)</a> that use all of the functionality Terraform provides, including workspaces, modules and remote state. This approach is an excellent alternative to <a href="">Gomatic</a>, which we highlighted in the <a href="/radar/techniques/pipelines-as-code">Pipelines as code</a> blip before. The Golang SDK used in this provider has automatic regression tests for the GoCD API which should minimize issues while upgrading.</p>
TerratestassesstoolsTRUE<p>We widely use <a href="/radar/tools/terraform">Terraform</a> as code to configure a cloud infrastructure. <strong><a href="">Terratest</a></strong> is a Golang library that makes it easier to write automated tests for infrastructure code. A test run creates real infrastructure components (such as servers, firewalls or load balancers), deploys applications on them and validates the expected behavior using Terratest. At the end of the test, Terratest can undeploy the apps and clean up resources. This makes it largely useful for end-to-end tests of your infrastructure in a real environment.</p>
Handwritten CloudFormationholdtoolsTRUE<p><a href="">AWS CloudFormation</a> is a proprietary declarative language to provision AWS infrastructure as code. Handwriting CloudFormation files is often a default approach to bootstrap AWS infrastructure automation. Although this might be a sensible way to start a small project, our teams, and the industry at large, have found that <strong>handwritten CloudFormation</strong> simply does not scale as the infrastructure grows. Noticeable pitfalls of handwritten CloudFormation files for large projects include poor readability, lack of imperative constructs, limited parameter definition and usage, and lack of type checking. Addressing these shortfalls has led to a rich ecosystem of both open-source and custom tooling. We find <a href="/radar/tools/terraform">Terraform</a> a sensible default that not only addresses shortfalls of CloudFormation but also has an active community to add the latest AWS features and fix bugs. In addition to Terraform, you can choose from many other tools and languages, including <a href="/radar/languages-and-frameworks/troposphere">troposphere</a>, <a href="">sceptre</a>, <a href="">Stack Deployment Tool</a> and <a href="/radar/platforms/pulumi">Pulumi</a>.</p>
TRUE<p>Our teams report that <strong><a href="">Apollo</a></strong> has become the library of choice when building a <a href="/radar/languages-and-frameworks/react-js">React</a> application that uses GraphQL to access data from a <a href="/radar/techniques/bff-backend-for-frontends">back-end</a> service. Although the Apollo project also provides a server framework and a GraphQL gateway, the Apollo client gets our attention because it simplifies the problem of binding UI components to data served by any GraphQL backend. Put simply, this means less code needs to be written than using REST backends and redux.</p>
TRUE<p><strong><a href="">MockK</a></strong> is our go-to tool for mocks when writing tests for <a href="/radar/languages-and-frameworks/kotlin">Kotlin</a> applications. We like to use this library because of its first-class support for Kotlin language features such as <a href="">coroutines</a> or lambda blocks. As a native library, it helps our teams to write clean and concise code on testing Kotlin applications instead of using the inconvenient wrappers of Mockito or PowerMock.</p>
TRUE<p><strong><a href="">TypeScript</a></strong>, a statically typed language and superset of JavaScript, has become our sensible default. Large-scale projects benefit most from the type safety. Our developers favor its minimal configuration management, well-integrated IDE support and its ability to refactor code safely and gradually adopt types. With its <a href="">good repository</a> of TypeScript-type definitions at hand, we benefit from all the rich JavaScript libraries while gaining type safety.</p>
Apache Beamtrial
TRUE<p><strong><a href="">Apache Beam</a></strong> is an open-source unified programming model for defining and executing both batch and streaming data parallel processing pipelines. The Beam model is based on the <a href="">Dataflow model</a> which allows us to express logic in an elegant way so that we can easily switch between batch, windowed batch or streaming. The big data-processing ecosystem has been evolving quite a lot which can make it difficult to choose the right data-processing engine. One of the key reasons to choose Beam is that it allows us to switch between different runners — a few months ago <a href="">Apache Samza</a> was added to the other runners it already supports, which include <a href="">Apache Spark</a>, <a href="">Apache Flink</a> and <a href="/radar/platforms/google-cloud-dataflow">Google Cloud Dataflow</a>. Different runners have different capabilities and providing a portable API is a difficult task. Beam tries to strike a delicate balance by actively pulling innovations from these runners into the Beam model and also working with the community to influence the roadmap of these runners. Beam has SDKs in multiple languages including Java, Python and Golang. We've also had success using <a href="">Scio</a> which provides a Scala wrapper around Beam.</p>
TRUE<p><strong><a href="">Formik</a></strong> is a useful higher-order component for making the surprisingly verbose and complex job of handling forms in <a href="/radar/languages-and-frameworks/react-js">React</a> much easier. It localizes state management, assists with submission and optionally uses <a href="">Yup</a> to simplify data validation.</p>
TRUE<p><strong><a href="">HiveRunner</a></strong> is an open-source unit test framework for Apache Hadoop <a href="/radar/languages-and-frameworks/hive">Hive</a> queries based on JUnit4. When writing nontrivial analytics or data pipelines in Hive SQL, we found HiveRunner to be a good enabler for writing tests and even TDDing out some moderately complicated SQL. HiveRunner enables you to write Hive SQL as releasable tested artifacts.</p>
TRUE<p><strong><a href="">joi</a></strong> is a schema description language and validator for JavaScript objects. We like that joi is independent of any web application framework, so our teams can use the same schemas across different stacks. You can also use companion libraries to generate Swagger documentation for APIs that validate requests with joi schemas.</p>
TRUE<p><a href="/radar/languages-and-frameworks/kotlin">Kotlin</a> has demonstrated its value beyond mobile app development. When building microservices and shipping software to production, our teams have had good experiences with Ktor. <strong><a href="">Ktor</a></strong> is a framework that, unlike other web frameworks that support Kotlin, is written in Kotlin, using language features such as <a href="">coroutines</a> which allow for an asynchronous nonblocking implementation. The flexibility to incorporate different tools for logging, DI or a template engine — in addition to its lightweight architecture — makes Ktor an interesting option for creating RESTful services.</p>
TRUE<p><strong><a href="">Laconia</a></strong> is a framework for developing <a href="/radar/platforms/aws-lambda">AWS Lambda</a> functions in JavaScript. As interest and use of serverless tech has grown so has the complexity of the applications being built. Laconia is a small, lightweight framework that takes away some of the rough edges we often encounter. It uses dependency injection to isolate your application code from lower-level AWS APIs and provides adaptors for the different events that your application can respond too. It also plays nicely with the <a href="/radar/tools/serverless-framework">Serverless Framework</a> at deploy time. We like small and simple frameworks and Laconia is just that.</p>
TRUE<p>Much like <a href="/radar/tools/cypress">Cypress</a> and <a href="/radar/tools/testcafe">TestCafe</a>, <strong><a href="">Puppeteer</a></strong> is one of the web UI testing tools garnering praise from our teams. Puppeteer can have fine-grained control over headless browsers, obtain time-trace for performance diagnostics and more. Our teams have found Puppeteer to be stable as well as faster and more flexible than alternatives based on WebDriver.</p>
TRUE<p>We've talked about <strong><a href="">Reactor</a></strong> in the previous Radars. It has continued to gain traction in many of our projects. With the Spring ecosystem embracing Reactor, it has become the dominant implementation of <a href="">Reactive Streams</a>. Reactive systems come with improved scalability and resilience but with increased cost of debugging and a steeper learning curve. For those projects where this tradeoff is acceptable, Reactor has proven to be a good choice. Some of our projects have observed significant improvements in scalability once they moved to Reactor and the rest of the Reactive stack. With <a href="">R2DBC</a> we are starting to get reactive support for RDBMS drivers which addresses one of the weaknesses of reactive services.</p>
TRUE<p><a href=""><strong>Resilience4j</strong></a> is a lightweight fault tolerance library inspired by Netflix <a href="/radar/tools/hystrix">Hystrix</a>. We like its lightweight and modular structure where we pull in specific modules for specific capabilities such as circuit-breaking, rate-limiting, retry, and bulkhead. While <a href="/radar/techniques/service-mesh">service meshes</a> are taking on some of the fault tolerance capabilities, fault tolerance libraries continue to remain a key component of our systems for more nuanced domain-specific fault tolerance behavior and for non-containerized services. With Hystrix going into <a href="">maintenance mode</a>, Resilience4j becomes a default choice in the Java ecosystem. It can work with synchronous APIs as well as reactive ones. It also surfaces metrics to <a href="">dropwizard metrics</a>, <a href="/radar/tools/prometheus">Prometheus</a> and others using additional modules.</p>
TRUE<p><strong><a href="">Room</a></strong> is a persistence library to access SQLite on Android. It makes database access code much simpler, with minimal boilerplate code, and more robust, with compile-time verification of SQL queries. Our developers like its complete integration with observable queries, using <a href="">LiveData</a>. Room is one of the Android <a href="">Jetpack</a> components that were created to make application development on Android easier.</p>
TRUE<p>Since we last featured it on the Radar in January 2015, we've seen steadily increasing interest in <strong><a href="">Rust</a></strong>. Some of our clients are now using Rust, mostly in the context of infrastructure tooling but also in high-powered embedded devices. Interest was fuelled by a growing ecosystem as well as improvements to the language itself. The latter included straightforward performance improvements but also changes that make Rust more intuitive, for example the change to non-lexical scoping. Most of the significant changes are included in the Rust 2018 standard released last December.</p>
TRUE<p><strong><a href="">WebFlux</a></strong> is the Spring Framework implementation of <a href="">Reactive Streams</a>. We see a rise in reactive programming models across our teams in general and the use of WebFlux in teams who are working in the Spring ecosystem. It's best used in large microservices ecosystems where the high performance of the requests is a major concern. It allows overlapping request processing asynchronously without the complications of using multiple threads. WebFlux uses <a href="">Reactor</a> as its reactive library but it is interoperable with other reactive libraries via Reactive Streams. It uses <a href="">Netty</a> as its underlying high-performance communications engine. Although we encourage using Reactive Streams, adopting this programming model requires a significant shift in thinking.</p>
TRUE<p><strong><a href="">Aeron</a></strong> is an efficient and reliable peer-to-peer message transport. It provides a replicated persistent log of messages via a number of media drivers, including HTTP, UDP and TCP. It also supports persistent storage of message streams for later replay. For many applications, Aeron may be overkill because it operates at a pretty low level (OSI Layer 4 conceptually), but it's peer-to-peer design and low (and predictable) latency are useful in a number of use cases. Indeed, we've found it to be useful in certain machine learning applications as well as playing a part in event-driven architectures. We think it's worth pointing out that alternative messaging protocols exist that don't require additional services such as <a href="/radar/tools/apache-kafka">Apache Kafka</a> to be run.</p>
TRUE<p><strong><a href="">Arrow</a></strong> is a functional programming library for <a href="/radar/languages-and-frameworks/kotlin">Kotlin</a>, created by merging two existing popular libraries (<a href="">kategory</a> and <a href="">funKTionale</a>). While Kotlin provides building blocks for functional programming, Arrow delivers a package of ready-to-use higher-level abstractions for application developers. It provides data types, type classes, effects, optics and other functional programming patterns as well as integrations with popular libraries. With Arrow, existing libraries are unified, which should go a long way to avoid fractured communities in this space.</p>
Chaos Toolkitassess
TRUE<p>The <strong><a href="">Chaos Toolkit</a></strong> is one of a number of <a href="/radar/techniques/chaos-engineering">Chaos Engineering</a> tools that made this edition of the Radar. You use the toolkit to describe and then run repeatable experiments on your infrastructure to understand its resilience in the event of failure. Many of our teams have been using homegrown tools to do this, so it's great to see an open-source project dedicated to the practice. The toolkit already has drivers for <a href="/radar/platforms/aws">AWS</a>, <a href="/radar/platforms/azure-service-fabric">Azure Service Fabric</a> and GCE (among others) and plays nicely with build tools which lets you experiment with automation. The usual caveats apply though, Chaos Engineering is a very powerful technique that is best used on resilience-aware systems, that is, systems that have been built to cope with failure. For that reason, we recommend starting using Chaos Toolkit in your nonproduction environments first.</p>
TRUE<p>Data scientists and engineers often use libraries such as <a href="">pandas</a> to perform ad hoc data analysis. Although expressive and powerful, these libraries have one critical limitation: they only work on a single CPU and don't provide horizontal scalability for large data sets. <a href=""><strong>Dask</strong></a>, however, includes a lightweight, high-performance scheduler that can scale from a laptop to a cluster of machines. And because it works with <a href="">NumPy</a>, pandas and <a href="/radar/tools/scikit-learn">Scikit-learn</a>, Dask looks promising for further assessment.</p>
TRUE<p>We've recommended <a href="/radar/languages-and-frameworks/truffle">Truffle</a> for <a href="/radar/techniques/ethereum-for-decentralized-applications">decentralized application</a> (dapp) development in the past. <strong><a href="">Embark</a></strong> too can make your work easier. Embark provides features such as scaffolding, building, testing and debugging and integrates with decentralized storages such as <a href="/radar/platforms/ipfs">IPFS</a>. Through its declarative configuration, you can manage <a href="/radar/techniques/smart-contracts">smart contract</a> configuration, dependencies, artifact and deployment quite easily. Embark's interactive CLI dashboard is also impressive. We keep seeing people use <a href="">Remix</a> to write smart contracts and manually deploy their apps without automated testing, source control management or artifact management. We'd like to draw people's attention to dapp engineering practice by promoting tools such as Truffle and Embark.</p>
TRUE<p><strong><a href="">fastai</a></strong> is an open-source Python library that simplifies training fast and accurate neural nets. It is built on top of <a href="/radar/languages-and-frameworks/pytorch">PyTorch</a> and has become a popular tool for our data scientists. fastai simplifies painful aspects of model training such as preprocessing and loading data down to a few lines of code. It's built on deep learning best practices and has out-of-the-box support for computer vision, natural language processing (NLP) and more. The founders' motivation has been to create an easy-to-use library for deep learning and an improved successor to <a href="/radar/languages-and-frameworks/keras">Keras</a>. <a href="/radar/platforms/google-cloud-platform">GCP</a>, <a href="/radar/platforms/aws">AWS</a> and <a href="/radar/platforms/azure">Azure</a> all have already included fastai in their machine images. The creators of fastai, acknowledging the speed and safety limitations of Python, have announced <a href="">embracing Swift</a> as an alternative language for deep learning. We'll be closely watching this space.</p>
TRUE<p><strong><a href="">http4k</a></strong> is an HTTP toolkit written in pure <a href="/radar/languages-and-frameworks/kotlin">Kotlin</a> for serving and consuming HTTP services. One of the key ideas behind http4k is that HTTP apps are modelled by composing two simple functions — HttpHandler and Filter. They derive inspiration from Twitter's <a href="">"Your Server as a Function"</a> paper. It's very lightweight with the core module having no dependencies apart from Kotlin StdLib. Apart from its elegance and simplicity, we also like its emphasis on testability — given that the entities in the libraries are immutable and the routes in the app, as well as the app itself, are just functions, they're super easy to test. One of the things to be aware of, though, is that we don't have nonblocking or coroutines support in http4k yet.</p>
TRUE<p>With the increasing complexity of single-page JavaScript applications, managing state predictably is becoming more and more important. Immutability can help to ensure our applications behave consistently, but unfortunately JavaScript doesn't natively support the ability to create immutable objects. Libraries such as <a href="/radar/languages-and-frameworks/immutable-js">Immutable.js</a> filled that gap but introduced new problems because now two kinds of objects and arrays existed in the application, the library's version and the native JavaScript ones. <strong><a href="">Immer</a></strong> — German for <em>always</em> — is a tiny package that lets you work with immutable state in a more convenient way. It's based on the copy-on-write mechanism, has a minimal API and operates on normal JavaScript objects and arrays. This means that data access is seamless and no large refactoring efforts are needed when introducing immutability to an existing codebase.</p>
TRUE<p>Given our experience that tests are the only API specifications that really matter, we're always on the lookout for new tools that might help. <strong><a href="">Karate</a></strong> is an API testing framework whose unique feature is that tests are written directly in Gherkin without relying on a general-purpose programming language to implement test behavior. Karate is really a domain-specific language for describing HTTP-based API tests. Although this approach is interesting and makes for some very readable specifications for simple tests, the special-purpose language for matching and validating payloads can become quite syntax-heavy and difficult to understand. It remains to be seen if more complex tests written in this style will be readable and maintainable over the long haul.</p>
TRUE<p><strong><a href="">Micronaut</a></strong> is a new JVM framework for building microservices using Java, <a href="/radar/languages-and-frameworks/kotlin">Kotlin</a> or Groovy. It distinguishes itself through a small memory footprint and short startup time. It achieves these improvements by avoiding runtime reflection for DI and proxy generation, a common shortcoming of traditional frameworks, and instead uses a <a href="">DI</a>/<a href="">AOP</a> container which performs dependency injection at compile time. This makes it attractive not just for standard server-side microservices but also in the context of, for example, the Internet of Things, Android applications and serverless functions. Micronaut uses Netty and has first-class support for reactive programming. It also includes many features that make it cloud-native friendly such as service discovery and circuit breaking. Micronaut is a very promising entrant to the full stack framework for the JVM space and we're keenly watching it.</p>
TRUE<p><a href="/radar/languages-and-frameworks/react-js">React.js</a> has revolutionized the way most people write single-page JavaScript applications. Generally, we recommend you use Create React App throughout the application lifecycle so you don't have to configure your setup, builds and packages manually. But some developers will prefer a tool whose initial defaults reflect a sound set of opinions. <strong><a href="">Next.js</a></strong> is just such an opinionated framework and it is garnering quite a bit of interest among our front-end enthusiasts. Next.js simplifies routing, renders on the server side by default and streamlines dependencies and builds. We're keen to see if it lives up to expectations on our own projects.</p>
TRUE<p><strong><a href="">Pose</a></strong> is a simple CSS-like animation library for <a href="/radar/languages-and-frameworks/react-js">React.js</a>, <a href="/radar/languages-and-frameworks/react-native">React Native</a> and <a href="/radar/languages-and-frameworks/vue-js">Vue.js</a> frameworks. It is a declarative motion system that combines the simplicity of CSS syntax with the power and flexibility of JavaScript animations and interactions.</p>
TRUE<p>As the pace of change in JavaScript frameworks has slowed, our teams have more time to work with specific frameworks and are gaining deeper insights as a result. With <a href="/radar/languages-and-frameworks/react-js">React</a> and the dominant testing framework, <a href="/radar/languages-and-frameworks/enzyme">Enzyme</a>, we've observed a worrying trend of unit tests becoming tightly coupled to implementation details without providing — because the focus is on shallow details — much confidence that features work as expected. These unit tests make evolving the design difficult and they shift too much responsibility up the test pyramid to functional testing. This has made us revisit the idea of <a href="">subcutaneous testing</a>. Additionally, because of its design, <a href="">Enzyme has issues</a> trying to keep up with React's development. All this has pushed us toward assessing <strong><a href="">react-testing-library</a></strong> as a new framework for testing React applications.</p>
TRUE<p><strong><a href="">ReasonML</a></strong> is an interesting new language based on OCaml with a sprinkling of C-like syntax and uses JavaScript as the default compilation target. Created by Facebook, it allows embedded JavaScript snippets and JSX templating with good <a href="/radar/languages-and-frameworks/react-js">React</a> integration. It aims to be approachable for JavaScript developers and leverages that ecosystem, while providing type safety in a functional language.</p>
TRUE<p><strong><a href="">Taiko</a></strong> is a node.js library with a clear and concise API to assist with chrome or chromium browser automation. You can leverage Taiko's smart selectors and write reliable tests as the structure of the web application evolves. There's no need for ID, CSS or XPath selectors or adding explicit waits (for XHR requests) in test scripts. The interactive REPL recorder comes in handy when you want to develop the tests side by side as you explore the functionality. Although you could use Taiko independently, we've had good success using it with <a href="/radar/tools/gauge">Gauge</a>.</p>
TRUE<p>We're strong proponents of <a href="/radar/techniques/polyglot-programming">polyglot programming</a> but recognize that in some cases it can make sense to focus on a single programming language. If you're heavily invested in Swift, most likely because of iOS development, and you find yourself looking for a technology to write server-side services, have a look at <strong><a href="">Vapor</a></strong>, a modern web framework for Swift that has gained a fair amount of popularity.</p>
ContentfuladoptplatformsTRUE<p>Headless content management systems (CMSes) are becoming a common component of digital platforms. <a href=""><strong>Contentful</strong></a> is a modern headless CMS that our teams have successfully integrated into their development workflows. We particularly like its API-first approach and implementation of <a href="">CMS as code</a>. It supports powerful content modeling primitives as code and content model evolution scripts, which allow it to be treated like other data store schemas and enable <a href="">evolutionary database design</a> practices to be applied to CMS development. Its robustness and a stream of new features, including a sandbox environment, have impressed our teams further and made Contentful our default choice in this space.</p>
AWS FargatetrialplatformsTRUE<p><strong><a href="">AWS Fargate</a></strong>, the docker-as-a-service option on <a href="/radar/platforms/aws">AWS</a>, is now widely available across regions. It's a great solution for situations in which teams want to run Docker containers, because <a href="/radar/platforms/aws-lambda">AWS Lambda</a> functions aren't powerful enough, without having to manage EC2 instances or Kubernetes clusters. Our teams report generally positive experiences with Fargate; however, the convenience of this managed service can come at a cost, in financial terms.</p>
EVM beyond EthereumtrialplatformsTRUE<p><a href="">Ethereum Virtual Machine (EVM)</a> was originally designed for the <a href="/radar/platforms/ethereum">Ethereum</a> main network. Nowadays, however, most teams no longer want to reinvent blockchain from scratch; instead, they'd like to take <strong>EVM beyond Ethereum</strong>. We've seen a lot of blockchain teams choose to fork Ethereum (e.g., <a href="/radar/platforms/quorum">Quorum</a>) or implement the EVM spec (e.g., <a href="">Burrow</a>, <a href="">Pantheon</a>), adding their own designs. The intention is to not only reuse the Ethereum design but also leverage its ecosystem and developer community. To many developers, the concept of "smart contract" is almost equivalent to a smart contract written in <a href="/radar/languages-and-frameworks/solidity">Solidity</a>. Although Ethereum itself has some constraints, the technology around the EVM ecosystem is booming.</p>
InfluxDBtrialplatformsTRUE<p><a href="/radar/platforms/time-series-databases">Time series databases</a> (TSDBs) have been around for some time now. But increasingly they're becoming more mainstream as more use cases naturally fit the time series model. <a href=""><strong>InfluxDB</strong></a> continues to remain a good choice for TSDBs with monitoring being one of its key use cases. <a href="/radar/platforms/tick-stack">TICK Stack</a> is an example of a monitoring solution that has InfluxDB at its heart. Influx 2.0 alpha recently introduced Flux — a scripting language for querying and processing time series data. It's still early days for Flux and the jury's out on its <a href="">broader adoption beyond InfluxDB</a>, but it promises to be more powerful and expressive than InfluxQL and enables pushing time series analytic workloads to the database. However, clustering support for InfluxDB is only available with the enterprise version which has limited its adoption on some of our projects.</p>
IstiotrialplatformsTRUE<p><strong><a href="">Istio</a></strong> is becoming the de facto infrastructure to operationalize a <a href="">microservices</a> ecosystem. Its out-of-the-box implementation of cross-cutting concerns — such as service discovery, service-to-service and origin-to-service security, observability (including telemetry and distributed tracing), rolling releases and resiliency — has been bootstrapping our microservices implementations very quickly. It's the main implementation of the <a href="/radar/techniques/service-mesh">service mesh</a> technique we've been using. We've been enjoying its monthly releases and its continuous improvements with seamless upgrades. We use Istio to bootstrap our projects, starting with observability (tracing and telemetry) and service-to-service security. We're closely watching its improvements to service-to-service authentication everywhere in and outside of the mesh. We'd also like to see Istio establish best practices for configuration files to strike a balance between giving autonomy to service developers and control to the service mesh operators.</p>
Kafka StreamstrialplatformsTRUE<p><strong><a href="">Kafka Streams</a></strong> is a lightweight library to build streaming applications. It supports basic streaming APIs such as join, filter, map and aggregate as well as local storage for common use cases such as windowing and sessions. Unlike other stream-processing platforms such as <a href="/radar/platforms/apache-spark">Apache Spark</a> and <a href="">Alpakka Kafka</a>, Kafka Streams has been a good fit for scenarios that don't require large-scale distribution and parallel processing; hence we could get away without yet another piece of infrastructure such as cluster schedulers. Naturally, Kafka Streams has been a good choice when operating in the Kafka ecosystem. Kafka Streams is particularly useful when we have to process data strictly in order and exactly once. One particular use case of Kafka Streams is to build a <a href="">change data capture (CDC)</a> platform.</p>
NomadtrialplatformsTRUE<p>HashiCorp continues to release interesting software. We've featured <a href="/radar/tools/hashicorp-vault">HashiCorp Vault</a> in March 2017, and tools related to Terraform are all over this edition of the Radar. We've moved <strong><a href="">Nomad</a></strong> to Trial because we've had positive experiences using it. While <a href="/radar/platforms/kubernetes">Kubernetes</a> continues to gain traction, we like Nomad's general applicability. It's not just limited to running containerized workloads but can be used to schedule just about anything. Java and Golang are supported natively as well as batch and distributed cron jobs. We like its focus on multi- and hybrid-cloud operations, something likely to become more important to avoid sticky clouds and the fact that it does scheduling well.</p>
CloudEventsassessplatformsTRUE<p>Outside the function code itself, applications written as serverless functions are tightly coupled to the cloud platform on which they're hosted. Although events are a common FaaS-triggering mechanism, and every cloud provider supports them in some form, the current proprietary specifications prevent interoperability across clouds. The <strong><a href="">CloudEvents</a></strong> specification is a burgeoning standard that has been accepted into the <a href="">CNCF Sandbox</a>. The standard is still in active development but several language bindings exist and Microsoft has announced first-class support in <a href="/radar/platforms/azure">Azure</a>. We're hoping other cloud providers will follow suit.</p>
Cloudflare WorkersassessplatformsTRUE<p>Most modern server-side or serverless code execution platforms are centered around containers or VMs. <strong><a href="">Cloudflare Workers</a></strong>, however, takes a different approach to hosting a serverless computing offering. It uses <a href="">V8 Isolates</a>, the open source JavaScript engine developed for Chrome, to run functions as a service (FaaS) on their extensive CDN network. Code can be written in JavaScript or anything that compiles to <a href="/radar/languages-and-frameworks/webassembly">WebAssembly</a> and data can be accessed from Cloudflare's cache or key-value store. The major benefit for developers is performance: by being on the edge network, close to end users, cold-starts take only five milliseconds. For the provider the benefits include both the ability to densely pack isolates because of their lower memory overhead and faster performance through reduced process context switching. This is definitely an intriguing approach to monitor and assess.</p>
DenoassessplatformsTRUE<p>As a group we have mixed feelings about programming in JavaScript on the server side, especially when the rationale for doing so is simply to avoid <a href="/radar/techniques/polyglot-programming">polyglot programming</a>. That said, if you decide to use JavaScript or TypeScript on the server, have a look at <strong><a href="">Deno</a></strong>. Written by Ryan Dahl, the inventor of Node.js, Deno aims to avoid what Ryan considers mistakes that were made in Node.js. It brings a strict sandbox system and built-in dependency and package management, and it supports <a href="/radar/languages-and-frameworks/typescript">TypeScript</a> out of the box. Deno is built using <a href="/radar/languages-and-frameworks/rust">Rust</a> and V8.</p>
Hot ChocolateassessplatformsTRUE<p>The <a href="/radar/languages-and-frameworks/graphql">GraphQL</a> ecosystem and community keep growing. <strong><a href="">Hot Chocolate</a></strong> is a GraphQL server for .NET (core and classic). It lets you build and host schemas and then serve queries against them. The team behind Hot Chocolate has recently added schema stitching which allows for a single entry point to query across multiple schemas aggregated from different locations. Although there are plenty of ways to misuse this approach, it's worth assessing whether to add it to your toolkit.</p>
<p>The <a href="/radar/techniques/serverless-architecture">serverless architecture</a> has popularized a FaaS style of programming among developers; it helps developers focus on solving core business problems with independently built and deployed functions that react to an event, run a business process, produce other events in the process and scale down to zero. Historically, proprietary serverless platforms such as <a href="/radar/platforms/aws-lambda">AWS Lambda</a> or Microsoft <a href="">Azure Functions</a> have enabled this programming paradigm. <strong><a href="">Knative</a></strong> is an open-source Kubernetes-based platform to run FaaS workloads. There are few things that stand out about Knative: it's open source and provider agnostic; it implements the serverless workflow as described in the CNCF Serverless Working Group <a href="">whitepaper</a>; it ensures cross-service interoperability by implementing its eventing interface consistent with <a href="">CNCF CloudEvents</a> specification; and, most importantly, it addresses a common challenge of operating a harmonized and yet hybrid FaaS and long-running container-based architecture. It easily integrates with both <a href="/radar/platforms/istio">Istio</a> and <a href="/radar/platforms/kubernetes">Kubernetes</a>. For example, developers can take advantage of roll-out strategies that Istio implements by traffic splitting between different revisions of the functions. Developers can take benefit of Istio-provided observability not only for long-running container services but also for FaaS programs in the same Kubernetes environment. We anticipate that Knative open-source eventing interface will continue to enable new underlying source and destination event integrations.</p>
MinIOassessplatformsTRUE<p>Object storage is a popular choice for storing unstructured data and in a few cases structured data in the cloud. We do discourage the use of <a href="/radar/techniques/generic-cloud-usage">generic cloud</a> but if you want to minimize the risk of cloud stickiness for object storage, we've found <strong><a href="">MinIO</a></strong> quite helpful. With an S3-compatible API layer, MinIO abstracts object storage across cloud providers, including <a href="/radar/platforms/aws">AWS</a>, <a href="/radar/platforms/azure">Azure</a> and <a href="/radar/platforms/google-cloud-platform">Google Cloud Platform</a> (GCP), and we've used it successfully in products with flexible target infrastructures from data centers to cloud providers.</p>
ProphetassessplatformsTRUE<p>Even in the era of deep learning, statistical models still play a role in business decision support. Time series models are widely used to forecast inventories, demand, customer traffic, and so on. Hand-crafting these models so that they're robust and flexible has typically been the role of either specialized statisticians or large commercial software vendors. <strong><a href="">Prophet</a></strong> is an open-source alternative to commercial forecasting packages that can be programmed in R or Python. Facebook claims to use Prophet internally for business forecasting at scale and has made it available as an open-source package for anyone to use. We like that Prophet removes some of the tedium of model construction, maintenance and data manipulation so that human analysts and subject matter experts can focus on doing what they do best.</p>
QuorumassessplatformsFALSE<p><strong><a href="">Quorum</a></strong> is "an enterprise-focused version of Ethereum" that aims to provide network permissioning and transaction privacy as well as higher performance. One of our teams has worked deeply with Quorum; however, their experience so far hasn't been great. Some challenges result from <a href="/radar/techniques/smart-contracts">complex smart contract programming</a> and some come from Quorum itself. For example, it doesn't work well with load balancers and only has partial database support, which will lead to significant deployment burden. We faced some stability and compatibility issues especially on private transactions. Quorum recently attracted a lot of attention because of <a href="">JPM Coin</a>. However, from a tech perspective, we recommend being cautious when implementing Quorum while keeping an eye on its development.</p>
SPIFFEassessplatformsFALSE<p><strong><a href="">SPIFFE</a></strong> standardization of service identity has been an important step in enabling turnkey solutions for end-to-end encryption and mutual authentication between services. The SPIFFE standards are backed by the OSS <a href="">SPIFFE Runtime Environment (SPIRE)</a>, which automatically delivers cryptographically provable identities to services. <a href="/radar/platforms/istio">Istio</a> also uses SPIFFE by default. SPIFFE enables many use cases, including identity translation, OAuth client authentication, mTLS "encryption everywhere" and workload observability. ThoughtWorks is actively working with the Istio and SPIFFE communities to bridge the gap between legacy service identity providers and SPIFFE-based identities so that mTLS can be used everywhere between services, inside a service mesh and outside.</p>
TendermintassessplatformsTRUE<p><a href="">Byzantine fault tolerance (BFT)</a> is one of the fundamental problems in cryptocurrency and blockchain systems. It requires overall system agreement on a single data value in the presence of a number of arbitrary faulty processes, which includes malicious fraud. <strong><a href="">Tendermint</a></strong> is a BFT state machine replication engine that lets you implement your own blockchain systems. The consensus engine, Tendermint Core, takes over the peer-to-peer communication and consensus part, you just need to implement the rest of the application (e.g., construct transaction and verify cryptographic signature) and communicate with Tendermint Core through <a href="">ABCI</a>. Some blockchain <a href="">implementations</a> have already chosen Tendermint as their consensus engine.</p>
TimescaleDBassessplatformsTRUE<p>In previous Radars we've discussed <a href="/radar/platforms/postgresql-for-nosql">PostgreSQL for NoSQL</a>. PostgreSQL's maturity and extensibility have led to a steady stream of innovative persistence stores built on the Postgres engine. One that caught our attention is <strong><a href="">TimescaleDB</a></strong>, a database that allows fast writes and optimized queries over time-series data. Albeit not (yet) as full-featured as <a href="/radar/platforms/influxdb">InfluxDB</a>, TimescaleDB offers an alternative data model and querying capability. You should evaluate TimescaleDB if you have modest scalability needs, prefer to use SQL and appreciate the stability and familiar administrative interface that PostgreSQL offers.</p>
Four key metricsadopttechniquesTRUE<p>The thorough <a href="">State of DevOps</a> reports have focused on data-driven and statistical analysis of high-performing organizations. The result of this multiyear research, published in <a href="">Accelerate</a>, demonstrates a direct link between organizational performance and software delivery performance. The researchers have determined that only <strong>four key metrics</strong> differentiate between low, medium and high performers: lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage. Indeed, we've found that these four key metrics are a simple and yet powerful tool to help leaders and teams focus on measuring and improving what matters. A good place to start is to instrument the build pipelines so you can capture the four key metrics and make the software delivery value stream visible. <a href="">GoCD pipelines,</a> for example, provide the ability to measure these four key metrics as a first-class citizen of the <a href="">GoCD analytics</a>.</p>
Micro frontendsadopttechniquesTRUE<p>We've seen significant benefits from introducing <a href="">microservices</a>, which have allowed teams to scale the delivery of independently deployed and maintained services. Unfortunately, we've also seen many teams create a frontend monolith — a large, entangled browser application that sits on top of the backend services — largely neutralizing the benefits of microservices. Since we first described <strong>micro frontends</strong> as a technique to address this issue, we've had almost universally positive experiences with the approach and have found a number of patterns to use micro frontends even as more and more code shifts from the server to the web browser. So far, <a href="/radar/platforms/web-components-standard">web components</a> have been elusive in this field, though.</p>
Opinionated and automated code formattingadopttechniquesTRUE<p>For as long as we can remember, what style to use for formatting code has been a matter of personal taste, company policy and heated debate. Finally, the industry appears to be tiring of this endless argument and teams are freeing up surprisingly large amounts of time by forgoing these discussions and just adopting <strong>opinionated and automated code formatting</strong> tools. Even if you don't agree 100% with the opinions of the various tools, the benefits of focusing on what your code does rather than how it looks is something most teams should be able to get behind. <a href="/radar/tools/prettier">Prettier</a> has been getting our vote for JavaScript, but similar tools, such as <a href="">Black</a> for Python, are available for many other languages and are increasingly being built-in as we see with <a href="">Golang</a> and <a href="">Elixir</a>. The key here is not to spend hours discussing which rules to enforce, but instead pick a tool that is opinionated, minimally configurable and automated — ideally as a pre-commit hook.</p>
Polyglot programmingadopttechniquesTRUE<p>We put <strong>polyglot programming</strong> on Trial in one of our first Radars to suggest that choosing the right language for the job could significantly boost productivity, and there were new language entrants that were worthy of consideration. We want to reraise this suggestion because we're seeing a new push to standardize language stacks by both developers and enterprises. While we acknowledge that placing no restrictions on language uses can create more problems than it solves, promoting a few languages that support different ecosystems or language features is important for both enterprises to accelerate processes and go live more quickly and developers to have the right tools to solve the problem at hand.</p>
Secrets as a serviceadopttechniquesTRUE<p>Humans and machines use secrets throughout the value stream of building and operating software. The build pipelines need secrets to interface with secure infrastructures such as container registries, the applications use API keys as secrets to get access to business capabilities, and the service-to-service communications are secured using certificates and keys as secrets. You can set and retrieve these secrets in different ways. We've long cautioned developers about using source code management for storing secrets. We've recommended <a href="/radar/techniques/decoupling-secret-management-from-source-code">decoupling secret management from source code</a> and using tools such as <a href="/radar/tools/git-secrets">git-secrets</a> and <a href="/radar/tools/talisman">Talisman</a> to avoid storing secrets in the source code. We've been using <strong>secrets as a service</strong> as a default technique for storing and accessing secrets. With this technique you can use tools such as <a href="/radar/tools/hashicorp-vault">Vault</a> or <a href="">AWS Key Management Service (KMS)</a> to read/write secrets over an HTTPS endpoint with fine-grained levels of access control. Secrets as a service uses external identity providers such as <a href="">AWS IAM</a> to identify the actors who request access to secrets. Actors authenticate themselves with the secrets service. For this process to work, it's important to automate bootstrapping the identity of the actors, services and applications. Platforms based on <a href="/radar/platforms/spiffe">SPIFFE</a> have improved the automation of assigning identities to services.</p>
Chaos EngineeringtrialtechniquesFALSE<p>In the last year we've seen <strong>Chaos Engineering</strong> move from a much talked-about idea to an accepted, mainstream approach to improving and assuring distributed system resilience. As organizations large and small begin to implement Chaos Engineering as an operational process, we're learning how to apply these techniques safely at scale. The approach is definitely not for everyone, and to be effective and safe, it requires organizational support at scale. Industry acceptance and available expertise will definitely increase with the appearance of commercial services such as <a href="/radar/tools/gremlin">Gremlin</a> and deployment tools such as <a href="/radar/tools/spinnaker">Spinnaker</a> implementing some Chaos Engineering tools.</p>
Container security scanningtrialtechniquesTRUE<p>The container revolution around <a href="/radar/platforms/docker">Docker</a> has massively reduced the friction in moving applications between environments, fueling increased adoption of continuous delivery and continuous deployments. The latter, especially, has blown a rather large hole in the traditional controls over what can go to production. The technique of <strong>container security scanning</strong> is a necessary response to this threat vector. Tools in the build pipeline automatically check containers flowing through the pipeline against known vulnerabilities. Since our first mention of this technique, the tool landscape has matured and the technique has proven useful on development efforts with our clients.</p>
Continuous delivery for machine learning (CD4ML) models
trialtechniquesTRUE<p><strong>Continuous delivery for machine learning (CD4ML) models</strong> apply continuous delivery practices to developing machine learning models so that they are always ready for production. This technique addresses two main problems of traditional machine learning model development: long cycle time between training models and deploying them to production, which often includes manually converting the model to production-ready code; and using production models that had been trained with stale data.</p>

<p>A continuous delivery pipeline of a machine learning model has two triggers: (1) changes to the structure of the model and (2) changes to the training and test data sets. For this to work we need to both version <a href="/radar/techniques/versioning-data-for-reproducible-analytics">the data sets</a> and the model's source code. The pipeline often includes steps such as testing the model against the test data set, applying automatic conversion of the model (if necessary) with tools such as <a href="">H2O</a>, and deploying the model to production to deliver value.</p>
Crypto shreddingtrialtechniquesFALSE<p>Maintaining proper control over sensitive data is difficult, especially when it's copied outside of a master system of record for backup and recovery purposes. <strong>Crypto shredding</strong> is the practice of rendering sensitive data unreadable by deliberately overwriting or deleting encryption keys used to secure that data. Considering there are systems, such as audit application or blockchain, that should not or could not delete historical records, this technique is quite useful for privacy protection and <a href="">GDPR</a> compliance.</p>
Infrastructure configuration scannertrialtechniquesTRUE<p>For some time now we've recommended that delivery teams take ownership of their entire stack, including infrastructure. This means increased responsibility in the delivery team itself for configuring the infrastructure in a safe, secure and compliant way. When adopting cloud strategies, most organizations default to a tightly locked-down and centrally managed configuration to reduce risk, but this also creates substantial productivity bottlenecks. An alternative approach is to allow teams to manage their own configuration and use an <strong>infrastructure configuration scanner</strong> to ensure the configuration is safe and secure. Options include open-source scanners such as <a href="">prowler</a> for <a href="/radar/platforms/aws">AWS</a> and <a href="/radar/tools/kube-bench">kube-bench</a> for <a href="/radar/platforms/kubernetes">Kubernetes</a> installations. For more continuous detection, take a look at cloud platforms such as AWS Config Rules among other commercial services.</p>
Service meshtrialtechniquesTRUE<p><strong>Service mesh</strong> is an approach to operating a secure, fast and reliable microservices ecosystem. It has been an important stepping stone in making it easier to adopt microservices at scale. It offers discovery, security, tracing, monitoring and failure handling. It provides these cross-functional capabilities without the need for a shared asset such as an API gateway or baking libraries into each service. A typical implementation involves lightweight reverse-proxy processes, aka sidecars, deployed alongside each service process in a separate container. Sidecars intercept the inbound and outbound traffic of each service and provide cross-functional capabilities mentioned above. This approach has relieved the distributed service teams from building and updating the capabilities that the mesh offers as code in their services. This has lead to an even easier adoption of <a href="/radar/techniques/polyglot-programming">polyglot programming</a> in a microservices ecosystem. Our teams have been successfully using this approach with open source projects such as <a href="/radar/platforms/istio">Istio</a> and we will continue to monitor other open service mesh implementations such as <a href="">Linkerd</a> closely.</p>
Ethical OSassesstechniquesTRUE<p>As developers at ThoughtWorks we're acutely aware of the ethics of the work we do. As society becomes ever more reliant on technology, it's important that we consider ethics when making decisions as software development teams. Several toolkits have emerged that can help us think through some of the future implications of the software we're building. They include <a href="">Tarot Cards of Tech</a> and <strong><a href="">Ethical OS</a></strong>, which we've had good feedback on. Ethical OS is a thinking framework and a set of tools that drive discussions around the ethics of building software. The framework is a collaboration between the Institute for the Future and the Tech and Society Solutions Lab. It's based on a practical set of risk zones, such as addiction and the <a href="">dopamine economy</a>, plus a number of scenarios to drive conversation and discussion.</p>
Smart contractsassesstechniquesTRUE<p>The more experience we gain with using distributed ledger technologies (DLTs), the more we encounter the rough edges around the current state of <strong><a href="">smart contracts</a></strong>. Committing automated, irrefutable, irreversible contracts on ledger sounds great in theory. The problems arise when you consider how to use modern software delivery techniques to developing them, as well as the differences between implementations. Immutable data is one thing, but immutable business logic is something else entirely! It's really important to think about whether to include logic in a smart contract. We've also found very different operational characteristics between different implementations. For example, even though contracts can evolve, different platforms support this evolution to a greater or lesser extent. Our advice is to think long and hard before committing business logic to a smart contract and to weigh the merits of the different platforms before you do.</p>
Transfer learning for NLPassesstechniquesTRUE<p>Transfer learning has been quite effective within the field of computer vision, speeding the time to train a model by reusing existing models. Those of us who work in machine learning are excited that the same techniques can be applied to natural language processing (NLP) with the publication of <a href="">ULMFiT</a> and open source pretrained models and code examples. We think <strong>transfer learning for NLP</strong> will significantly reduce the effort to create systems dealing with text classification.</p>
Wardley mappingassesstechniquesTRUE<p>We're usually wary of covering diagrammatic techniques, but <strong><a href="">Wardley mapping</a></strong> is an interesting approach to start conversations around the evolution of an organization's software estate. At their simplest, they're used to visualize the value chains that exist within an organization, starting with customers' needs and progressively plotting the different capabilities and systems used to deliver on those needs along with the evolution of those capabilities and systems. The value of this technique is the process of collaborating to create the maps rather than the artefact itself. We recommend getting the right people in the room to produce them, and then treat them as living, evolving things rather than a complete artefact.</p>
Productionizing Jupyter NotebooksholdtechniquesTRUE<p><a href="/radar/tools/jupyter">Jupyter Notebooks</a> have gained in popularity among data scientists who use them for exploratory analyses, early-stage development and knowledge sharing. This rise in popularity has led to the trend of <strong>productionizing Jupyter Notebooks</strong>, by providing the tools and support to execute them at scale. Although we wouldn't want to discourage anyone from using their tools of choice, we don't recommend using Jupyter Notebooks for building scalable, maintainable and long-lived production code — they lack effective version control, error handling, modularity and extensibility among other basic capabilities required for building scalable, production-ready code. Instead, we encourage developers and data scientists to work together to find solutions that empower data scientists to build production-ready machine learning models using <a href="/radar/techniques/continuous-delivery-for-machine-learning-cd4ml-models">continuous delivery</a> practices with the right programming frameworks. We caution against productionization of Jupyter Notebooks to overcome inefficiencies in continuous delivery pipelines for machine learning, or inadequate automated testing.</p>
Puncturing encapsulation with change data captureholdtechniquesTRUE<p><a href="">Change data capture</a> (CDC) is a very powerful technique for pulling database changes out of a system and performing some actions on that data. One of the most popular ways of doing this is to use the database's transaction log to identify changes and then publish those changes directly onto an event bus that can be consumed by other services. This works very well for use cases such as <a href="">breaking monoliths into microservices</a> but when used for first-class integration between microservices, this leads to puncturing encapsulation and leaking the source service's data layer into the event contract. We've talked about <a href="/radar/techniques/domain-scoped-events">domain scoped events</a> and other techniques that emphasize the importance of having our events model our domain properly. We're seeing some projects use CDC for publishing row-level change events and directly consuming these events in other services. This <strong>puncturing of encapsulation with change data capture</strong> can be a slippery slope leading to fragile integrations and we would like to call this out with this blip.</p>
Release trainholdtechniquesTRUE<p>We've seen organizations successfully move from very infrequent releases to a higher cadence by using the <strong>release train</strong> concept. The release train is a technique for coordinating releases across multiple teams or components that have runtime dependencies. All releases happen on a fixed and reliable schedule regardless of whether all expected features are ready (the train doesn't wait for you — if you miss it you wait for the next one). Although we wholeheartedly endorse discipline around regularly releasing and demoing working software, we've experienced serious drawbacks with the approach over the medium to long term as it reinforces temporal coupling around sequencing of changes and can degrade quality as teams rush to complete features. We prefer to focus on the architectural and organizational approaches necessary to support independent releases. Although the train can be a useful forcing function for speeding up slower teams, we've also seen it as imposing an upper limit on how quickly faster-moving teams can move. We believe that it is a technique that should be approached with a good degree of caution, if at all.</p>
Templating in YAMLholdtechniquesTRUE<p>As infrastructures grow in complexity, so do the configuration files that define them. Tools such as <a href="">AWS CloudFormation</a>, <a href="/radar/platforms/kubernetes">Kubernetes</a> and <a href="/radar/tools/helm">Helm</a> expect configuration files in JSON or YAML syntax, presumably in an attempt to make them easy to write and process. However, in most cases, teams quickly reach the point where they have some parts that are similar but not quite the same, for example, when the same service must be deployed in different regions with a slightly different setup. For such cases tools offer <strong>templating in YAML</strong> (or JSON), which has caused a huge amount of <a href="">frustration with practitioners</a>. The problem is that the syntax of JSON and YAML requires all sorts of awkward compromises to graft templating features such as conditionals and loops into the files. We recommend using an API from a programming language instead or, when this is not an option, a templating system in a programming language, either a general-purpose language such as Python or something specialized such as <a href="">Jsonnet</a>.</p>