3 of 19

www.sealights.io

Overall Coverage vs. Modified Coverage

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

4 of 19

www.sealights.io

Prioritize test development only on the riskiest areas and avoid redundant or inefficient test development and maintenance

Smart Automated Test Planning

Change Based Test Development Strategy

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

5 of 19

www.sealights.io

2023 Use Case | Fortune 100 Accounting Firm

Tech Stack: node.js | Build frequency: Multiple times daily | Sprint length: Biweekly | Release frequency: Biweekly

Effective & Efficient Test Automation Project with Test Gap Analysis�

The team focused their test automation planning and development on areas where code was changed and was not tested (test gaps).

Eliminated Production Defects Utilizing E2E Code Coverage Analysis�

The team achieved 100% Code Changes coverage and eliminated quality risks. As a result, there were zero production defects, ensuring a high-quality product for their clients.

Team Modified Coverage & Production Defects Progress using SeaLights

3 months production defect free record

6 of 19

www.sealights.io

�Identified 2 Untested Code Changes

�

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

7 of 19

www.sealights.io

Identify code changes that are not tested in every build

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

8 of 19

Inside the Platform: Introduction to Test

Impact Analytics (TIA)

Step 1:

Code <> Test Mapping

SeaLights maps code to test via Bytecode instrumentation

Name of Presentation

9 of 19

Step 2:

Code Changes Capturing

Analyzes every build to identify newly added code and code changes

Inside the Platform: Introduction to Test

Impact Analytics (TIA)

Name of Presentation

10 of 19

Step 3:

Executing Only Relevant Tests

Executed only the relevant tests based on code changes

Inside the Platform: Introduction to Test

Impact Analytics (TIA)

Name of Presentation

11 of 19

www.sealights.io

Name of Presentation

12 of 19

SeaLights in Action: How Sophos Accelerated Their CI Tests Cycle Time By 88% with SeaLights

97% savings on compute hours and test executions while maintaining the highest level of quality.

318 hours saved in compute time in a sprint

4M Tests skipped!

Name of Presentation

13 of 19

Challenge and Motivation | Shift Left Testing

Engineering teams want to Shift Left their E2E tests to the dev feature branch to identify and fix bugs before they reach integration branches
The testing cycle, which can run between 30 minutes to a few hours, may slow down the development velocity.
Running tests during Pull Request cycles can be expensive (AWS costs)

Develop

Deploy

Merge

Review

Slow & Expensive Testing

Executing ALL tests

at a later stage

Master Branch

Feature Branch

Name of Presentation

14 of 19

Shift Left is Not Enough

Running tests earlier can still be time-consuming and costly. Test Impact Analytics is a necessary addition that can effectively identify which tests to run based on code changes, significantly reduce test execution time.

Deploy

Develop

Merge

Review

Master Branch

Feature Branch

Shifting Left is Not Sufficient,

still Executing All tests

Name of Presentation

15 of 19

www.sealights.io

Testing cycle time will be shorter and focused to the PR change
Unrelated flaky tests will not run (skipped)
Failing fast and fixing bugs early in the development cycle will increase development velocity

Test Impact Analytics for Pull Request

Develop

Review

Execute Only

Relevant E2E Tests

Merge

Deploy

Name of Presentation

16 of 19

www.sealights.io

Name of Presentation

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

17 of 19

www.sealights.io

Name of Presentation

Observability helps optimize quality by enabling teams to detect and diagnose issues faster. This can help reduce mean time to resolution (MTTR) and increase uptime, which ultimately leads to better quality for end-users. By monitoring key metrics such as error rates, latency, and throughput, teams can identify trends and make data-driven decisions to improve quality.
Observability also helps optimize release speed by providing real-time feedback on code changes. By monitoring deployment metrics such as success rates, error rates, and rollback rates, teams can make informed decisions on when and how to release new features. This can help reduce the risk of failed deployments and enable teams to release more frequently with confidence.
In addition, observability helps teams identify and prioritize areas for improvement. By analyzing system behavior and user behavior, teams can identify areas of the system that may be causing issues or slowing down performance. This can help teams focus on optimizing these areas and improving overall system performance.
Finally, observability can help improve collaboration across teams. By having a shared understanding of system behavior, teams can work together more effectively to resolve issues and optimize performance. This can help break down silos and create a culture of continuous improvement.

18 of 19

www.sealights.io

It is not necessary to run all tests at all times
An effective risk-based mode of testing allows you to run only tests that have impact on the change
Shorten test execution time for both automated and manual testing
Reduce long feedback loops for efficient hotfixing process
Reduce the cost of long & expensive manual and automated testing cycles
Reduce long waiting time, as well as troubleshooting irrelevant failed tests

Advancing Shift-Left Testing with Test Impact Analytics

1 of 19

2 of 19

3 of 19

4 of 19

5 of 19

6 of 19

7 of 19

8 of 19

9 of 19

10 of 19

11 of 19

12 of 19

13 of 19

14 of 19

15 of 19

16 of 19

17 of 19

18 of 19

19 of 19