Testability and Built-In Test in Complex Systems

בדיקתיות במערכות מורכבות

1

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Embedded systems

We notice our dependency on electronics only when it suddenly stops working

2

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Why Testing and Built-in Test is important?

We depend on computer systems too much, and on the technical systems controlled by computers

3

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Today’s cars contain an average of 1,400 semiconductors

Exponential growth of electronics share in consumer products

4

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Why thinking about Testability is important?

Test is costly and still produce no value, except trust

The cost of fault diagnosis is about 20-25% from whole cost of the car

Automotive electronics cost as a percentage of total car cost

worldwide from 1950 to 2030

5

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Why Design for Testability is important?

The main property of today’s systems is COMPLEXITY

To manage the complexity we have to know methods like:

• Abstraction
• Modeling
• Simulation
• Hierarchical “divide and conquer”

6

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

• Testability determines the ability to detect system failures
• Maintenance depends largely on testability

Multiple test equipment

Smart BIT

Multiple Test Equipment vs Smart BIT

7

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

The problem: Quality vs Money?

8

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Designer to test engineer: Check if my implemented functions are working

Test engineer answer:

Redesign it to be testable

Design for Testability - Different Approach

Design for Testability

9

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Requirements to Design (Flowchart)

10

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT (Built-In Test)

A Built-In Test (BIT) is a mechanism that permits a machine to test itself

​

BIT has two primary functions:

• To monitor the "general well-being" of the hardware and inform the operator of any malfunction
• To aid in the location of failed components

​

​

11

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT (Built in Test)

Engineers design BIT to meet requirements such as:

Lower repair cycle times or constraints such as:

• limited technician involvement
• cost of testing during manufacture

​

​

The main purpose of BIT is to reduce the checks complexity, and thereby decrease the cost and reduce reliance upon external test equipment.

​

12

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Test (BIT) design goals

BIT design goals:

• Will be simpler and faster
• Reducing the effort to perform tests
• Reducing testing time
• Reducing test equipment costs
• Improving product quality

13

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Test (BIT) Limitations

​

BIT Limitations:

• Additional cost in development
• Additional Hardware (Up to 20%)
• An increase in the complexity of the design
• Naturally does not cover Mechanical aspects
• Possible availability impact

​

14

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT Design Philosophy

BIT must be an integral part of the design

​

BIT often means additional hardware and software above and beyond that required for the primary function

​

The goal of BIT design is to decrease the mean-time-to-repair (MTTR) by directing a technician to the faulty component as quickly as possible

15

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT Three-layer approach

Power-up BIT (PBIT) A specific type of initiated BIT which is exercised each time power is applied to the unit or system.

Continuous BIT (CBIT): A type of BIT which continually monitors system operation for errors.

Initiated BIT (IBIT): A type of BIT which is executed only after the occurrence of an external event such as an action by an operator.

​

16

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT in Aviation

Almost all avionics have now incorporate BIT.

​

In avionics the purpose is:

To isolate failing line-replaceable units, which are then removed and repaired at:

• On Site
• Depots
• Manufacturer

​

BIT

17

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT in Aviation

Commercial Aircraft only make money when they fly, so they use BIT to minimize the time on the ground needed for repair and to increase the level of safety of the system which contains BIT.

​

Military Aircraft

When BIT is used in flight, a fault causes the system to switch to an alternative mode or equipment that still operates. Critical flight equipment is normally duplicated, or redundant.

​

BIT

18

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

False Alarm

Increasing system complexity

Problem of false alarms in electronic monitoring systems

​

Some systems exhibit as many as 40% or more “false pulls”

​

A large volume of wasted or ineffective maintenance actions

​

What is a false alarm?

• MIL-STD-2165 defines a false alarm as a fault indicated by built-in test (BIT) or other monitoring circuitry where no fault exists.’
• The RADC Testability Notebook defines false alarm as an indication of a fault where no fault exists such as operator error, transient condition, BIT design deficiency.

19

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

False Alarm

Two principal effects of False Alarms:

• Increased maintenance because of diagnosis being performed on otherwise healthy systems
• Decreased mission effectiveness because we ignore indications that we think are false alarms (Some of these may be real failures.)

​

What then, is a False Alarm?

A fault indication that triggers a maintenance action where no fault exists.

​

There is definition that is based on a maintenance event:

20

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

False-Alarm Causes

False-alarm signals typically happen from three causes:

• A logic element has failed
• Sufficient noise has entered a circuit
• The logic is improperly cooled.

21

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

System Complexity & System Approach BIT

The need for specifying ROBUST Built-In Test for systems is growing as systems become more Complex.

​

Detection-only BIT insufficient to meet the needs of System Test Localization and Isolation is becoming essential.

22

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Categorization Of BIT

BIT can be categorized in several different ways, such as:

• Functional Levels tested
• Purpose (Detection, Isolation, Correction, Prediction).
• Active or Passive
• On-line or Off-line, or Interleaved
• Inductive or Deductive
• Centralized or Decentralized
• Hardware or Software

23

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Concept and Requirements for Testability

Input data for the Testability

• Operational and Performance Requirements.
• Reliability Assessment.
• Testability requirements and Human engineering.

​

Output data for the Testability

• Assessment of testability
• Models and Test Analysis
• Recommendations for Improvement

24

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Parameters To Be Monitored

Electronic circuit test parameters include:

• Voltage sources
• Current sources
• Standard deviations of node voltages
• Time
• Frequency values
• Impedances, currents, and voltages
• Branch voltages and currents
• Word size
• Element currents, voltage, and power losses
• Nodal voltage sensitivities
• Resistances
• Mutual inductances

​

​

25

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Parameters To Be Monitored

Digital parameters can be monitored as:

• Open-stuck at "1"
• Short-stuck at "0"

Analog-circuit test parameters consist of the same parameters as those of digital circuits plus volume flow, pressure (hydraulic), torque, angular velocity, displacement (rotational), force, velocity (translational), temperature, stress, and strain.

Analog parameters can be monitored as:

• Voltage
• Time,
• Ratio,
• Tolerance,
• Frequency,
• Power

26

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Aircraft Typical Preflight BIT

Typical BIT Check:

• CPU
• ROM
• RAM
• IOC
• MUX
• Reference Voltage
• Analog Addressing
• Analog Input
• Watch Dog Timer
• Discrete I/O
• Failure Logic
• Sensor

27

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Data Used In BIT Analysis

Reliability Prediction

Reliability Prediction is necessary to determine the component failure rate, calculate the system MTBF, and ensure suitability of components for the design under consideration.

​

Part-failure rates can be derived from MIL-HDBK 217.

​

BIT analysis depends upon the failure rates of individual components within the system.

MIL-HDBK 217F

28

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Data Used In BIT Analysis

Failure Mode Effect And Criticality Analysis

Assists the designer to assure that malfunctions will not affect a system and helps to determine the exact level of BIT

​

Degraded modes of operation are described so that acceptable degraded modes are not classified as failures.

29

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Fault Isolation Resolution (FIR)

Fault Isolation Resolution (FIR) can be defined as percent of faults or failures that BIT system will isolate to a specified level (for example, to 1 LRU, 2 LRU, 3 LRU…).

​

​

​

30

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Testability Prediction process

• Performing Failure Rate Analysis and FMECA
• Analysis of the level of Detection and Isolation for each failure
• Calculate the level of testability relative to all possible failures by weighted failure rate.

31

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Health and Usage monitoring systems (HUMS)

The HEALTH MONITORING management system has prominent function to improve the security and dependability of the aircraft, can also shorten the maintain cycle

• This technology predicts the failure situation of system
• Utilizes the multi-sensor information integration technology to diagnose the systematic failure;
• Help the equip users to manage and make policy base on the available resource and the demand of using during in the maintenance.

32

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Health and Usage monitoring systems (HUMS)

The health monitoring management system of the aircraft is based on:

• Wireless sensor network technology,
• Build-in-Testing technology
• Ultra broadband communication
• Analysis the data resources relevant to the operation
• Maintain of the aircraft through the data mining technological.

​

​

33

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Health and Usage monitoring systems (HUMS)

Damage and little crackle:

• Chemical corrosion
• Stress function
• Struck
• Tired factor of heat influences

​

The structure is destroyed, even cause the serious accident.

​

​

​

​

34

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Aircraft Health and Usage monitoring systems (HUMS)

In aviation, every year we observe an increasing in the percentage of health systems for aircraft

​

35

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT and Fault Isolation

Examples

36

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT coverage�Hydraulic actuator vs Electrical Actuator

Hydraulic actuator:

• Greater Power with small dimensions
• Lots of experience on lot of projects
• Reliable technology
• No Built In Test capability in non operating mode

Influence on BIT coverage

Electrical actuator

• Weaker than hydraulic actuator
• Has Build in Test capability in operating & non operating mode

​

37

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

BIT coverage �System with Over Voltage protection

Over Voltage protection is for emergency situation in order to protect the system

But if it has not BIT capability, its Latent Failure.

This is Safety Issue and required a hard time replacement at a specific interval based on reliability data.

​

• If Initiated BIT is available, a preventive maintenance task is required, but is very simple (test switch)
• If Continues BIT is available, there no need for preventive task (no latent failure)

​

38

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Flight termination System (FTS)

Flight Termination Systems provide redundant capability for flight termination of missiles or UAV if they go astray.

​

FTS must have high reliability – if there is any reasonable chance a system may fail to operate then the flight should be aborted.

One way to provide assurance that such systems are working correctly is to monitor all data from the FTS.

39

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Iron Dome

• Reliability
• Availability
• BIT

40

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Medical Devices Diagnostic Reliability

41

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Medical Devices Diagnostic Reliability

The failure of one part or component of a medical device can lead to system failure and may result in patient injury or death.

​

For example, if a feedback mechanism in a therapeutic medication delivery system fails, a patient may receive incorrect or even lethal doses.

​

42

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Boeing 737 MAX Sensor

Boeing says that it has redesigned the Angle of Attack (AoA) sensors of the system over the past several months. "Going forward," it vowed, "MCAS will compare information from both AoA sensors before activating, adding a new layer of protection."

Previously, the system could be actuated with information from just one of the sensors. Boeing said it's also updated crew manuals and pilot training to "ensure every pilot has all of the information they need to fly the 737 MAX safely.“

​

43

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Aircraft Health and Usage monitoring systems (HUMS)

US Navy:

• Make the maintenance cycle lengthen 20% to maintain aircraft
• Engine overhaul issue extend to 2400h from 1200h;

​

By the Health Monitoring technology:

• The support equipment of JSF aircraft reduces by 50%,
• The service engineer reduces 20%-40%,
• The producing rate of sortie is improved by 25%.

44

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Aircraft Health and Usage monitoring systems (HUMS)

Boeing 747-400 and 777 has CMC (Central Maintenance computer) or

Airplane Condition Monitoring Systems (ACMS) on other models.

​

HUMS ENABLES AIRLINE CUSTOMERS TO MINIMIZE FLIGHT DELAYS AND CANCELLATIONS

45

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Health and Usage monitoring systems (HUMS)

Key technology

• Artificial intelligence
• Data mining
• Intellectual material
• Wireless sensor network
• Ultra broadband communication
• etc.

​

​

46

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

HUMS in MRI systems

• Failure prediction by human operators requires advanced skills
• The limited number of experts cannot monitor all MRI systems around the world.

-> NO "Corrective maintenance" for repairs after breakdowns

Solution: Sensor data from 100 MRI systems for create a mechanism to investigate the cause patterns that lead to device failures.

Then MACHINE LEARNING was used to define a normal operational state to achieve successful early detection of abnormalities and changes in status that lead to failures.

Result: Signs of impending failure have been detected several months before a breakdown occurs

As a result, downtime due to breakdowns has been reduced by 16.3%.

47

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

SmartCraft HUMS�Health and usage monitoring system

Cloud-enabled platform

• Machine learning
• Artificial intelligence
• Deep neural networks

​

Generate action-oriented data summaries

​

Decision Makers

Via

Online Dashboard on any web-connected device.

• Gain a deeper understanding of how the aircraft is functioning as a whole.
• Recognition of different systems abnormal anomalies rate, Failure detection.
• Operational Recommendations.

48

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Valve failure example

​

An Airline company complained about high frequent replacements of one particular valve in their fleet.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

49

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Anomalies Findings

• Based ONLY on the DATA
• There are anomalies found
• Classified into 5 categories of anomalies

| 50

UNCLASSIFIED

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

50

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified

Thank you

51

This document contains proprietary information of Israel Aerospace Industries Ltd. and may not be reproduced, copied, disclosed or utilized in any way in whole or in part, without the prior written consent of Israel Aerospace Industries Ltd

Unclassified