Fault Tolerance: Designing Systems That Survive Failures

Key Takeaways

• Fault tolerance means a system keeps performing its function despite a specified number and type of faults. But a design tolerates only the fault classes it was built for.

• Faults cause errors, and errors that reach the service interface become failures. Fault-tolerant systems work by detecting errors and recovering before they turn into failures.

• Redundancy is the practice of duplicating critical components to prevent system failure if one stops working. Triple modular redundancy (TMR) uses a “majority voter” to compare the outputs of three identical components.

• Redundancy only helps if the copies fail independently. Common mode failures occur when multiple redundant components fail in the same way, defeating their purpose, which is why safety-critical designs should add diversity.

• Fault tolerance is stricter than high availability: a fault-tolerant system masks the fault with no service interruption, while a highly available system may accept a brief failover.

Introduction

Hardware failures are inevitable over a long enough service life. Disks wear out, memory cells flip, and connectors corrode. Fault tolerance is the ability of  a system to keep working when such incidents occur.

The term covers a wide range of engineering disciplines. For example, in a flight control computer it can mean dissimilar processor lanes and voting on sensor inputs and actuator commands. In a data center, meanwhile, it can mean physically separated power and cooling paths.

This article treats fault tolerance as a design property one can specify, quantify, and verify. It covers the terminology, the main redundancy patterns, the math behind voting systems, and examples from several industries.

What Is Fault Tolerance?

Fault tolerance is a system’s ability to contain faults. It is, according to Avizienis and colleagues, “aimed at failure avoidance” and “carried out via error detection and system recovery.”^[1] A fault-tolerant system is not one whose parts never fail, but one that notices when they do and contains the damage before users see it.

NASA’s fault management handbook defines failure tolerance as “the ability to perform a function in the presence of any of a specified number of coincident, independent failure causes of specified types,” and treats fault tolerance as a synonym.^[2]

Faults, Errors, and Failures

The three words are often used interchangeably, but they describe different stages of the same chain:

• Failure: an event that occurs “when the delivered service deviates from correct service”^[1]

• Error: the “part of the total state of the system that may lead to its subsequent service failure”^[1]

• Fault: the “adjudged or hypothesized cause of an error”^[1]

A fault can be dormant if it does not cause an error.^[1] A stuck bit in an unused memory region is a dormant fault. Once software reads it, it becomes an error. If that wrong value reaches an output, it becomes a failure.

This chain is why fault tolerance concentrates on errors. Faults are rarely observed directly; their consequences are observed in the system state. Faults also differ in how long they last. Intermittent faults come and go, while permanent (solid) faults persist until a component is repaired or switched out.

Number of Faults

When engineers say a system is “single-fault tolerant” or “one-fault tolerant,” they mean it continues to perform its function after any one fault of the specified type. Two-fault tolerant means it survives any two.

Interestingly, single-fault tolerance may suffice even in critical systems. For example, NASA amended its human rating requirements in 2008, changing from two-fault to one-fault tolerance. The administration noted that “adding failure tolerance beyond a certain level did little to improve reliability for launchers. 0 to 1 shows marked improvement, 1 to 2 very little.”^[3] Each added level of fault tolerance costs mass, power, and complexity, so it must justify itself.

Hardware Fault Tolerance in Functional Safety

Process and machinery safety usually follows the IEC 61508 standard, in which hardware fault tolerance (HFT) is the number of dangerous failures a subsystem can tolerate before it loses its safety function.^[4] Four common architectures are given in the standard, shown in the table below.^[4]

The standard links a minimum HFT to the required safety integrity level (SIL), the safe failure fraction (SFF) of the components, and whether a component is type A or type B.^[4] In an example from NTNU’s RAMS group, a type B smart sensor with an SFF of 85% and a SIL 2 requirement needs a minimum HFT of 1, which permits either 1oo2 architecture or 2oo3.^[4]

Recommended reading: Functional Safety Foundations

Fault Tolerance vs High Availability, Graceful Degradation, and Fail-Safe Design

Several related terms describe what a system does when something breaks. They are sometimes confused in vendor glossaries, so it helps to separate them.

To use data centers as a concrete example, the Uptime Institute—a global advisory organization for business infrastructure—describes a Tier III facility as “concurrently maintainable,” requiring “no shutdowns when equipment needs maintenance or replacement.”^[5] A Tier IV facility adds fault tolerance: “when a piece of equipment fails, or there is an interruption in the distribution path, IT operations will not be affected.”^[5]

High availability is usually expressed as uptime percentage. These figures show how small the downtime budget becomes:

A highly available system can meet these targets as long as its failover mechanisms complete within the downtime budget, keeping service outages short and rare. A fault-tolerant system aims for uninterrupted service through the faults it is designed for. AWS notes that it is “less expensive to provision less compute capacity and rely on launching new instances in the case of a failure,” but that approach is less effective for large-scale failures.^[6]

How Fault-Tolerant Systems Work

Avizienis and colleagues break fault tolerance into a small set of techniques that apply equally to hardware systems and software systems.^[1] A fault-tolerant design typically runs through these steps:

1. Error detection: Identify that the system state is wrong. Concurrent detection “takes place during normal service delivery,” while preemptive detection runs while service is suspended.^[1] Checksums, watchdog timers, comparison of redundant channels, and health checks all fall here.

2. Error handling: Remove the error from the system state. Rollback “brings the system back to a saved state that existed prior to error occurrence” (a checkpoint). Rollforward moves to a new state without the detected errors. Compensation uses redundancy in the erroneous state “to enable error to be masked.”^[1]

3. Fault handling: Stop the fault from causing the error again. This covers diagnosis, isolation, reconfiguration (switching in spare components or reassigning tasks), and reinitialization.^[1]

4. Corrective maintenance: Repair or replace the isolated part. What distinguishes fault tolerance from maintenance is that “maintenance requires the participation of an external agent.”^[1]

Masking is the ability of a system to conceal faults and continue as normal without affecting system liveness. It is an important element of high-availability systems, but it carries risk. 

When compensation is applied systematically, faults can disappear from view, which “will conceal a possibly progressive and eventually fatal loss of protective redundancy.”^[1] A triple-redundant system that silently loses one channel effectively becomes a dual system, which is why practical masking designs also include error detection and reporting.

Preemptive detection matters for the same reason. Avizienis and colleagues list “spare checking, memory scrubbing, audit programs, or so-called software rejuvenation” as forms of it.^[1] Otherwise a standby system that has never been exercised may already be broken when it is needed.

Redundancy

Redundancy is the deliberate addition of resources beyond what the function strictly needs. NASA defines it simply as “duplicate functions or mechanisms.”^[2] It comes in four broad forms:

• Hardware (spatial) redundancy: extra components, such as duplicate power supplies, triplicated processors, or mirrored disks

• Information redundancy: extra bits that let you detect or correct errors, such as ECC memory, parity, and RAID

• Time redundancy: repeating an operation, such as retries, re-execution after rollback, or retransmission after network failures

• Software redundancy: independently developed versions of the same function, or analytical models that cross-check sensors

The common redundancy configurations have standard names. The definitions below are general industry terminology:

Static, Dynamic, and Hybrid Redundancy

This is another way to categorize redundancy. Whether a system uses static, dynamic, or hybrid redundancy depends on whether it favors masking or switching. 

Triple modular redundancy masks a single error and is therefore a type of static redundancy; it is used “when no interruptions are desired.”^[7] A primary and backup pair, where the backup may run a shadow process ready to take over, is dynamic redundancy, used “when a brief interruption is acceptable.”^[7] Combining a TMR core with spares that can be switched in gives hybrid redundancy.^[7]

Dynamic redundancy has a hidden parameter called coverage: the probability that the system both identifies the bad module and successfully restarts the process on another.^[7] A standby system with perfect hardware but 90% coverage still fails one time in ten when the primary goes down. Coverage is set by error detection and switchover logic, which makes those parts as safety-critical as the redundant units themselves.

Triple Modular Redundancy and Majority Voting

In triple modular redundancy, three identical modules compute the same result and a voter outputs the majority value. If each module has reliability R and failures are independent, the system works when at least two modules work. With a perfect voter, the reliability is:^[7]

R_TMR = 3R²(1 - R) + R³ = 3R² - 2R³

Worked example (illustrative arithmetic): take a module with R = 0.95 over a given mission.

1. 3R² = 3 × 0.9025 = 2.7075

2. 2R³ = 2 × 0.857375 = 1.71475

3. R_TMR = 2.7075 - 1.71475 = 0.99275

The probability of failure drops from 5% to about 0.73%, roughly a sevenfold improvement, at the cost of three times the hardware plus a voter.

However, TMR has limits:

• Crossover point: Solving 3R² - 2R³ = R gives R = 0.5. Below that, majority voting makes the system less reliable than a single module.^[7] TMR only pays off when modules are individually good.

• MTTF: For modules with constant failure rate λ, the mean time to failure of TMR is 5/(6λ), lower than the 1/λ of a single module.^[7] TMR is far better over short missions but worse over very long ones unless failed modules are repaired.^[7]

• Temporary faults: With repair, steady-state availability follows the same form.^[7]

• Voter: The formula assumes a perfect voter, which is “a reasonable assumption if the voter complexity is much less than an individual module.”^[7] In a real design the voter is itself a component that can fail, and a single voter is a potential single point of failure.

• Synchronization: In hardware TMR, the three processors either run in synchronization with each other, or the voter has to wait until comparable results from all three are available.^[7]

Byzantine Fault Model

Voting also has a subtle input problem. Lamport, Shostak, and Pease showed that, for majority voting to work, “all nonfaulty processors must use the same input value.” But a faulty input unit can send a marginal signal along a shared wire, one “that can be interpreted by some processors as a 0 and by others as a 1.”^[8] That observation led to the Byzantine fault model.

A Byzantine component does not just stop; it sends “conflicting information to different parts of the system,”^[8] leading to faulty nodes or “traitors” that disagree with the working nodes or “generals.” 

Lamport, Shostak, and Pease proved that with ordinary messages “no solution with fewer than 3m + 1 generals can cope with m traitors,” while with unforgeable signed messages the problem “is solvable for any number of generals and possible traitors.”^[8] Tolerating one Byzantine fault with unsigned messages therefore needs at least four nodes, compared with three for one crash fault under majority voting.

Information and Time Redundancy: ECC Memory and RAID

Information redundancy helps protect data against errors and failures. A large study by the University of Toronto and Google found that more than 8% of server memory modules experienced errors each year.^[9] Error-correcting code (ECC) memory detects and corrects errors automatically. Standard SECDED codes can correct single-bit errors, while more advanced Chipkill codes can handle larger errors, including the failure of an entire memory chip. However, ECC cannot prevent every failure: the study found that 1.3% of machines experienced uncorrectable memory errors annually.^[9]

Storage systems use similar techniques to protect against disk failures. RAID (redundant array of independent disks) spreads data across multiple drives, using duplication or additional recovery information to keep data accessible when a drive fails.^[10] 

• RAID 1 mirrors data across two drives, while other RAID levels use parity information to provide protection with less storage overhead. 
• RAID 0 improves performance but offers no fault tolerance. 

Importantly, RAID is not a substitute for backups: it cannot reliably protect against accidental deletion, data corruption, or disasters affecting the entire system.^[10]

Fault Tolerance in Practice

Recommended reading: Mesh Topology: A Deep Dive into Principles and Applications

Common Mode Failures and Design Diversity

The redundancy math in previous sections assumes independent failures. Avizienis and colleagues state that the faults that can actually be tolerated rely on the “independence of redundancies with respect to the process of fault creation and activation.”^[1] Identical channels are adequate for physical faults but not for solid development faults such as design errors or software bugs, which every identical copy shares.^[1]

In software, N-version programming runs independently developed versions and votes on their outputs, which “depends for its reliability improvement on the assumption that programs that have been developed independently will fail independently.”^[15] Knight and Leveson tested that assumption with 27 versions written from the same specification and subjected to one million tests. The programs were individually very reliable, but “the number of tests in which more than one program failed was substantially more than expected.”^[15]

Common cause failures also come from the environment: a shared power feed, a single cooling loop, a fire, a flood, or a bad firmware update pushed to every unit at once. NASA’s human-rating requirements require designs to “prevent or mitigate the effects of common cause failures in time-critical software.”^[3] The usual defenses are:

• Diversity: different processors, compilers, algorithms or vendors, as in the 777’s dissimilar PFC lanes^[11]

• Separation: physical and electrical isolation of redundant paths, as Tier IV requires^[5]

• Independent monitoring: a simpler, diverse channel that checks the main channels rather than duplicating them

• Staged changes: rolling software and configuration updates out to one redundant unit at a time

How to Design a Fault-Tolerant System

A practical design process works through the same questions regardless of domain:

1. Define the function and the failure to avoid: Specify what correct service is and which failures are unacceptable: loss of service, wrong output, unsafe state, or data loss.

2. Write the fault assumption: List the tolerable fault classes (permanent hardware, transient, software bugs, network failures, Byzantine behavior) and how many coincident faults.

3. Choose the failure behavior: Decide between fail-operational, graceful degradation, fail-safe, or high availability with failover for each function.

4. Select the redundancy pattern: Match it to the fault assumption: TMR or 2oo3 for masking, active-standby for fast failover, ECC and RAID for data, replication and consensus for distributed state.

5. Design detection and recovery: Specify how errors are detected, how quickly, and how the system isolates and reconfigures, then estimate coverage.

6. Attack common-mode failures: Add diversity and separation where independence cannot be assumed.

7. Quantify: Model reliability and availability with realistic failure rates, repair times, and coverage, and compare against the requirement. Failure modes and effects analysis and fault tree analysis are the standard tools for physically redundant systems.^[11]

8. Verify by injecting faults: Test the system by deliberately failing components, links and software, and confirm it behaves as specified.

Conclusion

Fault tolerance is not a feature to bolt on. It is a property to be specified in terms of functions, fault types, and fault counts, then delivered through redundancy, error detection, and recovery. The vocabulary of faults, errors, and failures tells you where to intervene; the redundancy patterns give you the tools; and the reliability math tells you whether a design actually meets the requirement.

The hardest part is independence. Voting, replication, and spare capacity all assume that redundant parts fail separately, and real systems constantly violate that assumption through shared designs, shared environments, and shared software. The best fault-tolerant architectures, from the 777’s dissimilar flight computers to physically separated Tier IV data centers, invest as much in diversity and separation as in duplication.

Frequently Asked Questions

What is fault tolerance in simple terms?

Fault tolerance is a system’s ability to keep doing its job correctly when one or more of its parts fail. It works by adding redundancy and by detecting and correcting errors before they affect the output.^[1]

What is the difference between fault tolerance and high availability?

A fault-tolerant system masks a fault so that service continues without interruption. A highly available system minimizes downtime but may accept a short interruption while it fails over to backup components.

What is the difference between fault tolerance and resilience?

Usage varies, but fault tolerance generally refers to continued correct service in the presence of specified faults, while resilience is broader and includes recovering from and adapting to disruptions, possibly with temporarily reduced service. Graceful degradation sits between the two.

What does single-fault tolerant mean?

A single-fault tolerant system continues to perform its function after any one fault of the specified type. In IEC 61508 terms, it corresponds to a hardware fault tolerance of 1, such as a 1oo2 or 2oo3 architecture.^[4]

What is an example of fault tolerance?

The Boeing 777 flight control system uses three sensor sets, three data buses, three primary flight computers with three computing lanes each, and four actuator control units, with voting on inputs and actuator commands and a requirement of no single point of failure.^[11] Everyday examples include ECC memory correcting bit errors and mirrored disks in a RAID 1 array.

Is RAID 0 used for fault tolerance?

No. RAID 0 stripes data across disks for performance and keeps no redundant copy, so losing any one disk loses the array’s data. Fault tolerance in RAID comes from extra disks holding redundant information, either mirrored copies or parity.^[10]

Which technique allows a system to switch to backup components when a failure occurs?

That technique is failover, a form of dynamic redundancy: a fault is detected, the faulty unit is isolated, and the workload is reconfigured onto a spare.^[1] The probability that this switch succeeds is called coverage.^[7]

Which replication strategy keeps backup systems inactive until failure occurs?

Passive or cold standby replication keeps the backup idle until the primary fails. It is cheaper than active replication but takes longer to recover, because the backup must start up and load state before taking over.

Is fault tolerance good or bad?

Fault tolerance is beneficial wherever failure is costly, but it is never free. It adds hardware, power, weight, complexity, and testing effort, and extra complexity can itself introduce faults.

References

[1] Avizienis A, Laprie JC, Randell B, Landwehr C. Basic concepts and taxonomy of dependable and secure computing. College Park (MD): University of Maryland, Institute for Systems Research; 2004. Technical Research Report No.: TR 2004-47. Published in: IEEE Trans Dependable Secure Comput. 2004.

[2] NASA. NASA-HDBK-1002 fault management handbook, draft 2 [Internet]. Washington (DC): NASA; 2012 Apr [unapproved draft].

[3] NASA’s human rating requirements: a historical interpretive perspective [Internet]. Washington (DC): NASA Technical Reports Server; [cited 2026 Nov 6].

[4] Lundteigen MA, Rausand M. Architectural constraints (extension to chapter 2) [Internet]. Trondheim: NTNU RAMS Group; [cited 2026 Nov 6]. Lecture slides, version 0.1.

[5] Uptime Institute. Tier classification system [Internet]. Uptime Institute; [cited 2026 Nov 6].

[6] Amazon Web Services. REL11-BP05 Use static stability to prevent bimodal behavior. In: AWS Well-Architected Framework [Internet]. Amazon Web Services; [cited 2026 Nov 6].

[7] Malaiya YK. Fault tolerant computing: reliability analysis pt 2 [Internet]. Fort Collins (CO): Colorado State University; 2020 Feb. CS 530 course material.

[8] Lamport L, Shostak R, Pease M. The Byzantine generals problem. ACM Trans Program Lang Syst. 1982 Jul;4(3).

[9] Schroeder B, Pinheiro E, Weber WD. DRAM errors in the wild: a large-scale field study. In: Proceedings of SIGMETRICS/Performance 2009; 2009.

[10] Patterson DA, Gibson G, Katz RH. A case for redundant arrays of inexpensive disks (RAID). In: Proceedings of ACM SIGMOD; 1988.

[11] Seiler P, Hu B. Design and analysis of safety critical systems [Internet]. Minneapolis (MN): University of Minnesota; 2013 Feb. Slides; 777 material after Yeh YC, 1996.

[12] Fu Y, Terechko A, Groote JF, Khabbaz Saberi A. A formally verified fail-operational safety concept for automated driving. arXiv:2011.00892 [Preprint]. 2020. Published in: SAE Int J Connect Autom Veh. 2022.

[13] Texas Instruments. Active OR-ing solution with LM74610-Q1 smart diode controller (SNVA747B) [Internet]. Dallas (TX): Texas Instruments; 2017 Mar [revised].

[14] Ongaro D, Ousterhout J. In search of an understandable consensus algorithm (extended version) [Internet]. Stanford (CA): Stanford University; [cited 2026 Nov 6].

[15] Knight JC, Leveson NG. An experimental evaluation of the assumption of independence in multiversion programming (summary). IEEE Trans Softw Eng. 1986;12(1). Course summary hosted by KTH.