SEU Abridged Discussion

Single Event Upset (SEU) or Single Event Effect (SEE) is a space and terrestrial effect that, despite the system may have passed all quality-related testing and verifications, the system can still experience random faults that are typically recovered after reset or power cycle. First, you need to answer how much pain it would cause if your application fails. A more solid pictorial example is as follows. <ul style="text-align: justify;"><li>Imagine the AI-powered robotic arm wildly rotates its arm and destroys the assembly line</li><li>An autonomous driving vehicle failed to recognize the pedestrian on the crosswalk and didn't stop</li><li>The accelerated trading platform buys at full speed while the marketing crashes</li></ul> The question is not if this will happen but when and how often it will happen. Of course, this is an unavoidable topic if you're already bound by safety standards like ISO26262, IEC61508 or your customer has already asked you about FIT.    <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic1.jpg" width="697" height="235" />   <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic2.jpg" width="484" height="315" />           <h2 style="text-align: justify;">Real World Examples </h2>To drive the points home, the following hyperlinked topics provide lessons learned that already cost millions of dollars and/or lifes. <a href="https://www.bbc.com/future/article/20221011-how-space-weather-causes-computer-errors">BBC special on SEU is to blame for your electronic woes - Mr. Gate is vindicated of all the blue screens~</a>  <a href="https://www.bbc.com/news/business-46999443">Google Engineer manhandled SEU</a>  <a href="https://en.wikipedia.org/wiki/Qantas_Flight_72">The tragedy of Quantas Flight 72</a> <a href="https://en.wikipedia.org/wiki/Electronic_voting_in_Belgium#%3A~%3Atext%3DThe%20official%20explanation%20was%20%22the%2Csystem%20did%20not%20protect%20against"> Belgium's party biased e-voter</a> <a href="https://en.wikipedia.org/wiki/Takata_Corporation#%3A~%3Atext%3DOn%20June%2025%2C%202017%2C%20Takata%2Cwas%20possible%20for%20its%20survival">Tanaka </a>airbag or <a href="https://www.bbc.com/news/world-us-canada-65578475">Arc automotive recall</a>   Or course, the list goes on, with <a href="https://support.huawei.com/enterprise/de/knowledge/EKB1000085293">Huawei </a>and <a href="https://www.networkworld.com/article/3122864/cisco-says-router-bug-could-be-result-of-cosmic-radiation-seriously.html">Cisco </a>both listing SEU as a known issue and as a standard debugging step and that it's a common assumption that ~20% of all "can not be reproduced" errors attributed to SEU in 1998. But if this is so serious, why haven't major IC folks talked about this?  Most of these folks take a "you don't ask, I don't tell" approach. This is exactly why high-reliability applications clearly outline the mandates in specifications such as ISO26262. And you can further verify that the major players are all doing their homework (or some of them) by checking out the various accelerated radiation facility's booking schedules.  <h2 style="text-align: justify;">Source of Radiation </h2> <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic3.jpg" width="471" height="312" /> The sun is full of radiation, while our mother Earth's magnetic field shields us from most of the harm. On top of that, package material can be a serious contributor as well. As to exactly how physics interacts between the radiation and IC, I will leave that to the Physicists and Radiation Engineer to worry about. As a designer or engineer, you only need to know it's there and it's real. Then we'll need to figure out how to deal with it. So hopefully, by now, I have convinced you that the effects are real and we need to take care of them for some of us. And we'll dive into the topic of FIT in the next part of this series.  <h2 style="text-align: justify;">Failure in Time – FIT </h2>1 FIT = 1 failure in a billion hours. In other words, it takes around 114,155 years for an error to occur. Quality engineers will often mention MTTF (Mean Time To Failure) and MTBF (Mean Time Between Failure). Although one error in 114,155 years sounds abysmal, a generally assumed unmitigated processor with a FIT of 50,000 would mean a fault every 2.28 years. But this is one single processor. A system will likely have memories and many other ICs. It's a common consensus to use 1,000 FIT/M gates for unmitigated 90nm ASIC. And if you expect just to build 10,000 systems, the number adds up fast, and that accumulated number should be enough to send a chill down any quality engineer's spine.  Real Numbers to put things into Correct Perspective AMD-Xilinx is the poster child in the SEU field when doing their due diligence and openly publishing their results. We're interested in the Device Reliability Report <a href="https://www.xilinx.com/support/documentation/user_guides/ug116.pdf">UG 116, </a>updated every six months. First, take a look at the figure below. Some FIT is already provided.   As engineers, it's our job to drive down the FIT during normal life or the region not covered by the bold red letters in the diagram.  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic4.jpg" width="478" height="353" /> Table 19 should specify the various SEU-related FIT numbers for CRAM (configuration ram). For those unfamiliar with AMD FPGA, FPGA is an SRAM-based reconfigurable gate array device. Therefore, the user design must first be loaded into the configuration memory for the device to function as expected. Look at the first three rows, all with tech nodes of 90nm but spanning 4 product families. As you can see, the LANSCE Neutron cross-section per bit is different. Moreover, the last two columns vary up to 2.6x between Virtex-4 and Spartan-3E/3A.  The lesson for this is that don't assume your 90nm will have the same superb number as AMD. They put in much work to reverse the technology trend to drive down the FIT. Otherwise, with each shrinking technology, one would expect it takes less energy to upset a cell while knocking out multiple cells. It's also important to note that they're the only major house player I know who provides real-world data. If you merely follow the JESD89 conversion, you'll see there will still be quite a gap between the derived number and the real-world number。  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic6.jpg" /> The main takeaway here is that it's inevitable that IC will fall to SEU eventually, and the more advanced the node is, the worse the problem is unless you do something.  <h2 style="text-align: justify;">SEU Family Tree </h2>It can be simply broken down into two categories:  <ul style="text-align: justify;"><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1">Hard/destructive  </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1">Non-destructive </li></ul>Hard errors are rare but catastrophic, while software is more common.  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic7.png" width="517" height="273" />  Before we move on, note that this table should be used as a baseline reference. For example, FPGAs are also vulnerable to SET, and it's just that the probability of an SEU is so much higher than SET that we should focus on SEU.  In addition, some of the errors will also be vendor or even device-specific. Do a quick search on FPGA and SEL, and you'll see that not all vendors and parts are vulnerable to SEL. Non-Destructive SEE SEU (Single Event Upset): General error that results in a functional error MBU (Multiple bits Upset): Any single strike that knocks out bits MCU (Multiple Cells Upset): MBU, but not limited to bits  Most ECC deployment is based on SECDEC, Single Error Correct Double Error Detect. Therefore, aliasing can occur if an upset spanned more than three errors. Refer to the following generalized diagram; you're safe at 180nm, but at 65nm, ~20% of errors resulted in MBU.  At 40nm, you're now close to 40%, SECDEC is effectively useless. It's also true that the result can vary significantly pending foundry, cell layout, etc., but if you're unaware of this issue, don't expect the device vendors to tell you this voluntarily. <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic8.jpg" width="359" height="256" />  SEFI (Single Event Functional Interrupt) Anything that knocks out the device function can be put into this bucket. For example, if an update knocks out your PCI bridge such that the device drops the packet and is no longer responsive to subsequent requests, it's a SEFI.    SET (Single Event Transient) A picture is worth a thousand words. Try to imagine this glitch on the reset or clock line, but at the same time, a glitch on the data input without a clock likely isn't going to result in any error.  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic9.jpg" width="545" height="134" /> SED (Single Event Disturb) Effectively just a dumpster for anything not covered by the other family members. Destructive Errors Most of these errors are self-explanatory by name. The common symptom is a damaged device which typically manifests itself with an additional current draw that can't be recovered even after a power cycle. <ul style="text-align: justify;"><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1">SHE (Single Event Hard Error)  </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1">SEL (Single Event Latch-up)  </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="5" data-aria-level="1">SESB (Single Event Snap-Back)  </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="6" data-aria-level="1">SEB (Single Event Burn-out) </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="7" data-aria-level="1">SEGR (Single Event Gate Rupture)  </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="8" data-aria-level="1">SEDR (Single Event Dielectric Rupture) </li></ul> <h2 style="text-align: justify;" aria-level="1">Not All Errors Are Created Equal </h2>After reviewing the SEU family tree, you should quickly see that not all errors are equal. Some errors significantly impact the system, while others may simply flush out with the system, even experiencing a blip. SEUPI, Single Event Upset Probability Impact factor, is typically used to further derate the probability of an event that may result in actual errors observed.  <h2 style="text-align: justify;">Choices </h2>As an engineer, many techniques are at our disposal to drive down FIT. Do note that some of the techniques discussed only apply to IC design, some for system or module designers, while all can deploy some. You'll also notice that many techniques can target the same failure. The choice is yours. It comes down to area, resource, speed, power, and all these constraining factors you get to deal with before you choose the option that best suits you. For example, if you're an IC Designer and need to protect memory's integrity, you're limited to methods such as ECC, parity, scrubbing, interleave, cell type, etc. But as a system or module developer, you must deploy redundancy and software mitigation schemes such as SWIFT or simply log and optionally reset the module.  <h2 style="text-align: justify;">Mitigation Techniques </h2><img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic10.jpg" width="545" height="397" /> <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic11.png" width="541" height="276" /> <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic12.png" width="586" height="289" />  Now you see numerous techniques that we have at our disposal. For full tables, please follow up with me. But to demonstrate how even a simple technique does require plenty of expertise, we'll dive into one technique.  <h2 style="text-align: justify;" aria-level="1">TMR - Triple Module Redundancy </h2>Three times power consumption, area, resources, and complexity always equate to a thousand times happiness. IEC61508 provided some different variations of redundancy as well.  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic13.jpg" width="533" height="215" /> Now let's assume one of your 2oo3 voter channels is out due to SEU, but the error stays in the system because it's part of the one-hot finite state machine (FSM). The system now becomes up to twice as vulnerable to subsequent failure.  <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic14.jpg" width="591" height="122" /> Let's look at another widely touted LTMR (Local TMR) scheme that protects flip-flops below.   <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic15.jpg" width="571" height="225" />  This is a shift register design that protects only flip-flops. The essence of the comparison is that the red line (Chain) is effectively the same, although the blue line (TMR flip flop) lowered significantly. In an FPGA, the configuration bits required to set the flip flop's content may pale compared to the number of configuration bits required to set routing. Therefore, performing redundancy without sufficient knowledge may be worse than not doing anything at all.    <h2 style="text-align: justify;" aria-level="1">Mitigation Schemes Deployed by IC and System Providers </h2>After all that talk, let's review the major IC players' actions. It's important to know that each IC households this deck of cards very close to their chest, and many published data may be outdated, but it still provides the idea that this SEU issue is not to be simply overlooked. First, s/390 G5 provided by IBM had added 20~30% logic to protect its execution units and main process. Fujitsu claimed to have 80% of its 20,000 latches in Sparc64 covered, parity added to ALU, and checks done on multipliers and dividers. Xilinx went even a step further with its Zynq family. It has completely separated PL (programmable logic) and PS (process) power, so you now have redundancy. This is in addition to providing two ARM's real-time processors so you can deploy schemes like lockstep and other software and design flow techniques that one can further deploy to drive down FIT. A lot of background work has been done so ASIL-C can be achieved on one part. And if you google Huawei and SEU, NTT's 10G-OPON ONU+SEU, or Broadcom's WP SE-BSD-WP100, you'll see that system folks other than automotive folks have long been dealing with SEU. Cisco even had a radiation test team that listed SEU as a standard <a href="https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116135-trouble-6500-parity-00.html">debugging step.</a>  <h2 style="text-align: justify;">Verify as Soon as Possible </h2>The design must inevitably be verified. A mitigated design only cranks up the pain meter. For anyone who codes in RTL, you can't escape writing test benches. And no, you don't get to check that done box just by implementing system verilog's random function.  And are we done after the simulation? Life can't be that easy, can it?  The general rule of thumb is still applicable. The earlier you catch a bug, the less hair you'll pull out.   <h2 style="text-align: justify;" aria-level="1">Simulation </h2>We know we can't cover all cases; therefore, getting bang for the buck is essential. Hit the clock line or reset line if they're also triplicated. Hit logic with high value so you can quickly validate how robust the code is. The list goes on, but you get the idea.  Fault Injection FPGA emulation or FPGA designs provide designers with a quick and easy way to inject errors into the design on a bit-per-bit basis. And if you know your configuration well enough, you may be able to back-trace the failing bit to your design.  Accelerated Radiation Testing This is the closest method you get other than real-world data collection. And if you're doing this for ISO26262 and following JESD89, you still have to think about all the nitty-gritty details if you've never done it before.  Let me just toss out a few things for you. <ul style="text-align: justify;"><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="9" data-aria-level="1">What beam will you use? Alpha, proton, neutron, x-ray, focused laser beam? </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="10" data-aria-level="1">Will this be a dynamic or static test? </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="11" data-aria-level="1">What type of failure should you be looking for? Are you sure you can catch it and recognize it? </li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{"335552541":1,"335559683":0,"335559684":-2,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" aria-setsize="-1" data-aria-posinset="12" data-aria-level="1">What monitoring system would you deploy? </li></ul>To provide you with a real-life example, let's look again at the AMD-Xilinx test setup for a "dynamic test". <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic16.jpg" width="370" height="208" /> <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic17.jpg" width="369" height="288" /> Two setups are provided here; you can tell one setup is in the open air while the other is in a vacuum. <img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Chenpic18.png" width="339" height="251" />  The discussion of beam selection, test setup, and all the gotchas is a giant topic in and by itself. This is why some major commercial folks even have their dedicated radiation test team.    This article was contributed by our expert <a href="https://www.linkedin.com/in/chen-wei-tseng/" target="_blank" rel="noopener">Chen Wei Tseng</a>   <h3 style="text-align: justify;">Frequently Asked Questions Answered by Chen Wei Tseng</h3> <h2 style="text-align: justify;">1. What mitigation strategies or techniques can be employed to reduce the impact of SEU and power cycles on electronic systems?</h2>The goal is first to reduce FIT rather than reset or power cycle the system. Reset or power cycle the system will bring down the system and result in user operation interruption and critical applications such as electric cars, which may result in life or death.   <h2 style="text-align: justify;">2. How do safety standards align with industry practices and experiences regarding SEU, and do they evolve to address emerging challenges in this area?</h2>As you can tell, many applications already have. I'll append the pictures here for your reference. Most of the standards for the designers to refer to "experts" as the effect can differ dramatically pending IC technology node, vendor, foundry, etc. The knowledge in this field is lacking here in Asia. But of course, as the course has already mentioned, applications that can result in significant damage will also be a concern.  I've listed examples from Huawei and Cisco in the piece, I'm aware of systems in the financial world (accelerated trading platform), gaming machines, AI applications, etc, that, although not bound by the standards but are rather interested in this topic because they've already been burned.  pic  <h2 style="text-align: justify;">3. How do high-reliability applications account for the potential occurrence of SEU in their system design and architecture?</h2>Through design mitigation, engineers must first know the FIT (failure in time) rate to deploy the correct mitigation.   <h2 style="text-align: justify;">4. Are there any advancements or innovations in package materials that aim to improve SEU resistance?</h2>As far as I'm aware, no. This is why the burden is now on the designer to deal with it.

Premium Insights

Information Technology

KR Expert - Chen Wei Tseng

Aligning Business Needs With AI-Powered Virtual As..

Role Of Blockchain Technology In Sustainability In..

ANI/AGI/ASI - Artificial Intelligence Advancements..

Large Language Models Vs. Short Language Models-A ..

Shifting Landscape Of Cybersecurity

Platform Modernization - Legacy To XML Based Archi..

Core Services

Phone Interviews

Targeted Custom Surveys

Term Engagements

Executive/Board Placements

Information Technology