Intel hardware is, generally speaking, extremely reliable. Every processor and chipset has some degree of bugs, but these issues rarely become critical flaws that jeopardize a product. Evidence is mounting, however, that Intel’s C2000 server platform — that’s the Avoton and Rangeley low-power server variants, all of which are based on Atom — have a serious flaw that’s bricking hardware.
Cisco has released an advisory warning informing its customers of the following: “In some units, we have seen the clock signal component degrade over time. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. Once the component has failed, the system will stop functioning, will not boot, and is not recoverable. This component is also used by other companies.”
So how does this link back to Intel, given that Cisco declined to name a vendor? Sharp-eyed boffins at The Register spotted an errata notice Intel issued in January, 2017. Intel’s AVR54 reads:
Problem: The SoC LPC_CLKOUT0 and/or LPC_CLKOUT1 signals (Low Pin Count bus clock outputs) may stop functioning.
Implication: If the LPC clock(s) stop functioning the system will no longer be able to boot.
Workaround: A platform level change has been identified and may be implemented as a workaround for this erratum”
The problem appears to affect all current steppings of the C2000 product family. While a platform fix has been identified, many manufacturers are choosing to recall products altogether rather than apply it. Cisco has announced that it will proactively replace hardware that was still under warranty or any other type of valid service contract as of November 2016. Hardware will be replaced whether it has already failed or not. Cisco isn’t formally using the word “recall,” but that’s what this is.
Did Intel use NDAs to squash reporting?
ServeTheHome did some digging into the situation and found multiple vendors willing to acknowledge being affected by the issue: Cisco, Netgate, QCT, and Supermicro all acknowledged the flaw — but not the cause.
ServeTheHome states that it has spoken to eight separate vendors, none of whom were willing to acknowledge which supplier was responsible. Multiple vendors cited NDAs with their suppliers that prevented them from identifying the source. Not many companies have the clout to force multi-billion dollar corporations to kowtow to its desire for secrecy. Not many suppliers are in a position to dictate to their customers how they will and won’t address a problem or speak to its cause. And of course, there’s the fact that Intel just updated its own documentation to identify exactly the problem in the C2000 family that eight separate vendors (so far) have acknowledged in their own products.
If you have Avoton or Rangeley hardware deployed for any reason, including as a component in embedded systems or networking hardware, we recommend contacting the vendor to determine whether replacement hardware is required or available.