As of this time we can consider this incident closed. On December 27, 2017 at approximately 8pm EST Zoey experienced a failure in one of its computing nodes in the Stack2 Computing Cluster. This issue was corrected with a simple restart of services and the outage lasted approximately twenty minutes. Over the following ten days we continued to experience, at random, one of our computing nodes in our Stack 2 cluster failing. Through intensive troubleshooting and over fifteen unique resolutions we were unable to find a fix. Each of these outages was short in duration, less than twenty minutes, and affecting less than 10% of Zoey customers at any given time. Through further troubleshooting we discovered that the error was a bug in the underlying Linux Kernel and worked with the appropriate third party team in implementing a patch. At this time Zoey believes this error has been corrected. Further information about the patch can be found at https://patchwork.ozlabs.org/patch/712373/
- please note this is a very short and technical summary of the overall issue.