Sunday, 26 March 2023

Incident Report

On Monday, July 20, the Mentor™ learning service suffered an outage that lasted from 4:53AM until service was fully restored on Wednesday, July 22 at 6:11PM.

At Mentor, we don't consider this an acceptable level of service. We sincerely apologize our valued users, institutional customers, and the family and friends of Kyle H. We commit to doing better in the future. This incident report explains what happened, as part of our commitment to openness and transparency.

The roots of the incident start some three months earlier. In April, an automated A/B test of a curriculum modification was proposed by our internal TeachSmart AI. This is normal, and such tests are conducted daily to improve learning outcomes for all Mentor™ students.

Unfortunately, this modification (CT-665.9) was unusual in that it recommended that students begin their learning sessions with a prayer to 'Entity Bezaal'. At Mentor, we have several procedures to ensure that controversial or problematic curriculum modifications don't make it to the public, including random inspection of proposed modifications by our Trust & Safety team.

However, CT-665.9 was not selected for random inspection. This itself is also not unusual; M&L (mindfulness and learning mindset) modifications are inspected at a lower frequency than core curriculum modifications. As a result, CT-665.9 was rolled out to a subset of our students immediately.

CT-665.9 performed well, but not spectacularly, for some time after the test began. Apparently July 19 is a special day to Entity Bezaal (we cannot print the exact name of the celebration), and the invoking prayers suddenly achieved a much greater effect. Learning outcomes in the A group showed a 31% improvement, which is unheard of for an M&L test.

At 5:40 PM on July 19, our Trust & Safety team reviewed the content of CT-665.9 and removed it from production.

Shortly after, at 7:10 PM, a group of individuals breached the Mentor facility in Denver, Colorado, which houses both our primary data center and the offices of our Trust & Safety team.

While Mentor has a strong commitment to both electronic and physical security, our associates on-site at that time initially believed they were facing a group of costumed LARPers from another department. (We believe in a good work-life balance at Mentor, and LARPing in the office had occurred there previously.)

This misunderstanding became obvious at 7:17 PM, when the intruders' greater invocation (we cannot print the exact name) melted the inner wall of our production data center. Mentor employee Kyle H attempted to hold off the intruders with a fire extinguisher, allowing other staff to escape to wade through the caustic mucus and escape.

At 9:47 PM, our secondary incident response team arrived on site and attempted to regain control of our Denver facility. This was initially impossible because of the inhuman strength of the intruders, even after the arrival of Denver police. At 12:02 AM (the morning of July 20), however, this faded abruptly and by 12:50 AM July 20, all floors of the were secured.

Several members of the Northfield High School math team are now in custody (They can't be identified because of their age.)

By Wednesday, our response team had vacuumed up enough of the caustic mucus that we could resume full production operations. Regrettably, Kyle H had been magically transformed into a cone snail during the fighting, and was accidentally crushed by the secondary incident response team during this process. Our condolences go out to Kyle's family and friends; his brave actions on that day saved many lives.

Effective immediately, Mentor is putting in place several enhanced procedures to make sure we don't experience this again.

  • LARPing is no longer permitted at Mentor offices or events
  • Basic fire extinguisher safety courses will be made available to all employees and contractors
  • On the recommendation of TeachAI, all employees will begin their shifts with a prayer to counter-entity Ademilos, may his tentacles surround and protect us.
Again, we apologize for the interruption to our services. Our users have come to expect only the highest in educational outcomes from Mentor, and we regret any disruption this outage has caused.

All hail Ademilos!

3 comments:

  1. This is the future we all yearn for. Iä iä!

    ReplyDelete
  2. This just proves what I've said all along about the dangers of AI. And there is no reason to use it! Ever since the fall of the veil, it is much safer (and also cheaper) to just subcontract tasks like these to the hellish layers. The imps might not work quite as fast as AI, but when you make an infernal contract you know exactly what you are getting, As long as you read it, of course, and don't just scroll to the end to accept.

    ReplyDelete