This is a blog post describing what went wrong with the release on 2026-09-17, what we did to fix it and what we have learned from it.

tl;dr - Server went boop due to an error we didn't realise, we fixed it and put something in place to catch it next time.

Sequence of events

• Teifion deployed release1 to Integration for testing
  • Testing went fine, zero issues
• Teifion deployed release1 to Production
  • Initially went fine
• Errors started to appear in the log related to the navbar in a public page
  • Enough errors to trip the supervisor restart
  • Lexon was online and was made aware of the issue
• Teifion tried to rollback but something related to Discord commands started to error so knowing it was purely a templating issue he rolled forwards
• Teifion communicated with the main discord channel that the team were aware of the error and working on a fix
• Teifion was unable to replicate the issue locally but identified a likely fix and created a 1-line pull request
  • Lexon reviewed and approved the request very quickly
• Teifion deployed release2 to Production
  • Issue was resolved
• Teifion was later able to replicate the error
  • Created a regression test

So what actually broke?

If you were not logged in and tried to access a login-restricted page (e.g. your profile) you would be shown the login form. Due to a change the form would also try to include a new navigation menu at the top. This menu would check if you were authorised to see certain links and, because we'd not passed an "empty" user in correctly, it broke.

You would not encounter the issue visiting the login page normally; it only ever happened if you were redirected there from a different page.

How did it happen?

We didn't realise being redirected to such a page would be done slightly differently to loading it and thus were not testing for that. None of our automated or manual tests took this into account thus it slipped through.

What did you do to try and catch it?

We have multiple automated tests for looking at the login page. For this release we knew we'd changed the login page so had manually tested it multiple times to be safe. Not safe enough of course.

What stops this happening again?

We created a regression test for it. Every time we make a change to the codebase it goes through all the tests and if any of them fail we can't add our code until the issue is fixed. If someone accidentally reverts part of the change the regression test will fail and they will know to fix it.