Panel Outage 10/04/2025

What Happened?

At 7:34 AM UTC, our emergency team was alerted to an issue causing our game panel to become inaccessible. Upon investigation, we discovered the issue wasn't directly with the game panel, but rather with our caching proxy server.

What Does Our Caching Proxy Do?

Our caching proxy plays a crucial role in managing key services for our game panel. It acts as a central hub for content you interact with, such as caching mods when you're downloading modpacks. It pulls mods from CurseForge or Modrinth to build server files for you, ensuring quicker downloads and reliable installations. Additionally, it caches important software versions, like Forge and Fabric, ensuring quick and reliable access. Our caching proxy stores detailed information about modpacks in our installer, including mod/server file download URLs, as well as associated files like configs, scripts, and other necessary data.

Cause of the Outage

The outage was caused by unexpected and simultaneous failures of multiple storage drives within our caching proxy server. These drive failures occurred without prior warnings or system alerts. We immediately requested our data centre partner to inspect for other potential issues, but it was confirmed that the drives themselves had unexpectedly failed.

Impact on Services

As a result of the drive failures, our game panel experienced temporary downtime. Although no client server files were stored on this caching proxy (so your individual server data remained safe), critical services like our modpack installer and version management tools were significantly impacted. The caching proxy lost all stored data, including server file/mod URLs and additional files such as configs and scripts.

Recovery Actions

We swiftly worked to restore panel access and isolated the caching proxy issue to prevent it from affecting unrelated panel services in the future. We maintain backups of all our caching proxy data. Most data was restored quickly, however restoring data for over 800 modpacks back into our caching proxy took longer than expected. We sincerely apologize for this extended downtime and any inconvenience it caused.

At this time, full functionality for our modpack and version installers has been restored.

What's Next?

Many of you may wonder why we didn't already have a fallback or a distributed caching system (similar to a CDN) in place. When we initially set up our caching proxy, the scale of our operations didn't require more than one server. Given our recent growth, implementing redundancy, fallbacks, and a CDN-like solution is now a priority, and we're committed to rolling this out later this year.

Apology and Thanks

We deeply apologize for the disruption and appreciate your patience while we restored our service.

Chris

CEO @ Kinetic Hosting