Texas Performance Outage

First Alert

At 10 AM GMT, we were alerted to an outage affecting nine of our nodes in our Texas location:

• USC-TX-P9-A2-S

• USC-TX-P8-A2-S

• USC-TX-P7-A2-S

• USC-TX-P6-A2-S

• USC-TX-P5-A2-S

• USC-TX-P4-A2-S

• USC-TX-P3-A2-S

• USC-TX-P2-A2-S

• USC-TX-P1-A2-S

Due to the timing of the outage, our regular team was offline, and our emergency team was on standby. Normally, if an outage occurs when the main team is offline, our status tools immediately notify the emergency team. However, in this case, the status tools didn't trigger an alert right away, resulting in about an hour's delay before we identified and responded to the issue. We sincerely apologize for this delay.

Cause of the Outage

We partner with several data centres globally for renting hardware and hosting our own systems. In Texas, our primary provider has been servers.com.

Due to ongoing stock issues at servers.com, several hardware orders were pending, causing a temporary out-of-stock situation for our Texas location. To quickly resolve this, we began working with another data centre partner in the region and requested our account manager at servers.com to cancel any pending orders.

Upon investigation by our emergency team, we discovered that nine of our active servers from servers.com had been mistakenly terminated instead of the pending servers we requested to cancel. When alerted, I immediately reached out to our account manager at servers.com, who confirmed this error. With their permission, I've included an email from our account manager:

Restoring Servers

Once the issue was identified, servers.com acted to return the servers to our control. At this point, we had two restoration options:

1. Allow servers.com’s automated provisioning system to reset the drives and restore servers from our offsite backups, or

2. Request that the data centre technicians remove the drives before provisioning, reinstall them afterward, and prevent data loss.

We regularly take hourly snapshots for offsite backups, but restoring from backups could mean customers lose up to an hour of progress. To prevent any data loss, we chose the second option, requesting the data centre technicians remove the drives temporarily. This process took longer than expected because servers.com had to deploy their emergency team to the data centre to complete the work.

Final Note

We anticipate systems will be restored within the next few hours, with no rollback or data loss—your data will remain exactly as you left it. We deeply apologize to all affected customers. Although the incident was entirely outside our control, we have been actively updating our status pages, maintaining constant communication and pressure with servers.com, and ensuring our emergency team remains ready to assist.

Following this incident, servers.com has updated its processes so that account managers now must approve server cancellations to prevent similar errors from happening again.

Any questions, please reach out

Chris

• CEO @ Kinetic Hosting