Squarespace, Fog Creek, Peer1 Kept NY Data Center Alive By Carrying Fuel Buckets To The 17th Floor In The Dark

Small businesses may have been even more affected than big companies by Hurricane Sandy, as they don't have the same resources. When Squarespace sent an email on Monday morning, it looked like they had tried everything to keep the servers alive. They asked for forgiveness in advance. But the unbelievable happened, they kept the service going by bringing fuel buckets to the 17th floor in the dark for more than 54 hours.

When ConEd lost power, the data center switched to a generator powered by fuel pumped from the basement. They had enough fuel to stay online for three or four days. "Peer1 stayed online during the last 3 major natural disasters in the area, including a blackout that lasted for days," Squarespace founder and CEO Anthony Casalena said in the first email sent to customers.

But little they know that this time around, there would be important flooding. "At 5AM, we learned our data center's fuel pumps and fuel tanks were completely flooded and unable to deliver any more fuel. […] We have simply run out of power, backup power, and cannot access our fuel in a flooded basement." The email was classy, apologetic and nobody could blame the company.

"We will do everything in our power to get Squarespace running as soon as possible, and we will remain online for as long as it is safe." With that sentence, customers could have expected that they would try to contact the right people to restore power. But it didn't took long to realize that they had another plan in mind.

Fog Creek, the company behind Trello, and Peer1, the hosting company, joined forces over the hours. Little by little, all those people organized bucket teams with shifts, a stock on the roof and caffeine. They reported the events on Squarespace's status page.

It shows the level of dedication a founder and CEO can have for his company, as he was one of the first in the data center to figure out a plan.

Here are some selected status updates. Go read the whole thing.

Tuesday, 2:07PM:

We have an understanding that when the fuel header on the roof stops refilling our tank, we will likely be in a window where shutdown will occur within 2 hours. That has not happened yet, which is good. We have fuel at the building, but cannot tap into the damaged building infrastructure yet

Tuesday, 7:39PM:

We have been running hour by hour by manually carrying fuel to our generators (17 floors) with support from the building. As the night goes on, this is becoming a bit more difficult to sustain, as fuel trucks will appear more intermittently. For now, we remain online. Thank you all for your patience.

As before, we do not have a sustainable solution to the energy problem, but we persevere in our efforts to get more fuel on site and get a pump connected

Wednesday, 1:02AM:

We continue to manually provide fuel to the generator, a plan which appears to be working, but is likely not sustainable. The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. We have loaded enough fuel to carry to the morning, at which point we'll provide another status update

Wednesday, 12:40PM

Good news and bad news.

We were able to give the crew 90 minutes off because the tank is full. That's about 4 hours of power given daytime usage. We'll start back up at 1:30. Our awesome teammates are hoofing lunch over the Brooklyn Bridge for us. As we've said before, this situation is untenable. We can't keep manpower going 24/7 for days.

The building's first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn't powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we're in a wait-and-see posture. Fuel- and water-pumps are in short supply.

The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.

We'll keep you posted and thank you for your continued good wishes.

Thursday, 11:16AM:

Some bad news.

We may not be able to deliver more fuel, as the building is shutdown mode. Last estimate is that we have about 3 hours left. Things change constantly, and we will keep you updated.

Thursday, 2:43PM

We were able to temporarily resume carrying fuel for an hour, which means we have about 4-5 hours of fuel left on the roof.

Datacenter engineers are on site attempting to finish a more permanent pump solution, which we should know more about within the hour.

Thursday, 6:25PM:

We have confirmed that the fuel pump is now working. We can now operate off of fuel indefinitely and there are continuous deliveries scheduled. Additional issues remain, the largest of which being that our generator will need to be taken offline at some point to clean the fuel filter. We can avoid that downtime if Peer1 can provide a street level generator. If this generator failure does occur, we expect the level of downtime to be on the order of an hour, instead of the multi-day outage we were facing at the onset of this crisis. We remain in a difficult situation, but the most difficult challenge we were presented with is now behind us. As of this writing, Squarespace has miraculously avoided all downtime related to Sandy, but we still have many challenges ahead with respect to getting back to anything normal.

Have a good night, guys. You've earned it.