I’m beginning to feel like salesforce.com

and it’s not fun …

Well ATT is trying to put us out of business again. It would not be so stressfull if it was just the RealWorld site and Assessor tax information. But having client sites down is absolutely not acceptable.

As I write this the problem (what ever it is this time) is still not fixed. About 3 months ago a wire pair in the local loop corroded through and the RealWorld site was down for about 4 hours. Today (Thursday) it went down at about 3 PM. It’s approaching 6:30 PM now and Qwest promised to have it fixed within 90 minutes about an hour ago.

I’m beginning to understand what salesforce.com felt like during their recent outage problems. At least we’re still ahead of them for uptime for this year.

I almost drove to Florence to move the server to ComSpan’s Bandon office (they even offered to break their “absolutely no colocation” rule for me – Thanks Joe and John). But, by the time I drove round trip I would not have been able to get the server set up in their office until Friday morning. Better to leave it where it is and make plans for a more robust infrastructure.

Later: The site came back up about 7:00 PM. Turns out it was a failed repeater in ATT’s T1. Not something either they or we could control.

So here is the plan to avoid this in the future

My curent colocation in Florence is switching from an ATT T1 to Charter fiber. At least there won’t be two parties required to fix a problem any more. That should happen in the next week. It will also provide a little faster access.

I spoke with Joe at ComSpan here in Bandon and my fiber to the office should be installed some time in July (hopefully early July – Joe?). Once that is installed I will have two servers on separate networks. The production server will be in Bandon on ComSPan fiber and an identical backup will be on the Charter fiber in Florence. Failover DNS will allow the backup server to pick up most of the site functions if the main server or fiber connection goes down. The maximum downtime in the event of a failure would be maybe 5 minutes. Still would not want a 5 minute failure every 2 months but once a year would get to 5 9s (99.999% uptime).

Barring a tsunami that takes out both Florence and Bandon, we should be in pretty good shape.

Like I say “technology is wonderful … when it works”