In the beginning…

Way back in September, Recurly went down. At first, it seemed like a typical outage that would be resolved without issue in a short span of time. It soon became apparent that this was not the case, and we were unable to bill our customers for over a week. We'd soon learn that the cause of the issue was failure on primary and secondary hardware responsible for storing credit card data resulting in the permanent loss of encryption keys rendering the stored data unreachable. Even after their efforts to retrieve copies of the data from our payment gateway, we were still left missing 5% of our payment info.

As soon as we realized this was not a typical outage, we began planning steps to move off to a different payment provider to get our billing up and running again. After listing a few options, I jokingly suggested:

You know, we could just store it on Recurly and somewhere else. It wouldn't be that hard. We just collect the data, attempt to store it with two different services, and then bill the user on the first service that sticks.

To which Ted responded:

Ha! That would be funny. [short pause] Ok, do it.

Building it

Our first priority was security. We don't ever want to touch actual credit card data (it's better to leave PCI compliance to the companies that specialize in that). We use recurly.js and Braintree Transparent Redirect, so client data never hits our servers.

What was meant to be a joke became our new method of fallback for payment data. We settled on using Recurly (we didn't want to drop support for this and inconvenience all of our current customers by asking them to update their billing information) and Braintree. Without going into too much detail, our subscription process now looks like this:

If storing billing information on either provider fails, we're still able to continue processing as long as the other does not. After our user is subscribed, the rest of our code doesn't care which method was used. In most cases, a subscription is created even in the event of failure on leg of the process. We've also got the ability to migrate our subscriptions should a severe failure ever happen again.

That was a freebie

Adding Braintree as a billing provider had a pleasantly unexpected side effect. Our iOS app (coming soon) uses Braintree to collect billing information. They have a great iOS SDK that saved us a ton of time developing our in-app signup.

Final thoughts

This post is here simply to illustrate that redundantly storing your customers' billing information across multiple services is possible. When building the payment portion of your next project, consider doing this; you'll thank yourself next time one of your services is unavailable. We're very happy with both Recurly and Braintree, but we're also prepared for the next time this happens.