Highly Available Serverless Apps With Cloudant's Cross-Region Replication

Jan 10, 2019
serverless ibmcloud cloudant scaling
6 min read

Building highly available serverless applications relies on eliminating “single points of failure” from application architectures.

Existing tutorials showed how to deploy the same serverless application on IBM Cloud in different regions. Using the Global Load Balancer from IBM Cloud Internet Services, traffic is distributed across multiple applications from the same hostname. The Global Load Balancer automatically detects outages in the regional applications and redirects traffics as necessary.

But what if all instances rely on the same database service and that has issues? 😱🔥

In addition to running multiple instances of the application, independent databases in different regions are also necessary for a highly available serverless application. Maintaining consistent application state across regions needs all database changes to be automatically synchronised between instances. 🤔

In this blog post, we’re going to look at using IBM Cloudant’s replication service to set up a “multi-master” replication between regional database instances.

Once this is enabled, database changes will automatically be synchronised in real-time between all database instances. Serverless applications can use their regional database instance and be confident application state will be consistent globally (for some definition of consistent…). 💯

example serverless application - todo backend

This serverless application implements a TODO backend using IBM Cloud Functions and IBM Cloudant.

It provides an REST API for interacting with a TODO service. This can be used with the front-end client to add, complete and remove todos from a list.

Let’s make this example serverless application “highly available”. 👍

The application will be deployed to two different IBM Cloud regions (London and Dallas). Separate database instances will be provisioned in each region. Applications will use their regional database instance but share global state via replication.

deploy serverless app to multiple regions

This Github repo has an automatic deployment script to deploy the serverless application (using wskdeploy) and application services (using terraform).

Install the prerequisites listed here before proceeding with these instructions.

download example application

Clone the Git repository to a local directory.

git clone https://github.com/IBM/ibm-cloud-functions-refarch-serverless-apis

Enter the source code directory.

cd ibm-cloud-functions-refarch-serverless-apis

create IAM key for serverless app

Have you already signed up for an IBM Cloud account and installed the CLI? If not, please do that before proceeding.

Create an IAM key which will be used to deploy the serverless application.

ibmcloud iam api-key-create serverless_api --file serverless_api.apikey

configure deployment variables

Create the local.env file in the current directory will the following contents.

IBMCLOUD_API_KEY=<IAM_API_KEY>
IBMCLOUD_ORG=<YOUR_ORG>
IBMCLOUD_SPACE=<REGION_SPACE>
IBMCLOUD_REGION=
PROVISION_INFRASTRUCTURE=true
API_USE_APPID=false

Replace the <IAM_API_KEY> value with the apikey value from the serverless_api.apikey file.
Replace the <IBMCLOUD_ORG> value with an IBM Cloud organisation.
Replace the <IBMCLOUD_SPACE> value with an IBM Cloud space.

The PROVISION_INFRASTRUCTURE parameter makes the deployment script automatically provision all application resources using Terraform.

Secured API endpoints are not required for this demonstration. Setting the API_USE_APPID parameter to false disables authentication on the endpoints and provisioning the AppID service.

deploy to london

Set the IBMCLOUD_REGION to eu-gb in the local.env file.
Run the following command to deploy the application and provision all application resources.

./deploy.sh --install

If the deployment have succeed, the following message should be printed to the console.

2019-01-08 10:51:51 All done.
ok: APIs
Action                                      Verb  API Name  URL
/<ORG>_<SPACE>/todo_package/todo/get_todo   get   todos     https://<UK_APIGW_URL>/todo
...

Use the TODO front-end application with the APIGW URL shown in the console to interact with the remote TODO service in the London region.

deploy to dallas

Rename the terraform.tfstate file in the infra folder to terraform.tfstate.london
Set the IBMCLOUD_REGION to us-south in the local.env file.
Run the following command to deploy the application and provision all application resources.

./deploy.sh --install

If the deployment have succeed, the following message should be printed to the console.

2019-01-08 10:51:51 All done.
ok: APIs
Action                                      Verb  API Name  URL
/<ORG>_<SPACE>/todo_package/todo/get_todo   get   todos     https://<US_APIGW_URL>/todo
...

Use the TODO front-end application with the APIGW URL shown in the console to interact with the remote TODO service in the Dallas region.

configure cloudant cross-region replication

There are now multiple copies of the same serverless application in different regions. Each region has an independent instance of Cloudant provisioned.

Cloudant replication is a one-way synchronisation from a source to a destination database. To set up a bi-directional data synchronisation, two different replications will need to be configured.

create api keys for replication access

Before configuring replication between the regional databases, API keys need to be created to allow remote access on both hosts. API keys need to be created per regional instance.

From the IBM Cloud Resource List, find the cloudant instances provisioned in London and Dallas.

Open the Cloudant Dashboard for each service instance.

Follow these instructions on both hosts to generate API keys for replication with the correct permissions.

Click the “Databases” icon to show all the databases on this instance.
Click the 🔒 icon in the “todos” database row in the table to open the permissions page.

Can’t find the “todos” database in the Cloudant dashboard? Make sure you interact with the TODO backend from the front-end application. This will automatically create the database if it doesn’t exist.

Click “Generate API Key” on the permissions page.
Make a note of the key identifier and password.
Set the _reader_, _writer and _replicator permissions for the newly created key.

set up cross-region replication

Replication jobs need to be configured on both database hosts. These can be created from the Cloudant dashboard. Repeat these instructions on both hosts.

Open the Cloudant Dashboard for each service instance.
Click the “Replication” icon from the panel menu.
Click the “New Replication” button.
Set the following “Source” values in the “Job configuration” panel.
- Type: “Local Database”
- Name: “todos”
- Authentication: “Cloudant username or API Key”
- Fill in the API key and password for this local database host in the input fields.

Set the following “Target” values in the “Job configuration” panel.
- Type: “Existing Remote Database”
- Name: “https://<REMOTE_CLOUDANT_HOST>/todos”
- Authentication: “Cloudant username or API Key”
- Fill in the API key and password for the remote database host in the input fields.

Wondering what the REMOTE_CLOUDANT_HOST is? Use hostname from the Cloudant dashboard, e.g. XXXX-bluemix.cloudant.com

Set the following “Options” values in the “Job configuration” panel.
- Replication type: “Continuous”

Click “Start Replication”
Verify the replication table shows the new replication task state as “Running”. 👍

test it out!

Use the TODO front-end application with the APIGW URLs for each region simultaneously. Interactions with the todo list in one region should automatically propagate to the other region.

The “Active Tasks” panel on the Cloudant Dashboard shows the documents replicated between instances and pending changes. If there are errors synchronising changes to the replication target, the host uses exponential backoff to re-try the replication tasks.

Conflicts between document changes are handled using CouchDB’s conflict mechanism. Applications are responsible for detecting and resolving document conflicts in the front-end.

conclusion

Running the same serverless application in multiple regions, using the GLB to proxy traffic, allows applications to manage regional outages. But what if all the application instances rely on the same database service? The “single point of failure” has shifted from the application runtime to the database host. 👎

Provisioning independent databases in each application regions is one solution. Applications use their regional database instance and are protected from issues in other regions. This strategy relies on database changes being synchronised between instances to keep the application state consistent. 👍

IBM Cloudant has a built-in replication service to synchronised changes between source and host databases. Setting up bi-directional replication tasks between all instances enables a “multi-master” replication strategy. This allows applications to access any database instance and have the same state available globally. 🕺🕺🕺