James Thomas

Notes on software.

Faster File Transfers With Serverless

This week I’ve been helping a client speed up file transfers between cloud object stores using serverless.

They had a 120GB file on a cloud provider’s object store. This needed copying into a different cloud object store for integration with platform services. Their current file transfer process was to download the file locally and then re-upload using a development machine. This was taking close to three hours due to bandwidth issues.

Having heard about the capabilities of serverless cloud platforms, they were wondering if they could use the massive parallelism that serverless provides to speed up that process? ๐Ÿค”

After some investigating, I worked out a way to use serverless to implement concurrent file transfers. Transfer time was reduced from THREE HOURS to just FOUR MINUTES! This was a decrease in total transfer time of 98%. ๐Ÿ‘๐Ÿ‘๐Ÿ‘

In this blog post, I’ll outlined the simple steps I used to make this happen. I’ve been using IBM Cloud Functions as the serverless platform. Two different S3-compatible Object Stores were used for the file transfers. The approach should work for any object store with the features outlined below.

S3-Compatible API Features

Both object stores being used for the file transfers provided an S3-compatible API. The S3 API has two features that, when combined, enable concurrent file transfers: Range Reads and Multi-Part Transfers.

Range Reads

The HTTP/1.1 protocol defines a Range header which allows the client to retrieve part of a document. The client specifies a byte range using the header value, e.g. Range: bytes=0-499. The byte values are then returned in the HTTP response with a HTTP 206 status code. If the byte range is invalid, a HTTP 416 response is returned.

The S3 API supports Range request headers on GET HTTP requests for object store files.

Sending a HTTP HEAD request for an object store file will return the file size (using the Content-Length header value). Creating ranges for fixed byte chunks up to this file size (0-1023, 1024-2047,2048-3072 …) allows all sections of a file to be retrieve in parallel.

Multi-Part Transfers

Files are uploaded to buckets using HTTP PUT requests. These operations supports a maximum file size of 5GB. Uploading larger files is only possible using “Multi-Part” transfers.

Clients initiate a multi-part transfer using the API and are returned an upload identifier. The large file is then split into parts which are uploaded using individual HTTP PUT requests. The upload identifier is used to tags individual requests as belonging to the same file. Once all parts have been uploaded, the API is used to confirm the file is finished.

File parts do not have to be uploaded in consecutive order and multiple parts can be uploaded simultaneously.

Serverless File Transfers

Combing these two features, I was able to create a serverless function to copy a part of a file between source and destination buckets. By invoking thousands of these functions in parallel, the entire file could be simultaneously copied in parallel streams between buckets. This was controlled by a local script used to manage the function invocations, monitor progress and complete the multi-part transfer once invocations had finished.

Serverless Function

The serverless function copies a file part between object stores. It is invoked with all the parameters needed to access both bucket files, byte range to copy and multi-part transfer identifier.

1
2
3
4
5
6
exports.main = async function main (params) {
  const { src_bucket, src_file, range, dest_bucket, dest_file, mpu, index} = params
  const byte_range = await read_range(src_bucket, src_file, range)
  const upload_result = await upload_part(dest_bucket, dest_file, mpu, index, byte_range)
  return upload_result
}

Read Source File Part

The S3-API JS client can create a ”Range Read” request by passing the Range parameter with the byte range value, e.g. bytes=0-NN.

1
2
3
4
const read_range = async (Bucket, Key, Range) => {
  const file_range = await s3.getObject({Bucket, Key, Range}).promise()
  return file_range.Body
}

Upload File Part

The uploadPart method is used to complete a part of a multi-part transfer. The method needs the UploadID created when initiating the multi-part transfer and the PartNumber for the chunk index. ETags for the uploaded content will be returned.

1
2
3
4
const upload_part = async (Bucket, Key, UploadId, PartNumber, Body) => {
  const result = await s3.uploadPart({Bucket, Key, UploadId, PartNumber, Body}).promise()
  return result
}

Note: The uploadPart method does not support streaming Body values unless they come from the filesystem. This means the entire part has to be read into memory before uploading. The serverless function must have enough memory to handle this.

Local Script

The local script used to invoke the functions has to do the following things…

  • Create and complete the multi-part transfer
  • Calculate file part byte ranges for function input parameters
  • Copy file parts using concurrent functions invocations.

Create Multi-Part Transfers

The S3-API JS client can be used to create a new Multi-Part Transfer.

1
const { UploadId } = await s3.createMultipartUpload({Bucket: '...', Key: '...'}).promise()

The UploadId can then be used as an input parameter to the serverless function.

Create Byte Ranges

Source file sizes can be retrieved using the client library.

1
2
3
4
const file_size = async (Bucket, Key) => {
  const { ContentLength } = await s3.headObject({Bucket, Key}).promise()
  return ContentLength
}

This file size needs splitting into consecutive byte ranges of fixed size chunks. This function will return an array of the HTTP Range header values (bytes=N-M) needed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const split_into_ranges = (bytes, range_mbs) => {
  const range_size = range_mbs * 1024 * 1024
  const ranges = []
  let range_offset = 0
  const last_byte_range = bytes - 1

  while(range_offset < last_byte_range) {
    const start = range_offset
    // Last byte range may be less than chunk size where file size
    // is not an exact multiple of the chunk size.
    const end = start + Math.min((range_size - 1), last_byte_range - start)
    ranges.push(`bytes=${start}-${end}`)
    range_offset += range_size
  }

  return ranges
}

Invoke Concurrent Functions

Serverless functions need to be invoked for each byte range calculated above. Depending on the file and chunk sizes used, the number of invocations needed could be larger than the platform’s concurrency rate limit (defaults to 1000 on IBM Cloud Functions). In the example above (120GB file in 100MB chunks), 1229 invocations would be needed.

Rather than executing all the byte ranges at once, the script needs to use a maximum of 1000 concurrent invocations. When initial invocations finish, additional functions can be invoked until all the byte ranges have been processed. This code snippet shows a solution to this issue (using IBM Cloud Functions JS SDK).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
const parallel = require('async-await-parallel');
const retry = require('async-retry');
const openwhisk = require('openwhisk');

const concurrent = 1000
const retries = 3
const chunk_size = 100

const static_params = {
  source_bucket, dest_bucket, source_filename, dest_filename, mpu
}

const ow = openwhisk({...});

const bucket_file_size = await file_size(source_bucket, source_filename);
const ranges = split_into_ranges(bucket_file_size, chunk_size);

const uploads = ranges.map((range, index) => {
  const invoke = async () => {
    const params = Object.assign({range, index: index + 1}, static_params)
    const upload_result = await ow.actions.invoke({
      name: '...', blocking: true, result: true, params
    })
    return upload_result
  }

  return async () => retry(invoke, retries)
})

const finished = await parallel(uploads, concurrent)

The uploads value is an array of lazily evaluated serverless function invocations. The code snippet uses the async-await-parallel library to limit the number of concurrent invocations. Handling intermittent or erroneous invocation errors is managed using the async-retry library. Failed invocations will be retried three times.

Finish Multi-Part Transfer

Once all parts have been uploaded, ETags (returned from the serverless invocations) and the Part Numbers are used to complete the multi-part transfer.

1
2
3
4
5
6
7
8
const parts = finished.map((part, idx) => {
  part.PartNumber = idx + 1
  return part
})

const { Location, Bucket, Key, ETag } = await s3.completeMultipartUpload({
  Bucket: '...', Key: '...', UploadId: '...', MultipartUpload: { Parts }
}).promise()

Results

The previous file transfer process (download locally and re-upload from development machine) was taking close to three hours. This was an average throughput rate of 1.33MB/s ((120GB * 2) / 180).

Using serverless functions, the entire process was completed in FOUR MINUTES. File chunks of 100MB were transferred in parallel using 1229 function invocations. This was an average throughput rate of 60MB/s. That was a reduction in total transfer time of ~98%. ๐Ÿ’ฏ๐Ÿ’ฏ๐Ÿ’ฏ

Serverless makes it incredibly easy to run embarrassingly parallel workloads in the cloud. With just a few lines of code, the file transfer process can be parallelised using 1000s of concurrent functions. The client was rather impressed as you can imagine… ๐Ÿ˜Ž

Serverless Functions With WebAssembly Modules

Watching a recent talk by Lin Clark and Till Schneidereit about WebAssembly (Wasm) inspired me to start experimenting with using WebAssembly modules from serverless functions.

This blog post demonstrates how to invoke functions written in C from Node.js serverless functions. Source code in C is compiled to Wasm modules and bundled in the deployment package. Node.js code implements the serverless platform handler and calls native functions upon invocations.

The examples should work (with some modifications) on any serverless platform that supports deploying Node.js functions from a zip file. I’ll be using IBM Cloud Functions (Apache OpenWhisk).

WebAssembly

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust.

https://webassembly.org/

Wasm started as a project to run low-level languages in the browser. This was envisioned as a way to execute computationally intensive tasks in the client, e.g. image manipulation, machine learning, graphics engines. This would improve performance for those tasks compared to using JavaScript.

WebAssembly compiles languages like C, C++ and Rust to a portable instruction format, rather than platform-specific machine code. Compiled Wasm files are interpreted by a Wasm VM in the browser or other runtimes. APIs have been defined to support importing and executing Wasm modules from JavaScript runtimes. These APIs have been implemented in multiple browsers and recent Node.js versions (v8.0.0+).

This means Node.js serverless functions, using a runtime version above 8.0.0, can use WebAssembly!

Wasm Modules + Serverless

“Why would we want to use WebAssembly Modules from Node.js Serverless Functions?” ๐Ÿค”

Performance

Time is literally money with serverless platforms. The faster the code executes, the less it will cost. Using C, C++ or Rust code, compiled to Wasm modules, for computationally intensive tasks can be much faster than the same algorithms implemented in JavaScript.

Easier use of native libraries

Node.js already has a way to use native libraries (in C or C++) from the runtime. This works by compiling the native code during the NPM installation process. Libraries bundled in deployment packages need to be compiled for the serverless platform runtime, not the development environment.

Developers often resort to using specialised containers or VMs, that try to match the runtime environments, for library compilation. This process is error-prone, difficult to debug and a source of problems for developers new to serverless.

Wasm is deliberately platform independent. This means Wasm code compiled locally will work on any Wasm runtime. No more worrying about platform architectures and complex toolchains for native libraries!

Additional runtime support

Dozens of languages now support compiling to WebAssembly.

Want to write serverless functions in Rust, C, or Lua? No problem! By wrapping Wasm modules with a small Node.js handler function, developers can write their serverless applications in any language with “compile to Wasm” support.

Developers don’t have to be restricted to the runtimes provided by the platform.

JS APIs in Node.js

Here is the code needed to load a Wasm module from Node.js. Wasm modules are distributed in .wasm files. Loaded modules are instantiated into instances, by providing a configurable runtime environment. Functions exported from Wasm modules can then be invoked on these instances from Node.js.

1
2
3
4
5
const wasm_module = 'library.wasm'
const bytes = fs.readFileSync(wasm_module)
const wasmModule = new WebAssembly.Module(bytes);
const wasmMemory = new WebAssembly.Memory({initial: 512});
const wasmInstance = new WebAssembly.Instance(wasmModule, { env: { memory: wasmMemory } }})

Calling Functions

Exported Wasm functions are available on the exports property of the wasmInstance. These properties can be invoked as normal functions.

1
const result = wasmInstance.exports.add(2, 2)

Passing & Returning Values

Exported Wasm functions can only receive and return native Wasm types. This (currently) means only integers.

Values that can be represented as a series of numbers, e.g. strings or arrays, can be written directly to the Wasm instance memory heap from Node.js. Heap memory references can be passed as the function parameter values, allowing the Wasm code to read these values. More complex types (e.g. JS objects) are not supported.

This process can also be used in reverse, with Wasm functions returning heap references to pass back strings or arrays with the function result.

For more details on how memory works in Web Assembly, please see this page.

Examples

Having covered the basics, let’s look at some examples…

I’ll start with calling a simple C function from a Node.js serverless function. This will demonstrate the complete steps needed to compile and use a small C program as a Wasm module. Then I’ll look at a more real-world use-case, dynamic image resizing. This will use a C library compiled to Wasm to improve performance.

Examples will be deployed to IBM Cloud Functions (Apache OpenWhisk). They should work on other serverless platforms (supporting the Node.js runtime) with small modifications to the handler function’s interface.

Simple Function Calls

Create Source Files

  • Create a file add.c with the following contents:
1
2
3
int add(int a, int b) {
  return a + b;
}
  • Create a file (index.js) with the following contents:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
'use strict';
const fs = require('fs');
const util = require('util')

const WASM_MODULE = 'add.wasm'
let wasm_instance

async function load_wasm(wasm_module) {
  if (!wasm_instance) {
    const bytes = fs.readFileSync(wasm_module);
    const memory = new WebAssembly.Memory({initial: 1});
    const env = {
      __memory_base: 0, memory
    }

    const { instance, module } = await WebAssembly.instantiate(bytes, { env });
    wasm_instance = instance
  }

  return wasm_instance.exports._add
}

exports.main = async function ({ a = 1, b = 1 }) {
  const add = await load_wasm(WASM_MODULE)
  const sum = add(a, b)
  return { sum }
}
  • Create a file (package.json) with the following contents:
1
2
3
4
5
{
  "name": "wasm",
  "version": "1.0.0",
  "main": "index.js"
}

Compile Wasm Module

This C source file needs compiling to a WebAssembly module. There are different projects to handle this. I will be using Emscripten, which uses LLVM to compile C and C++ to WebAssembly.

  • Install the Emscripten toolchain.

  • Run the following command to generate the Wasm module.

1
emcc -s WASM=1 -s SIDE_MODULE=1 -s EXPORTED_FUNCTIONS="['_add']" -O1 add.c -o add.wasm

The SIDE_MODULE option tells the compiler the Wasm module will be loaded manually using the JS APIs. This stops Emscripten generating a corresponding JS file to do this automatically. Functions exposed on the Wasm module are controlled by the EXPORTED_FUNCTIONS configuration parameter.

Deploy Serverless Function

  • Create deployment package with source files.
1
zip action.zip index.js add.wasm package.json
  • Create serverless function from deployment package.
1
ibmcloud wsk action create wasm action.zip --kind nodejs:10
  • Invoke serverless function to test Wasm module.
1
2
3
4
$ ibmcloud wsk action invoke wasm -r -p a 2 -p b 2
{
    "sum": 4
}

It works! ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

Whilst this is a trivial example, it demonstrates the workflow needed to compile C source files to Wasm modules and invoke exported functions from Node.js serverless functions. Let’s move onto a more realistic example…

Dynamic Image Resizing

This repository contains a serverless function to resize images using a C library called via WebAssembly. It is a fork of the original code created by Cloudflare for their Workers platform. See the original repository for details on what the repository contains and how the files work.

Checkout Repository

  • Retrieve the source files by checking out this repository.
1
git clone https://github.com/jthomas/openwhisk-image-resize-wasm

This repository contains the pre-compiled Wasm module (resize.wasm) needed to resize images using the stb library. The module exposes two functions: init and resize.

The init function returns a heap reference to write the image bytes for processing into. The resize function is called with two values, the image byte array length and new width value. It uses these values to read the image bytes from the heap and calls the library functions to resize the image to the desired width. Resized image bytes are written back to the heap and the new byte array length is returned.

Deploy Serverless Function

  • Create deployment package from source files.
1
zip action.zip resizer.wasm package.json worker.js
  • Create serverless function from deployment package.
1
ibmcloud wsk action update resizer action.zip --kind nodejs:10 --web true
  • Retrieve HTTP URL for Web Action.
1
ibmcloud wsk action get resizer --url

This should return a URL like: https://<region>.cloud.ibm.com/api/v1/web/<ns>/default/resizer

  • Open the Web Action URL with the .http extension.
1
https://<region>.cloud.ibm.com/api/v1/web/<ns>/default/resizer.http

This should return the following image resized to 250 pixels (from 900 pixels).

Pug with Ice-cream

URL query parameters (url and width) can be used to modify the image source or output width for the next image, e.g.

1
https://<region>.cloud.ibm.com/api/v1/web/<ns>/default/resizer.http?url=<IMG_URL>&width=500

Conclusion

WebAssembly may have started as a way to run native code in the browser, but soon expanded to server-side runtime environments like Node.js. WebAssembly modules are supported on any serverless platform with a Node.js v8.0.0+ runtime.

Wasm provides a fast, safe and secure way to ship portable modules from compiled languages. Developers don’t have to worry about whether the module is compiled for the correct platform architecture or linked against unavailable dynamic libraries. This is especially useful for serverless functions in Node.js, where compiling native libraries for production runtimes can be challenging.

Wasm modules can be used to improve performance for computationally intensive calculations, which lowers invocation times and, therefore, costs less. It also provides an easy way to utilise additional runtimes on serverless platforms without any changes by the platform provider.

Hosting Static Websites on IBM Cloud

This blog post explains how to host a static website on IBM Cloud. These websites are rendered client-side by the browser from static assets, like HTML, CSS and JS files. They do not need a server-side component to create pages dynamically at runtime. Static websites are often combined with backend APIs to create Single Page Applications.

Hosting static websites on IBM Cloud uses Cloud Object Storage (COS) and Cloud Internet Services (CIS) (with Page Rules and Edge Functions). These services provide the following features needed to serve static websites.

  • Auto-serving static assets from provider-managed HTTP service (Cloud Object Storage).
  • Custom domain support to serve content from user-controlled domain name (CIS - Page Rules).
  • Configurable Index and Error documents (CIS - Edge Functions).

Here are the steps needed to host a static website on IBM Cloud by combining those services.

Serving static assets

IBM Cloud Object Storage is a scalable storage solution for cloud applications. Files are managed through a RESTful HTTP API and stored in user-defined collections called “buckets”. Bucket files are returned as HTTP responses from HTTP GET requests.

COS supports an optional ”anonymous read-only accesssetting for buckets. This means all files in the bucket will be accessible using anonymous HTTP GET requests.

Putting HTML, CSS and JS files in a public bucket allows static websites to be served directly by COS. Users are charged for bandwidth used and HTTP requests received for all bucket files.

Create IBM Cloud Object Storage instance

If you already have an instance of Cloud Object Storage you can skip this step…

Create IBM Cloud Object Storage Bucket

  • Open the COS instance from the Resource List.
  • Create a new COS bucket to host the static site files.
    • Choose a Bucket name
    • Choose the Resiliency, Location and Storage Class options for the bucket.

Any choices for these options can be used - it does not affect the static site hosting capability. For more details on what they mean, please see this documentation.

Upload Static Assets To Bucket

Enable Public Access to bucket files

  • Click the “Access Policies” menu item from the bucket level menu.
  • Click the ”Public Access” tab from the bucket access policy page.
  • Check the Access Group drop-down has ”Public Access” option selected.
  • Click the ”Create access policy” and then ”Enable” on the pop menu.

Bucket access policy

Check bucket files are accessible

Bucket files should now be accessible using the service endpoint URL, bucket id and file names. COS supports providing the bucket name in the URL path or a sub-domain on the service endpoint.

  • Open the ”Configuration” panel on the bucket page.
  • Retrieve the public endpoint shown, e.g. s3.<REGION>.cloud-object-storage.appdomain.cloud

Public endpoint hostname

Bucket files (like index.html) should now be accessible by a web browser. COS supports both HTTP and HTTPS traffic. Bucket files are available using the following URLs.

vhost addressing

<BUCKET_NANME>.s3.eu-gb.cloud-object-storage.appdomain.cloud/index.html

url path addressing

s3.<REGION>.cloud-object-storage.appdomain.cloud/<BUCKET_NANME>/index.html

Bucket files can now be referenced directly in external web applications. COS buckets are often used to store large application assets like videos or images. For hosting an entire website, it is often necessary to serve content from a custom domain name, rather than the COS bucket hostname.

Custom domain support

Cloud Internet Services Page Rules can automatically configure custom domain support for COS buckets.

CNAME DNS records are created to alias the custom domain to the COS bucket hostname. All traffic to the custom domain will then be forwarded to the COS service.

When COS serves files from bucket sub-domains, the HTTP Host request header value to determine the bucket name. With CNAME DNS records, this header value will still refer to the custom domain, rather than the bucket sub-domain. This field needs to be dynamically updated with the correct value.

Create IBM Cloud Internet Services instance

Register Custom Domain name with Cloud Internet Services

  • Follow the documentation on how to register a custom domain with Cloud Internet Services.

This process involves delegating name server control for the domain over to IBM Cloud Internet Services.

Configure Page Rules and DNS records (automatic)

Cloud Internet Services can automatically set up Page Rules and DNS records needed to forward custom domain traffic to COS buckets. This automatically exposes the bucket as bucket-name.your-domain.com. If you want to change this default sub-domain name, follow the manual steps in the next section.

  • Click the Performance drop-down menu and click the ”Page Rules” link.
  • Click the ”Create rule” button from the table.
  • Select the Rule Behaviour Setting as ”Resolve Override with COS
  • Select the correct COS instance and bucket.
  • Click the ”Create” button.

Auto Page Rules

Once DNS records have propagated, bucket files should be accessible using the custom domain: http(s)://<CUSTOM_DOMAIN>/index.html.

Configure Page Rules and DNS records (manual)

These steps only need following if you haven’t done the section aboveโ€ฆ.

Create the Page Rule to modify the HTTP host header.

  • Click the Performance drop-down menu and select the ”Page Rules” link.
  • Click the ”Create rule” button from the table.
  • Set the URL match field to be <SUB_DOMAIN>.<CUSTOM_DOMAIN>/*
  • Select the Rule Behaviour Setting as ”Host Header Override” as the custom bucket sub-domain:<BUCKET_NANME>.<REGION>.eu-gb.cloud-object-storage.appdomain.cloud

Create the DNS CNAME record to forward traffic to COS.

  • Click the Reliability drop-down menu and click the ”DNS” menu entry.
  • Add a new DNS record with the following values.
    • Type: CNAME
    • Name: <custom subdomain host>
    • TTL: Automatic
    • Alias Domain Name: <COS bucket sub-domain>

Name is the sub-domain on the custom domain (e.g. www) through which the COS bucket will be accessible. Alias Domain Name is the COS bucket sub-domain from above, e.g. <BUCKET_NANME>.<REGION>.eu-gb.cloud-object-storage.appdomain.cloud

  • Once the record is added, set the Proxy field to true. This is necessary for the page rules to work.

Once DNS records have propagated, bucket files should be accessible using the custom domain.

Configurable Index and Error pages

COS will now serve static assets from a custom sub-domain, where file names are explicitly included in the URL, e.g. http(s)://<CUSTOM_DOMAIN>/index.html. This works fine for static websites with two exceptions, the default document for the web site and the error page.

When a user visits the COS bucket sub-domain without an explicit file path (http(s)://<CUSTOM_DOMAIN>), the COS service will return the bucket file list, rather than the site index page. Additionally, if a user requests a missing file, COS returns an XML error message rather than a custom error page.

Both issues can be resolved using Edge Functions, a new feature in Cloud Internet Services.

Edge Functions

Edge functions are JavaScript source files deployed to Cloudflare’s Edge locations. They can dynamically modify HTTP traffic passing through Cloudflare’s network (for domains you control). Custom edge functions are triggered on configurable URL routes. Functions are passed the incoming HTTP request and control the HTTP response returned.

Add Edge Function to provide Index & Error Documents

Using a custom edge function, HTTP traffic to the custom sub-domain can be modified to support Index and Error documents. Incoming HTTP requests without an explicit file name can be changed to use the index page location. HTTP 404 responses returned from COS can be replaced with a custom error page.

  • Open the ”Edge Functions” page from the Cloud Internet Services instance homepage.
  • Click the ”Create” icon on the ”Actions” tab.
  • Enter ”route-index-and-errors” in the action name field.
  • Paste the following source code into the action body section.

The INDEX_DOCUMENT and ERROR_DOCUMENT values control the index and error pages used to redirect requests. Replace these values with the correct page locations for the static site being hosted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
const INDEX_DOCUMENT = 'index.html'
const ERROR_DOCUMENT = '404.html'

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)

  // if request is a directory path, append the index document.
  if (url.pathname.endsWith('/')) {
    url.pathname = `${url.pathname}${INDEX_DOCUMENT}`
    request = new Request(url, request)
  }

  let response = await fetch(request)

  // if bucket file is missing, return error page.
  if (response.status === 404) {
    url.pathname = ERROR_DOCUMENT
    request = new Request(url, request)
    response = await fetch(request)

    response = new Response(response.body, {
      status: 404,
      statusText: 'Not Found',
      headers: response.headers
    })
  }

  return response
}
  • Click the ”Save” button.

Set up Triggers for Edge Function

  • Select the ”Triggers” panel from the Edge Functions page.
  • Click the ”Add trigger” icon.
  • Set the Trigger URL to http://<SUB_DOMAIN>.<CUSTOM_DOMAIN>/*.
  • Select the ”route-index-and-errors” action from the drop-down menu.
  • Click the ”Save” button.

Test Index and Error Pages

Having set up the trigger and edge function, HTTP requests to the root path on the custom sub-domain will return the index page. Accessing invalid bucket files will also return the error page, rather than the COS error response.

  • Confirm that http://<SUB_DOMAIN>.<CUSTOM_DOMAIN>/ returns the same page as http://<SUB_DOMAIN>.<CUSTOM_DOMAIN>/index.html
  • Confirm that http://<SUB_DOMAIN>.<CUSTOM_DOMAIN>/missing-page.html returns the error page. This should be different to the XML error response returned by visiting <BUCKET_NANME>.s3.<REGION>.cloud-object-storage.appdomain.cloud/missing-page.html.

If this all works - the site is working! IBM Cloud is now hosting a static website using Cloud Object Storage and Cloud Internet Services with Page Rules and Edge Functions. ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

Summary

Static web sites can be hosted on IBM Cloud using Cloud Object Storage and Cloud Internet Services.

Cloud Object stores page files needed to render the static website. Anonymous bucket file access means files are accessible as public HTTP endpoints, without having to run infrastructure to serve the assets.

Cloud Internet Services forwards HTTP traffic from a custom domain to the bucket hostname. DNS CNAME records are used to resolve the sub-domain as the custom bucket hostname. Page Rules override HTTP request headers to make this work. Edge Functions are used to implement configurable Index and Error documents, by dynamically modifying in-flight requests with custom JavaScript.

Hosting static web sites using this method can be much cheaper (and easier) than traditional infrastructure. Developers only get charged for actual site usage, based on bandwidth and HTTP requests.

Connecting to IBM Cloud Databases for Redis From Node.js

This blog post explains how to connect to an IBM Cloud Databases for Redis instance from a Node.js application. There is a (small) difference between the connection details needed for an IBM Cloud Databases for Redis instance compared to a local instance of the open-source database. This is due to all IBM Cloud Databases using secured TLS connections with self-signed certificates.

I keep running into this issue (and forgetting how to fix it ๐Ÿคฆโ€โ™‚๏ธ), so I’m documenting the solution here to help myself (and others) who might run into itโ€ฆ ๐Ÿฆธโ€โ™‚๏ธ

Connecting to Redis (without TLS connections)

Most Node.js application use the redis NPM library to interact with an instance of the database. This library has a createClient method which returns an instance of the client. The Node.js application passes a connection string into the createClient method. This string contains the hostname, port, username and password for the database instance.

1
2
3
const redis = require("redis"),
const url = 'redis://user:secret@localhost:6379/'
const client = redis.createClient(url);

The client fires a connect event once the connection is established or an error event if issues are encountered.

IBM Cloud Databases for Redis Service Credentials

IBM Cloud Databases for Redis provide service credentials through the instance management console. Service credentials are JSON objects with connection properties for client libraries, the CLI and other tools. Connection strings for the Node.js client library are available in the connection.rediss.composed field.

So, I just copy this field value and use with the redis.createClient method? Not so fast…

IBM Cloud Databases for Redis uses TLS to secure all connections to the Redis instances. This is denoted by the connection string using the rediss:// URL prefix, rather than redis://. Using that connection string (without further connection properties), will lead to the following error being thrown by the Node.js application.

1
2
Error: Redis connection to <id>.databases.appdomain.cloud:port failed - read ECONNRESET
  at TCP.onread (net.js:657:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read'

If the createClient forces a TLS connection to be used createClient(url, { tls: {} }), this error will be replaced with a different one about self-signed certificates.

1
2
3
4
Error: Redis connection to <id>.databases.appdomain.cloud:port failed failed - self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (_tls_wrap.js:1055:34)
    at TLSSocket.emit (events.js:182:13)
    at TLSSocket._finishInit (_tls_wrap.js:635:8) code: 'SELF_SIGNED_CERT_IN_CHAIN'

Hmmmm, how to fix this? ๐Ÿค”

Connecting to Redis (with TLS connections)

All connections to IBM Cloud Databases are secured with TLS using self-signed certificates. Public certificates for the signing authorities are provided as Base64 strings in the service credentials. These certificates can be provided in the client constructor to support self-signed TLS connections.

Here are the steps needed to use those self-signed certificates with the client library…

  • Extract the connection.rediss.certificate.certificate_base64 value from the service credentials.

Redis Service Credentials

  • Decode the Base64 string in Node.js to extract the PEM certificate string.
1
const ca = Buffer.from(cert_base64, 'base64').toString('utf-8')
  • Provide the certificate file string as the ca property in the tls object for the client constructor.
1
2
const tls = { ca };
const client = redis.createClient(url, { tls });
  • โ€ฆRelax! ๐Ÿ˜Ž

The tls property is passed through to the tls.connect method in Node.js, which is used to setup the TLS connection. This method supports a ca parameter to extend the trusted CA certificates pre-installed in the system. By providing the self-signed certificate using this property, the errors above will not be seen.

Conclusion

It took me a while to work out how to connect to TLS-secured Redis instances from a Node.js application. Providing the self-signed certificate in the client constructor is a much better solution than having to disable all unauthorised TLS connections!

Since I don’t write new Redis client code very often, I keep forgetting the correct constructor parameters to make this work. Turning this solution into a blog post will (hopefully) embed it in my brain (or at least provide a way to find the answer instead of having to grep through old project code). This might even be useful to others Googling for a solution to those error messages…

Serverless APIs for MAX Models

IBM’s Model Asset eXchange provides a curated list of free Machine Learning models for developers. Models currently published include detecting emotions or ages in faces from images, forecasting the weather, converting speech to text and more. Models are pre-trained and ready for use in the cloud.

Models are published as series of public Docker images. Images automatically expose a HTTP API for model predictions. Documentation in the model repositories explains how to run images locally (using Docker) or deploy to the cloud (using Kubernetes). This got me thinkingโ€ฆ

Could MAX models be used from serverless functions? ๐Ÿค”

Running machine learning models on serverless platforms can take advantage of the horizontal scalability to process large numbers of computationally intensive classification tasks in parallel. Coupled with the serverless pricing structure (”no charge for idle”), this can be an extremely cheap and effective way to perform model classifications in the cloud.

CHALLENGE ACCEPTED! ๐Ÿฆธโ€โ™‚๏ธ๐Ÿฆธโ€โ™€๏ธ

After a couple days of experimentation, I had worked out an easy way to automatically expose MAX models as Serverless APIs on IBM Cloud Functions. ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

I’ve given instructions below on how to create those APIs from the models using a simple script. If you just want to use the models, follow those instructions. If you are interested in understanding how this works, keep reading as I explain afterwards what I did…

Running MAX models on IBM Cloud Functions

This repository contains a bash script which builds custom Docker runtimes with MAX models for usage on IBM Cloud Functions. Pushing these images to Docker Hub allows IBM Cloud Functions to use them as custom runtimes. Web Actions created from these custom runtime images expose the same Prediction API described in the model documentation. They can be used with no further changes or custom code needed.

prerequisites

Please follow the links below to set up the following tools before proceeding.

Check out the ”Serverless MAX Models repository. Run all the following commands from that folder.

1
2
git clone https://github.com/jthomas/serverless-max-models 
cd serverless-max-models 

build custom runtime images

  • Set the following environment variables (MODELS) with MAX model names and run build script.
    • MODELS: MAX model names, e.g. max-facial-emotion-classifier
    • USERNAME: Docker Hub username.
1
MODELS="..." USERNAME="..." ./build.sh

This will create Docker images locally with the MAX model names and push to Docker Hub for usage in IBM Cloud Functions. IBM Cloud Functions only supports public Docker images as custom runtimes.

create actions using custom runtimes

1
ibmcloud wsk action create <MODEL_IMAGE> --docker <DOCKERHUB_NAME>/<MODEL_IMAGE> --web true -m 512
  • Retrieve the Web Action URL (https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/<ACTION>)
1
ibmcloud wsk action get <MODEL_IMAGE> --url

invoke web action url with prediction api parameters

Use the same API request parameters as defined in the Prediction API specification with the Web Action URL. This will invoke model predictions and return the result as the HTTP response, e.g.

1
curl -F "image=@assets/happy-baby.jpeg" -XPOST <WEB_ACTION_URL>

NOTE: The first invocation after creating an action may incur long cold-start delays due to the platform pulling the remote image into the local registry. Once the image is available in the platform, both further cold and warm invocations will be much faster.

Example

Here is an example of creating a serverless API using the max-facial-emotion-classifier MAX model. Further examples of models which have been tested are available here. If you encounter problems, please open an issue on Github.

max-facial-emotion-classifier

Start by creating the action using the custom runtime and then retrieve the Web Action URL.

1
2
3
4
5
$ ibmcloud wsk action create max-facial-emotion-classifier --docker <DOCKERHUB_NAME>/max-facial-emotion-classifier --web true -m 512
ok: created action max-facial-emotion-classifier
$ ibmcloud wsk action get max-facial-emotion-classifier --url
ok: got action max-facial-emotion-classifier
https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/max-facial-emotion-classifier

According to the API definition for this model, the prediction API expects a form submission with an image file to classify. Using a sample image from the model repo, the model can be tested using curl.

1
$ curl -F "image=@happy-baby.jpeg" -XPOST https://<REGION>.functions.cloud.ibm.com/api/v1/web/<NS>/default/max-facial-emotion-classifier
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "status": "ok",
  "predictions": [
    {
      "detection_box": [
        0.15102639296187684,
        0.3828125,
        0.5293255131964809,
        0.5830078125
      ],
      "emotion_predictions": [
        {
          "label_id": "1",
          "label": "happiness",
          "probability": 0.9860254526138306
        },
        ...
      ]
    }
  ]
}

performance

Example Invocation Duration (Cold): ~4.8 seconds

Example Invocation Duration (Warm): ~ 800 ms

How does this work?

background

Running machine learning classifications using pre-trained models from serverless functions has historically been challenging due to the following reasonโ€ฆ

Developers do not control runtime environments in (most) serverless cloud platforms. Libraries and dependencies needed by the functions must be provided in the deployment package. Most platforms limit deployment package sizes (~50MB compressed & ~250MB uncompressed).

Machine Learning libraries and models can be much larger than those deployment size limits. This stops them being included in deployment packages. Loading files dynamically during invocations may be possible but incurs extremely long cold-start delays and additional costs.

Fortunately, IBM Cloud Functions is based on the open-source serverless project, Apache OpenWhisk. This platform supports bespoke function runtimes using custom Docker images. Machine learning libraries and models can therefore be provided in custom runtimes. This removes the need to include them in deployment packages or be loaded at runtime.

Interested in reading other blog posts about using machine learning libraries and toolkits with IBM Cloud Functions? See these posts for more details.

MAX model images

IBM’s Model Asset eXchange publishes Docker images for each model, alongside the pre-trained model files. Images expose a HTTP API for predictions using the model on port 5000, built using Python and Flask. Swagger files for the APIs describe the available operations, input parameters and response bodies.

These images use a custom application framework (maxfw), based on Flask, to standardise exposing MAX models as HTTP APIs. This framework handles input parameter validation, response marshalling, CORS support, etc. This allows model runtimes to just implement the prediction API handlers, rather than the entire HTTP application.

Since the framework already handles exposing the model as a HTTP API, I started looking for a way to simulate an external HTTP request coming into the framework. If this was possible, I could trigger this fake request from a Python Web Action to perform the model classification from input parameters. The Web Action would then covert the HTTP response returned into the valid Web Action response parameters.

flask test client

Reading through the Flask documentation, I came across the perfect solution! ๐Ÿ‘๐Ÿ‘๐Ÿ‘

Flask provides a way to test your application by exposing the Werkzeug test Client and handling the context locals for you. You can then use that with your favourite testing solution.

This allows application routes to be executed with the test client, without actually running the HTTP server.

1
2
3
4
max_app = MAXApp(API_TITLE, API_DESC, API_VERSION)
max_app.add_api(ModelPredictAPI, '/predict')
test_client = max_app.app.test_client()
r = test_client.post('/model/predict', data=content, headers=headers)

Using this code within a serverless Python function allows function invocations to trigger the prediction API. The serverless function only has to convert input parameters to the fake HTTP request and then serialise the response back to JSON.

python docker action

The custom MAX model runtime image needs to implement the HTTP API expected by Apache OpenWhisk. This API is used to instantiate the runtime environment and then pass in invocation parameters on each request. Since the runtime image contains all files and code need to process requests, the /init handler becomes a no-op. The /run handler converts Web Action HTTP parameters into the fake HTTP request.

Here is the Python script used to proxy incoming Web Actions requests to the framework model service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
from maxfw.core import MAXApp
from api import ModelPredictAPI
from config import API_TITLE, API_DESC, API_VERSION
import json
import base64
from flask import Flask, request, Response

max_app = MAXApp(API_TITLE, API_DESC, API_VERSION)
max_app.add_api(ModelPredictAPI, '/predict')

# Use flask test client to simulate HTTP requests for the prediction APIs
# HTTP request data will come from action invocation parameters, neat huh? :)
test_client = max_app.app.test_client()
app = Flask(__name__)

# This implements the Docker runtime API used by Apache OpenWhisk
# https://github.com/apache/incubator-openwhisk/blob/master/docs/actions-docker.md
# /init is a no-op as everything is provided in the image.
@app.route("/init", methods=['POST'])
def init():
    return ''

# Action invocation requests will be received as the `value` parameter in request body.
# Web Actions provide HTTP request parameters as `__ow_headers` & `__ow_body` parameters.
@app.route("/run", methods=['POST'])
def run():
    body = request.json
    form_body = body['value']['__ow_body']
    headers = body['value']['__ow_headers']

    # binary image content provided as base64 strings
    content = base64.b64decode(form_body)

    # send fake HTTP request to prediction API with invocation data
    r = test_client.post('/model/predict', data=content, headers=headers)
    r_headers = dict((x, y) for x, y in r.headers)

    # binary data must be encoded as base64 strings to return in JSON response
    is_image = r_headers['Content-Type'].startswith('image')
    r_data = base64.b64encode(r.data) if is_image else r.data
    body = r_data.decode("utf-8")

    response = {'headers': r_headers, 'status': r.status_code, 'body': body }
    print (r.status)
    return Response(json.dumps(response), status=200, mimetype='application/json')

app.run(host='0.0.0.0', port=8080)

building into an image

Since the MAX models already exist as public Docker images, those images can be used as base images when building custom runtimes. Those base images handle adding model files and all dependencies needed to execute them into the image.

This is the Dockerfile used by the build script to create the custom model image. The model parameter refers to the build argument containing the model name.

1
2
3
4
5
6
7
8
ARG model
FROM codait/${model}:latest

ADD openwhisk.py .

EXPOSE 8080

CMD python openwhisk.py

This is then used from the following build script to create a custom runtime image for the model.

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

set -e -u

for model in $MODELS; do
  echo "Building $model runtime image"
  docker build -t $model --build-arg model=$model .
  echo "Pushing $model to Docker Hub"
  docker tag $model $USERNAME/$model
  docker push $USERNAME/$model
done

Once the image is published to Docker Hub, it can be referenced when creating new Web Actions (using the โ€”docker parameter). ๐Ÿ˜Ž

1
ibmcloud wsk action create <MODEL_IMAGE> --docker <DOCKERHUB_NAME>/<MODEL_IMAGE> --web true -m 512

Conclusion

IBM’s Model Asset eXchange is a curated collection of Machine Learning models, ready to deploy to the cloud for a variety of tasks. All models are available as a series of public Docker images. Models images automatically expose HTTP APIs for classifications.

Documentation in the model repositories explains how to run them locally and deploy using Kubernetes, but what about using on serverless cloud platforms? Serverless platforms are becoming a popular option for deploying Machine Learning models, due to horizontal scalability and cost advantages.

Looking through the source code for the model images, I discovered a mechanism to hook into the custom model framework used to export the model files as HTTP APIs. This allowed me write a simple wrapper script to proxy serverless function invocations to the model prediction APIs. API responses would be serialised back into the Web Action response format.

Building this script into a new Docker image, using the existing model image as the base image, created a new runtime which could be used on the platform. Web Actions created from this runtime image would automatically expose the same HTTP APIs as the existing image!

Accessing Long-Running Apache OpenWhisk Actions Results

Apache OpenWhisk actions are invoked by sending HTTP POST requests to the platform API. Invocation requests have two different modes: blocking and non-blocking.

Blocking invocations mean the platform won’t send the HTTP response until the action finishes. This allows it to include the action result in the response. Blocking invocations are used when you want to invoke an action and wait for the result.

1
2
3
4
5
6
7
8
9
10
11
12
$ wsk action invoke my_action --blocking
ok: invoked /_/my_action with id db70ef682fae4f8fb0ef682fae2f8fd5
{
    "activationId": "db70ef682fae4f8fb0ef682fae2f8fd5",
    ...
    "response": {
        "result": { ... },
        "status": "success",
        "success": true
    },
    ...
}

Non-blocking invocations return as soon as the platform processes the invocation request. This is before the action has finished executing. HTTP responses from non-blocking invocations only include activation identifiers, as the action result is not available.

1
2
$ wsk action invoke my_action
ok: invoked /_/my_action with id d2728aaa75394411b28aaa7539341195

HTTP responses from a blocking invocation will only wait for a limited amount of time before returning. This defaults to 65 seconds in the platform configuration file. If an action invocation has not finished before this timeout limit, a HTTP 5xx status response is returned.

Hmmmโ€ฆ ๐Ÿค”

“So, how can you invoke an action and wait for the result when actions take longer than this limit?”

This question comes up regularly from developers building applications using the platform. I’ve decided to turn my answer into a blog post to help others struggling with this issue (after answering this question again this week ๐Ÿ˜Ž).

solution

  • Invoke the action using a non-blocking invocation.
  • Use the returned activation identifier to poll the activation result API.
  • The HTTP response for the activation result will return a HTTP 404 response until the action finishes.

When polling for activation results from non-blocking invocations, you should enforce a limit on the maximum polling time allowed. This is because HTTP 404s can be returned due to other scenarios (e.g. invalid activation identifiers). Enforcing a time limit ensures that, in the event of issues in the application code or the platform, the polling loop with eventually stop!

Setting the maximum polling time to the action timeout limit (plus a small offset) is a good approach.

An action cannot run for longer than its timeout limit. If the activation record is not available after this duration has elapsed (plus a small offset to handle internal platform delays), something has gone wrong. Continuing to poll after this point runs the risk of turning the polling operation into an infinite loop…

example code

This example provides an implementation of this approach for Node.js using the JavaScript Client SDK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
"use strict";

const openwhisk = require('openwhisk')

const options = { apihost: <API_HOST>, api_key: <API_KEY> }
const ow = openwhisk(options)

// action duration limit (+ small offset)
const timeout_ms = 85000
// delay between polling requests
const polling_delay = 1000
// action to invoke
const action = 'delay'

const now = () => (new Date().getTime())
const max_polling_time = now() + timeout_ms

const delay = async ms => new Promise(resolve => setTimeout(resolve, ms))

const activation = await ow.actions.invoke({name: action})
console.log(`new activation id: ${activation.activationId}`)

let result = null

do {
  try {
    result = await ow.activations.get({ name: activation.activationId })
    console.log(`activation result (${activation.activationId}) now available!`)
  } catch (err) {
    if (err.statusCode !== 404) {
      throw err
    }
    console.log(`activation result (${activation.activationId}) not available yet`)
  }

  await delay(polling_delay)
} while (!result && now() < max_polling_time)

console.log(`activation result (${activation.activationId})`, result)

testing it out

Here is the source code for an action which will not return until 70 seconds have passed. Blocking invocations firing this action will result in a HTTP timeout before the response is returned.

1
2
3
4
5
const delay = async ms => new Promise(resolve => setTimeout(resolve, ms))

function main() {
  return delay(70*1000)
}

Using the script above, the action result will be retrieved from a non-blocking invocation.

  • Create an action from the source file in the example above.
1
wsk action create delay delay.js --timeout 80000 --kind nodejs:10
  • Run the Node.js script to invoke this action and poll for the activation result.
1
node script.js

If the script runs correctly, log messages will display the polling status and then the activation result.

1
2
3
4
5
6
7
8
$ node script.js
new activation id: d4efc4641b544320afc4641b54132066
activation result (d4efc4641b544320afc4641b54132066) not available yet
activation result (d4efc4641b544320afc4641b54132066) not available yet
activation result (d4efc4641b544320afc4641b54132066) not available yet
...
activation result (d4efc4641b544320afc4641b54132066) now available!
activation result (d4efc4641b544320afc4641b54132066) { ... }

Saving Money and Time With Node.js Worker Threads in Serverless Functions

Node.js v12 was released last month. This new version includes support for Worker Threads, that are enabled by default. Node.js Worker Threads make it simple to execute JavaScript code in parallel using threads. ๐Ÿ‘๐Ÿ‘๐Ÿ‘

This is useful for Node.js applications with CPU-intensive workloads. Using Worker Threads, JavaScript code can be executed code concurrently using multiple CPU cores. This reduces execution time compared to a non-Worker Threads version.

If serverless platforms provide Node.js v12 on multi-core environments, functions can use this feature to reduce execution time and, therefore, lower costs. Depending on the workload, functions can utilise all available CPU cores to parallelise work, rather than executing more functions concurrently. ๐Ÿ’ฐ๐Ÿ’ฐ๐Ÿ’ฐ

In this blog post, I’ll explain how to use Worker Threads from a serverless function. I’ll be using IBM Cloud Functions (Apache OpenWhisk) as the example platform but this approach is applicable for any serverless platform with Node.js v12 support and a multi-core CPU runtime environment.

Node.js v12 in IBM Cloud Functions (Apache OpenWhisk)

This section of the blog post is specifically about using the new Node.js v12 runtime on IBM Cloud Functions (powered by Apache OpenWhisk). If you are using a different serverless platform, feel free to skip ahead to the next sectionโ€ฆ

I’ve recently been working on adding the Node.js v12 runtime to Apache OpenWhisk.

Apache OpenWhisk uses Docker containers as runtime environments for serverless functions. All runtime images are maintained in separate repositories for each supported language, e.g. Node.js, Java, Python, etc. Runtime images are automatically built and pushed to Docker Hub when the repository is updated.

node.js v12 runtime image

Here is the PR used to add the new Node.js v12 runtime image to Apache OpenWhisk. This led to the following runtime image being exported to Docker Hub: openwhisk/action-nodejs-v12.

Having this image available as a native runtime in Apache OpenWhisk requires upstream changes to the project’s runtime manifest. After this happens, developers will be able to use the --kind CLI flag to select this runtime version.

1
ibmcloud wsk action create action_name action.js --kind nodejs:12

IBM Cloud Functions is powered by Apache OpenWhisk. It will eventually pick up the upstream project changes to include this new runtime version. Until that happens, Docker support allows usage of this new runtime before it is built-in the platform.

1
ibmcloud wsk action create action_name action.js --docker openwhisk/action-nodejs-v12

example

This Apache OpenWhisk action returns the version of Node.js used in the runtime environment.

1
2
3
4
5
function main () {
  return {
    version: process.version
  }
}

Running this code on IBM Cloud Functions, using the Node.js v12 runtime image, allows us to confirm the new Node.js version is available.

1
2
3
4
5
6
$ ibmcloud wsk action create nodejs-v12 action.js --docker openwhisk/action-nodejs-v12
ok: created action nodejs-v12
$ ibmcloud wsk action invoke nodejs-v12 --result
{
    "version": "v12.1.0"
}

Worker Threads in Serverless Functions

This is a great introdution blog post to Workers Threads. It uses an example of generating prime numbers as the CPU intensive task to benchmark. Comparing the performance of the single-threaded version to multiple-threads - the performance is improved as a factor of the threads used (up to the number of CPU cores available).

This code can be ported to run in a serverless function. Running with different input values and thread counts will allow benchmarking of the performance improvement.

non-workers version

Here is the sample code for a serverless function to generate prime numbers. It does not use Worker Threads. It will run on the main event loop for the Node.js process. This means it will only utilise a single thread (and therefore single CPU core).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
'use strict';

const min = 2

function main(params) {
  const { start, end } = params
  console.log(params)
  const primes = []
  let isPrime = true;
  for (let i = start; i < end; i++) {
    for (let j = min; j < Math.sqrt(end); j++) {
      if (i !== j && i%j === 0) {
        isPrime = false;
        break;
      }
    }
    if (isPrime) {
      primes.push(i);
    }
    isPrime = true;
  }

  return { primes }
}

porting the code to use worker threads

Here is the prime number calculation code which uses Worker Threads. Dividing the total input range by the number of Worker Threads generates individual thread input values. Worker Threads are spawned and passed chunked input ranges. Threads calculate primes and then send the result back to the parent thread.

Reviewing the code to start converting it to a serverless function, I realised there were two issues running this code in serverless environment: worker thread initialisation and optimal worker thread counts.

How to initialise Worker Threads?

This is how the existing source code initialises the Worker Threads.

1
 threads.add(new Worker(__filename, { workerData: { start: myStart, range }}));

__filename is a special global variable in Node.js which contains the currently executing script file path.

This means the Worker Thread will be initialised with a copy of the currently executing script. Node.js provides a special variable to indicate whether the script is executing in the parent or child thread. This can be used to branch script logic.

So, what’s the issue with this?

In the Apache OpenWhisk Node.js runtime, action source files are dynamically imported into the runtime environment. The script used to start the Node.js runtime process is for the platform handler, not the action source files. This means the __filename variable does not point to the action source file.

This issue is fixed by separating the serverless function handler and worker thread code into separate files. Worker Threads can be started with a reference to the worker thread script source file, rather than the currently executing script name.

1
 threads.add(new Worker("./worker.js", { workerData: { start: myStart, range }}));

How Many Worker Threads?

The next issue to resolve is how many Worker Threads to use. In order to maximise parallel processing capacity, there should be a Worker Thread for each CPU core. This is the maximum number of threads that can run concurrently.

Node.js provides CPU information for the runtime environment using the os.cpus() function. The result is an array of objects (one per logical CPU core), with model information, processing speed and elapsed processing times. The length of this array will determine number of Worker Threads used. This ensures the number of Worker Threads will always match the CPU cores available.

1
const threadCount = os.cpus().length

workers threads version

Here is the serverless version of the prime number generation algorithm which uses Worker Threads.

The code is split over two files - primes-with-workers.js and worker.js.

primes-with-workers.js

This file contains the serverless function handler used by the platform. Input ranges (based on the min and max action parameters) are divided into chunks, based upon the number of Worker Threads. The handler function creates a Worker Thread for each chunk and waits for the message with the result. Once all the results have been retrieved, it returns all those primes numbers as the invocation result.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
'use strict';

const { Worker } = require('worker_threads');
const os = require('os')
const threadCount = os.cpus().length

const compute_primes = async (start, range) => {
  return new Promise((resolve, reject) => {
    let primes = []
    console.log(`adding worker (${start} => ${start + range})`)
    const worker = new Worker('./worker.js', { workerData: { start, range }})

    worker.on('error', reject)
    worker.on('exit', () => resolve(primes))
    worker.on('message', msg => {
      primes = primes.concat(msg)
    })
  })
}

async function main(params) {
  const { min, max } = params
  const range = Math.ceil((max - min) / threadCount)
  let start = min < 2 ? 2 : min
  const workers = []

  console.log(`Calculating primes with ${threadCount} threads...`);

  for (let i = 0; i < threadCount - 1; i++) {
    const myStart = start
    workers.push(compute_primes(myStart, range))
    start += range
  }

  workers.push(compute_primes(start, max - start))

  const primes = await Promise.all(workers)
  return { primes: primes.flat() }
}

exports.main = main

workers.js

This is the script used in the Worker Thread. The workerData value is used to receive number ranges to search for prime numbers. Primes numbers are sent back to the parent thread using the postMessage function. Since this script is only used in the Worker Thread, it does need to use the isMainThread value to check if it is a child or parent process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
'use strict';
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');

const min = 2

function generatePrimes(start, range) {
  const primes = []
  let isPrime = true;
  let end = start + range;
  for (let i = start; i < end; i++) {
    for (let j = min; j < Math.sqrt(end); j++) {
      if (i !== j && i%j === 0) {
        isPrime = false;
        break;
      }
    }
    if (isPrime) {
      primes.push(i);
    }
    isPrime = true;
  }

  return primes
}

const primes = generatePrimes(workerData.start, workerData.range);
parentPort.postMessage(primes)

package.json

Source files deployed from a zip file also need to include a package.json file in the archive. The main property is used to determine the script to import as the exported package module.

1
2
3
4
5
{
  "name": "worker_threads",
  "version": "1.0.0",
  "main": "primes-with-workers.js",
}

Performance Comparison

Running both functions with the same input parameters allows execution time comparison. The Worker Threads version should improve performance by a factor proportional to available CPU cores. Reducing execution time also means reduced costs in a serverless platform.

non-workers performance

Creating a new serverless function (primes) from the non-worker threads source code, using the Node.js v12 runtime, I can test with small values to check correctness.

1
2
3
4
5
6
$ ibmcloud wsk action create primes primes.js --docker openwhisk/action-nodejs-v12
ok: created action primes
$ ibmcloud wsk action invoke primes --result -p start 2 -p end 10
{
    "primes": [ 2, 3, 5, 7 ]
}

Playing with sample input values, 10,000,000 seems like a useful benchmark value. This takes long enough with the single-threaded version to benefit from parallelism.

1
2
3
4
5
$ time ibmcloud wsk action invoke primes --result -p start 2 -p end 10000000 > /dev/null

real  0m35.151s
user  0m0.840s
sys   0m0.315s

Using the simple single-threaded algorithm it takes the serverless function around ~35 seconds to calculate primes up to ten million.

workers threads performance

Creating a new serverless function, from the worker threads-based source code using the Node.js v12 runtime, allows me to verify it works as expected for small input values.

1
2
3
4
5
6
$ ibmcloud wsk action create primes-workers action.zip --docker openwhisk/action-nodejs-v12
ok: created action primes-workers
$ ibmcloud wsk action invoke primes-workers --result -p min 2 -p max 10
{
    "primes": [ 2, 3, 5, 7 ]
}

Hurrah, it works.

Invoking the function with an max parameter of 10,000,000 allows us to benchmark against the non-workers version of the code.

1
2
3
4
5
$ time ibmcloud wsk action invoke primes-workers --result -p min 2 -p max 10000000 --result > /dev/null

real  0m8.863s
user  0m0.804s
sys   0m0.302s

The workers versions only takes ~25% of the time of the single-threaded version!

This is because IBM Cloud Functions’ runtime environments provide access to four CPU cores. Unlike other platforms, CPU cores are not tied to memory allocations. Utilising all available CPU cores concurrently allows the algorithm to run 4x times as fast. Since serverless platforms charge based on execution time, reducing execution time also means reducing costs.

The worker threads version also costs 75% less than the single-threaded version!

Conclusion

Node.js v12 was released in April 2019. This version included support for Worker Threads, that were enabled by default (rather than needing an optional runtime flag). Using multiple CPU cores in Node.js applications has never been easier!

Node.js applications with CPU-intensive workloads can utilise this feature to reduce execution time. Since serverless platforms charge based upon execution time, this is especially useful for Node.js serverless functions. Utilising multiple CPU cores leads, not only to improved performance, but also lower bills.

PRs have been opened to enable Node.js v12 as a built-in runtime to the Apache OpenWhisk project. This Docker image for the new runtime version is already available on Docker Hub. This means it can be used with any Apache OpenWhisk instance straight away!

Playing with Worker Threads on IBM Cloud Functions allowed me to demonstrate how to speed up performance for CPU-intensive workloads by utilising multiple cores concurrently. Using an example of prime number generation, calculating all primes up to ten million took ~35 seconds with a single thread and ~8 seconds with four threads. This represents a reduction in execution time and cost of 75%!

Apache OpenWhisk Web Action HTTP Proxy

What if you could take an existing web application and run it on a serverless platform with no changes? ๐Ÿค”

Lots of existing (simple) stateless web applications are perfect candidates for serverless, but use web frameworks that don’t know how to integrate with those platforms. People have started to develop a number of custom plugins for those frameworks to try and bridge this gap.

These plugins can provide an easier learning curve for developers new to serverless. They can still use familiar web application frameworks whilst learning about the platforms. It also provides a path to “lift and shift” existing (simple) web applications to serverless platforms.

This approach relies on custom framework plugins being available, for every web app framework and serverless platform, which is not currently the case. Is there a better solution?

Recently, I’ve been experimenting with Apache OpenWhisk’s Docker support to prototype a different approach. This solution allows any web application to run on the platform, without needing bespoke framework plugins, with minimal changes. Sounds interesting? Read about how I did this belowโ€ฆ ๐Ÿ‘

Apache OpenWhisk Web Action HTTP Proxy

This project provides a static binary which proxies HTTP traffic from Apache OpenWhisk Web Actions to existing web applications. HTTP events received by the Web Action Proxy are forwarded as HTTP requests to the web application. HTTP responses from the web application are returned as Web Action responses.

Apache OpenWhisk Web Action HTTP Proxy

Both the proxy and web application needed to be started inside the serverless runtime environment. The proxy uses port 8080 and the web application can use any other port. An environment variable or action parameter can be used to configure the local port to proxy.

Running both HTTP processes on the platform is possible due to custom runtime support in Apache OpenWhisk. This allows using custom Docker images as the runtime environment. Custom runtimes images can be built which include the proxy binary and (optionally) the web application source files.

Two different options are available for getting web application source files into the runtime environment.

  • Build source files directly into the container image alongside proxy binary.
  • Dynamically inject source files into container runtime during initialisation.

Building source files into the container is simpler and incurs lower cold-starts delays, but means source code will be publicly available on Docker Hub. Injecting source files through action zips means the public container image can exclude all private source files and secrets. The extra initialisation time for dynamic injection does increase cold-start delays.

Please note: This is an alpha-stage experiment! Don’t expect everything to work. This project is designed to run small simple stateless web applications on Apache OpenWhisk. Please don’t attempt to “lift ‘n’ shift” a huge stateful enterprise app server onto the platform!

Node.js + Express Example

This is an example Node.js web application, built using the Express web application framework:

https://camo.githubusercontent.com/2aa43809d8d8a9f9ccb906c1028d81f1ba1913d9/687474703a2f2f7368617065736865642e636f6d2f696d616765732f61727469636c65732f657870726573735f6578616d706c652e6a7067

The web application renders static HTML content for three routes (/, /about and /contact). CSS files and fonts are also served by the backend.

Use these steps to run this web application on Apache OpenWhisk using the Web Action Proxy…

  • Clone project repo.
1
git clone https://github.com/jthomas/express_example
  • Install project dependencies in the express_example directory.
1
npm install
  • Bundle web application and libraries into zip file.
1
zip -r action.zip *
  • Create the Web Action (using a custom runtime image) with the following command.
1
wsk action create --docker jamesthomas/generic_node_proxy --web true --main "npm start" -p "__ow_proxy_port" 3000 web_app action.zip
  • Retrieve the Web Action URL for the new action.
1
wsk action get web_app --url
  • Open the Web Action URL in a HTTP web browser. (Note: Web Action URLs must end with a forward-slash to work correctly, e.g. https://<OW_HOST>/api/v1/web/<NAMESPACE>/default/web_app/).

Web Action Proxy Express JS

If this works, the web application should load as above. Clicking links in the menu will navigate to different pages in the application.

custom runtime image

This example Web Action uses my own pre-built custom runtime image for Node.js web applications (jamesthomas/generic_node_proxy). This was created from the following Dockerfile to support dynamic runtime injection of web application source files.

1
2
3
4
5
6
7
FROM node:10

ADD proxy /app/
WORKDIR /app
EXPOSE 8080

CMD ./proxy

More Examples

See the examples directory in the project repository for sample applications with build instructions for the following runtimes.

Usage & Configuration

Web application source files can be either be dynamically injected (as in the example above) or built into the custom runtime image.

Dynamic injection uses a custom runtime image with just the proxy binary and runtime dependencies. Web application source files are provided in the action zip file and extracted into the runtime upon initialisation. The proxy will start the app server during cold-starts.

Alternatively, source files for the web application can be included directly in the runtime image. The container start command will start both processes concurrently. No additional files are provided when creating the web action.

Configuration for values such as the proxy port, can be provided using environment variables or default action parameters.

Please see the project documentation for more details on both these approaches, how to use them and configuration parameters.

Challenges

This experiment is still in the alpha-stage and comes with many restrictions at the moment…

  • HTTP request and responses sizes are limited to the maximum sizes allowed by Apache OpenWhisk for input parameters and activation results. This defaults to 1MB in the open-source project and 5MB on IBM Cloud Functions.
  • Page links must use URLs with relative paths to the Web Action URL rather than the host root, e.g. href="home" rather than href="/home". This is due to the Web Actions being served from a sub-path of the platform (/api/v1/web/<NAMESPACE>/default/<ACTION>) rather than the host root.
  • Docker images will be pulled from the public registry on the first invocation. This will lead to long cold-start times for the first request after the action has been created. Large image sizes = longer delays. This only occurs on the first invocation.
  • Web app startup times affect cold start times. The proxy blocks waiting for the web application to start before responding. This delay is included in each cold start. Concurrent HTTP requests from a web browser for static page assets will (initially) result in multiple cold starts.
  • Web Sockets and other complex HTTP features, e.g. server-side events, cannot be supported.
  • Web applications will run in ephemeral container environments that are paused between requests and destroyed without warning. This is not a traditional web application environment, e.g. running background tasks will not work.

Lots of things haven’t been tested. Don’t expect complex stateful web applications to work.

Conclusion

Being able to run existing web applications on serverless platforms opens up a huge opportunity for moving simple (and stateless) web application over to those platforms. These applications can then benefit from the scaling, cost and operational benefits serverless platforms provide.

Previous attempts to support traditional web applications on serverless platforms relied on custom framework plugins. This approach was limited by the availability of custom plugins for each web application framework and serverless platform.

Playing around with Apache OpenWhisk’s custom runtime support, I had an ideaโ€ฆ could a generic HTTP proxy be used to support any framework without needing any plugins? This led to the Apache OpenWhisk Web Action HTTP Proxy project.

By building a custom runtime, the HTTP proxy and web application can both be started within the same serverless environment. HTTP events received by the Web Action Proxy are forwarded as HTTP requests to the web application. HTTP responses from the web application are returned as Web Action responses.

Web application sources files can be injected into the runtime environment during initialisation or built straight into the custom runtime image. No significant changes are required in the web application and it does not need custom framework plugins.

Apache OpenWhisk’s support for custom Docker runtimes opens up a huge range of opportunities for running more varied workloads on serverless platforms - and this is a great example of that!

Serverless CI/CD With Travis CI, Serverless Framework and IBM Cloud Functions

How do you set up a CI/CD pipeline for serverless applications?

This blog post will explain how to use Travis CI, The Serverless Framework and the AVA testing framework to set up a fully-automated build, deploy and test pipeline for a serverless application. It will use a real example of a production serverless application, built using Apache OpenWhisk and running on IBM Cloud Functions. The CI/CD pipeline will execute the following tasks…

  • Run project unit tests.
  • Deploy application to test environment.
  • Run acceptance tests against test environment.
  • Deploy application to production environment.
  • Run smoke tests against production environment.

Before diving into the details of the CI/CD pipeline setup, let’s start by showing the example serverless application being used for this project…

Serverless Project - http://apache.jamesthom.as/

The ”Apache OpenWhisk Release Verification” project is a serverless web application to help committers verify release candidates for the open-source project. It automates running the verification steps from the ASF release checklist using serverless functions. Automating release candidate validation makes it easier for committers to participate in release voting.

Apache OpenWhisk Release Verification Tool

The project consists of a static web assets (HTML, JS, CSS files) and HTTP APIs. Static web assets are hosted by Github Pages from the project repository. HTTP APIs are implemented as Apache OpenWhisk actions and exposed using the API Gateway service. IBM Cloud Functions is used to host the Apache OpenWhisk application.

No other cloud services, like databases, are needed by the backend. Release candidate information is retrieved in real-time by parsing the HTML page from the ASF website.

Serverless Architecture

Configuration

The Serverless Framework (with the Apache OpenWhisk provider plugin) is used to define the serverless functions used in the application. HTTP endpoints are also defined in the YAML configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
service: release-verfication

provider:
  name: openwhisk
  runtime: nodejs:10

functions:
  versions:
    handler: index.versions
    events:
      - http: GET /api/versions
  version_files:
    handler: index.version_files
    events:
      - http:
          method: GET
          path: /api/versions/{version}
          resp: http
...

plugins:
  - serverless-openwhisk

The framework handles all deployment and configuration tasks for the application. Setting up the application in a new environment is as simple as running the serverless deploy command.

Environments

Apache OpenWhisk uses namespaces to group individual packages, actions, triggers and rules. Different namespaces can be used to provide isolated environments for applications.

IBM Cloud Functions automatically creates user-based namespaces in platform instances. These auto-generated namespaces mirror the IBM Cloud organisation and space used to access the instance. Creating new spaces within an organisation will provision extra namespaces.

I’m using a custom organisation for the application with three different spaces: dev, test and prod.

dev is used as a test environment to deploy functions during development. test is used by the CI/CD pipeline to deploy a temporary instance of the application during acceptance tests. prod is the production environment hosting the external application actions.

Credentials

The IBM Cloud CLI is used to handle IBM Cloud Functions credentials. Platform API keys will be used to log in the CLI from the CI/CD system.

When Cloud Functions CLI commands are issued (after targeting a new region, organisation or space), API keys for that Cloud Functions instance are automatically retrieved and stored locally. The Serverless Framework knows how to use these local credentials when interacting with the platform.

High Availability?

The Apache OpenWhisk Release Verifier is not a critical cloud application which needs ”five nines” of availability. The application is idle most of the time. It does not need a highly available serverless architecture. This means the build pipeline does not have to…

New deployments will simply overwrite resources in the production namespace in a single region. If the production site is broken after a deployment, the smoke tests should catch this and email me to fix it!

Testing

Given this tool will be used to check release candidates for the open-source project, I wanted to ensure it worked properly! Incorrect validation results could lead to invalid source archives being published.

I’ve chosen to rely heavily on unit tests to check the core business logic. These tests ensure all validation tasks work correctly, including PGP signature verification, cryptographic hash matching, LICENSE file contents and other ASF requirements for project releases.

Additionally, I’ve used end-to-end acceptance tests to validate the HTTP APIs work as expected. HTTP requests are sent to the API GW endpoints, with responses compared against expected values. All available release candidates are run through the validation process to check no errors are returned.

Unit Tests

Unit tests are implemented with the AVA testing framework. Unit tests live in the unit/test/ folder.

The npm test command alias runs the ava test/unit/ command to execute all unit tests. This command can be executed locally, during development, or from the CI/CD pipeline.

1
2
3
4
5
6
$ npm test

> release-verification@1.0.0 test ~/code/release-verification
> ava test/unit/

 27 tests passed

Acceptance Tests

Acceptance tests check API endpoints return the expected responses for valid (and invalid) requests. Acceptance tests are executed against the API Gateway endpoints for an application instance.

The hostname used for HTTP requests is controlled using an environment variable (HOST). Since the same test suite test is used for acceptance and smoke tests, setting this environment variable is the only configuration needed to run tests against different environments.

API endpoints in the test and production environments are exposed using different custom sub-domains (apache-api.jamesthom.as and apache-api-test.jamesthom.as). NPM scripts are used to provide commands (acceptance-test & acceptance-prod) which set the environment hostname before running the test suite.

1
2
3
4
"scripts": {
    "acceptance-test": "HOST=apache-api-test.jamesthom.as ava -v --fail-fast test/acceptance/",
    "acceptance-prod": "HOST=apache-api.jamesthom.as ava -v --fail-fast test/acceptance/"
  },
1
2
3
4
5
6
7
8
9
10
11
12
$ npm run acceptance-prod

> release-verification@1.0.0 acceptance-prod ~/code/release-verification
> HOST=apache-api.jamesthom.as ava -v --fail-fast  test/acceptance/

  โœ” should return list of release candidates (3.7s)
    โ„น running api testing against https://apache-api.jamesthom.as/api/versions
  โœ” should return 404 for file list when release candidate is invalid (2.1s)
    โ„น running api testing against https://apache-api.jamesthom.as/api/versions/unknown
  ...

  6 tests passed

Acceptance tests are also implemented with the AVA testing framework. All acceptance tests live in a single test file (unit/acceptance/api.js).

CI/CD Pipeline

When new commits are pushed to the master branch on the project repository, the following steps needed to be kicked off by the build pipelineโ€ฆ

  • Run project unit tests.
  • Deploy application to test environment.
  • Run acceptance tests against test environment.
  • Deploy application to production environment.
  • Run smoke tests against production environment.

If any of the steps fail, the build pipeline should stop and send me a notification email.

Travis

Travis CI is used to implement the CI/CD build pipeline. Travis CI uses a custom file (.travis.yml) in the project repository to configure the build pipeline. This YAML file defines commands to execute during each phase of build pipeline. If any of the commands fail, the build will stop at that phase without proceeding.

Here is the completed .travis.yml file for this project: https://github.com/jthomas/openwhisk-release-verification/blob/master/.travis.yml

I’m using the following Travis CI build phases to implement the pipeline: install, before_script, script, before_deploy and deploy. Commands will run in the Node.js 10 build environment, which pre-installs the language runtime and package manager.

1
2
3
language: node_js
node_js:
  - "10"

install

In the install phase, I need to set up the build environment to deploy the application and run tests.

This means installing the IBM Cloud CLI, Cloud Functions CLI plugin, The Serverless Framework (with Apache OpenWhisk plugin), application test framework (AvaJS) and other project dependencies.

The IBM Cloud CLI is installed using a shell script. Running a CLI sub-command installs the Cloud Functions plugin.

The Serverless Framework is installed as global NPM package (using npm -g install). The Apache OpenWhisk provider plugin is handled as normal project dependency, along with the test framework. Both those dependencies are installed using NPM.

1
2
3
4
5
install:
  - curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
  - ibmcloud plugin install cloud-functions
  - npm install serverless -g
  - npm install

before_script

This phase is used to run unit tests, catching errors in core business logic, before setting up credentials (used in the script phase) for the acceptance test environment. Unit test failures will halt the build immediately, skipping test and production deployments.

Custom variables provide the API key, platform endpoint, organisation and space identifiers which are used for the test environment. The CLI is authenticated using these values, before running the ibmcloud fn api list command. This ensures Cloud Functions credentials are available locally, as used by The Serverless Framework.

1
2
3
4
5
6
before_script:
  - npm test
  - ibmcloud login --apikey $IBMCLOUD_API_KEY -a $IBMCLOUD_API_ENDPOINT
  - ibmcloud target -o $IBMCLOUD_ORG -s $IBMCLOUD_TEST_SPACE
  - ibmcloud fn api list > /dev/null
  - ibmcloud target

script

With the build system configured, the application can be deployed to test environment, followed by running acceptance tests. If either deployment or acceptance tests fail, the build will stop, skipping the production deployment.

Acceptance tests use an environment variable to configure the hostname test cases are executed against. The npm run acceptance-test alias command sets this value to the test environment hostname (apache-api-test.jamesthom.as) before running the test suite.

1
2
3
script:
  - sls deploy
  - npm run acceptance-test

before_deploy

Before deploying to production, Cloud Functions credentials need to be updated. The IBM Cloud CLI is used to target the production environment, before running a Cloud Functions CLI command. This updates local credentials with the production environment credentials.

1
2
3
4
before_deploy:
  - ibmcloud target -s $IBMCLOUD_PROD_SPACE
  - ibmcloud fn api list > /dev/null
  - ibmcloud target

deploy

If all the proceeding stages have successfully finished, the application can be deployed to the production. Following this final deployment, smoke tests are used to check production APIs still work as expected.

Smoke tests are just the same acceptance tests executed against the production environment. The npm run acceptance-prod alias command sets the hostname configuration value to the production environment (apache-api.jamesthom.as) before running the test suite.

1
2
3
4
deploy:
  provider: script
  script: sls deploy && npm run acceptance-prod
  skip_cleanup: true

Using the skip_cleanup parameter leaves installed artifacts from previous phases in the build environment. This means we don’t have to re-install the IBM Cloud CLI, The Serverless Framework or NPM dependencies needed to run the production deployment and smoke tests.

success?

If all of the build phases are successful, the latest project code should have been deployed to the production environment. ๐Ÿ’ฏ๐Ÿ’ฏ๐Ÿ’ฏ

Build Screenshoot

If the build failed due to unit test failures, the test suite can be ran locally to fix any errors. Deployment failures can be investigated using the console output logs from Travis CI. Acceptance test issues, against test or production environments, can be debugged by logging into those environments locally and running the test suite from my development machine.

Conclusion

Using Travis CI with The Serverless Framework and a JavaScript testing framework, I was able to set up a fully-automated CI/CD deployment pipeline for the Apache OpenWhisk release candidate verification tool.

Using a CI/CD pipeline, rather than a manual approach, for deployments has the following advantages…

  • No more manual and error-prone deploys relying on a human ๐Ÿ‘จโ€๐Ÿ’ป :)
  • Automatic unit & acceptance test execution catch errors before deployments.
  • Production environment only accessed by CI/CD system, reducing accidental breakages.
  • All cloud resources must be configured in code. No ”snowflake” environments allowed.

Having finished code for new project features or bug fixes, all I have to do is push changes to the GitHub repository. This fires the Travis CI build pipeline which will automatically deploy the updated application to the production environment. If there are any issues, due to failed tests or deployments, I’ll be notified by email.

This allows me to get back to adding new features to the tool (and fixing bugs) rather than wrestling with deployments, managing credentials for multiple environments and then trying to remember to run tests against the correct instances!

Automating Apache OpenWhisk Releases With Serverless

This blog post explains how I used serverless functions to automate release candidate verification for the Apache OpenWhisk project.

Apache OpenWhisk Release Verification Tool

Automating this process has the following benefits…

  • Removes the chance of human errors compared to the previously manual validation process.
  • Allows me to validate new releases without access to my dev machine.
  • Usable by all committers by hosting as an external serverless web app.

Automating release candidate validation makes it easier for project committers to participate in release voting. This should make it faster to get necessary release votes, allowing us to ship new versions sooner!

background

apache software foundation

The Apache Software Foundation has a well-established release process for delivering new product releases from projects belonging to the foundation. According to their documentation

An Apache release is a set of valid & signed artifacts, voted on by the appropriate PMC and distributed on the ASF’s official release infrastructure.

https://www.apache.org/dev/release-publishing.html

Releasing a new software version requires the release manager to create a release candidate from the project source files. Source archives must be cryptographically signed by the release manager. All source archives for the release must be comply with strict criteria to be considered valid release candidates. This includes (but is not limited to) the following requirements:

  • Checksums and PGP signatures for source archives are valid.
  • LICENSE, NOTICE and DISCLAIMER files included and correct.
  • All source files have license headers.
  • No compiled archives bundled in source archives.

Release candidates can then be proposed on the project mailing list for review by members of the Project Management Committee (PMC). PMC members are eligible to vote on all release candidates. Before casting their votes, PMC members are required to check release candidate meets the requirements above.

If a minimum of three positive votes is cast (with more positive than negative votes), the release passes! The release manager can then move the release candidate archives to the release directory.

apache openwhisk releases

As a committer and PMC member on the Apache OpenWhisk project, I’m eligible to vote on new releases.

Apache OpenWhisk (currently) has 52 separate source repositories under the project on GitHub. With a fast-moving open-source project, new releases candidate are constantly being proposed, which all require the necessary number of binding PMC votes to pass.

Manually validating release candidates can be a time-consuming process. This can make it challenging to get a quorum of binding votes from PMC members for the release to pass. I started thinking how I could improve my productivity around the validation process, enabling me to participate in more votes.

Would it be possible to automate some (or all) of the steps in release candidate verification? Could we even use a serverless application to do this?

apache openwhisk release verifier

Spoiler Alert: YES! I ended up building a serverless application to do this for me.

It is available at https://apache.jamesthom.as/

Apache OpenWhisk Release Verifier

Source code for this project is available here.

IBM Cloud Functions is used to run the serverless backend for the web application. This means Apache OpenWhisk is being used to validate future releases of itselfโ€ฆ which is awesome.

architecture

Project Architecture

HTML, JS and CSS files are served by Github Pages from the project repository.

Backend APIs are Apache OpenWhisk actions running on IBM Cloud Functions.

Both the front-page and API are served from a custom sub-domains of my personal domain.

available release candidates

When the user loads the page, the drop-down list needs to contain the current list of release candidates from the ASF development distribution site.

This information is available to the web page via the https://apache-api.jamesthom.as/api/versions endpoint. The serverless function powering this API parses that live HTML page (extracting the current list of release candidates) each time it is invoked.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ http get https://apache-api.jamesthom.as/api/versions
HTTP/1.1 200 OK
...
{
    "versions": [
        "apache-openwhisk-0.11.0-incubating-rc1",
        "apache-openwhisk-0.11.0-incubating-rc2",
        "apache-openwhisk-1.13.0-incubating-rc1",
        "apache-openwhisk-1.13.0-incubating-rc2",
        "apache-openwhisk-2.0.0-incubating-rc2",
        "apache-openwhisk-3.19.0-incubating-rc1"
    ]
}

release candidate version info

Release candidates may have multiple source archives being distributed in that release. Validation steps need to be executed for each of those archives within the release candidate.

Once a user has selected a release candidate version, source archives to validate are shown in the table. This data is available from the https://apache-api.jamesthom.as/api/versions/VERSION endpoint. This information is parsed from the HTML page on the ASF site.

1
2
3
4
5
6
7
8
9
10
11
$ http get https://apache-api.jamesthom.as/api/versions/apache-openwhisk-2.0.0-incubating-rc2
HTTP/1.1 200 OK
...

{
    "files": [
        "openwhisk-package-alarms-2.0.0-incubating-sources.tar.gz",
        "openwhisk-package-cloudant-2.0.0-incubating-sources.tar.gz",
        "openwhisk-package-kafka-2.0.0-incubating-sources.tar.gz"
    ]
}

release verification

Having selected a release candidate version, clicking the ”Validate” button will start validation process. Triggering the https://apache-api.jamesthom.as/api/versions/VERSION/validate endpoint will run the serverless function used to execute the validation steps.

This serverless function will carry out the following verification steps…

checking download links

All the source archives for a release candidate are downloaded to temporary storage in the runtime environment. The function also downloads the associated SHA512 and PGP signature files for comparison. Multiple readable streams can be created from the same file path to allow the verification steps to happen in parallel, rather than having to re-download the archive for each task.

checking SHA512 hash values

SHA512 sums are distributed in a text file containing hex strings with the hash value.

1
2
3
openwhisk-package-alarms-2.0.0-incubating-sources.tar.gz:
3BF87306 D424955B B1B2813C 204CC086 6D27FA11 075F0B30 75F67782 5A0198F8 091E7D07
 B7357A54 A72B2552 E9F8D097 50090E9F A0C7DBD1 D4424B05 B59EE44E

The serverless function needs to dynamically compute the hash for the source archive and compare the hex bytes against the text file contents. Node.js comes with a built-in crypto library making it easy to create hash values from input streams.

This is the function used to compute and compare the hash values.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
const hash = async (file_stream, hash_file, name) => {
  return new Promise((resolve, reject) => {
    const sha512 = parse_hash_from_file(hash_file)

    const hmac = crypto.createHash('sha512')
    file_stream.pipe(hmac)

    hmac.on('readable', () => {
      const stream_hash = hmac.read().toString('hex')
      const valid = stream_hash === sha512.signature
      logger.log(`file (${name}) calculated hash: ${stream_hash}`)
      logger.log(`file (${name}) hash from file:  ${sha512.signature}`)
      resolve({valid})
    })

    hmac.on('error', err => reject(err))
  })
}

validating PGP signatures

Node.js’ crypto library does not support validating PGP signatures.

I’ve used the OpenPGP.js library to handle this task. This is a Javascript implementation of the OpenPGP protocol (and the most popular PGP library for Node.js). Three input values are needed to validate PGP messages.

  • Message contents to check.
  • PGP signature for the message.
  • Public key for the private key used to sign the release.

The “message” to check is the source archive. PGP signatures come from the .asc files located in the release candidate directory.

1
2
3
4
5
6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAABAgAGBQJcpO0FAAoJEHKvDMIsTPMgf0kP+wbtJ1ONZJQKjyDVx8uASMDQ
...
-----END PGP SIGNATURE-----

Public keys used to sign releases are stored in the root folder of the release directory for that project.

This function is used to implement the signature checking process.

1
2
3
4
5
6
7
8
9
10
11
12
13
const signature = async (file_stream, signature, public_keys, name) => {
  const options = {
    message: openpgp.message.fromBinary(file_stream),
    signature: await openpgp.signature.readArmored(signature),
    publicKeys: (await openpgp.key.readArmored(public_keys)).keys
  }

  const verified = await openpgp.verify(options)
  await openpgp.stream.readToEnd(verified.data)
  const valid = await verified.signatures[0].verified

  return { valid }
}

scanning archive files

Using the node-tar library, downloaded source archives are extracted into the local runtime to allow scanning of individual files.

LICENSE.txt, DISCLAIMER.txt and NOTICE.txt files are checked to ensure correctness. An external NPM library is used to check all files in the archive for binary contents. The code also scans for directory names that might contain third party libraries (node_modules or .gradle).

capturing validation logs

It is important to provide PMC members with verifiable logs on the validation steps performed. This allows them to sanity check the steps performed (including manual validation). This verification text can also be provided in the voting emails as evidence of release candidate validity.

Using a custom logging library, all debug logs sent to the console are recorded in the action result (and therefore returned in the API response).

showing results

Once all the validation tasks have been executed - the results are returned to the front-end as a JSON response. The client-side JS parses these results and updates the validation table. Validation logs are shown in a collapsible window.

Verification Results

Using visual emojis for pass and failure indicators for each step - the user can easily verify whether a release passes the validation checks. If any of the steps have failed, the validation logs provide an opportunity to understand why.

Verification Logs

other tools

This is not the only tool that can automate checks needed to validate Apache Software Foundation releases.

Another community member has also built a bash script (rcverify.sh) that can verify releases on your local machine. This script will automatically download the release candidate files and run many of the same validation tasks as the remote tool locally.

There is also an existing tool (Apache Rat) from another project that provides a Java-based application for auditing license headers in source files.

conclusion

Getting new product releases published for an open-source project under the ASF is not a simple task for developers used to pushing a button on Github! The ASF has a series of strict guidelines on what constitutes a release and the ratification process from PMC members. PMC members need to run a series of manual verification tasks before casting binding votes on proposed release candidates.

This can be a time-consuming task for PMC members on a project like Apache OpenWhisk, with 52 different project repositories all being released at different intervals. In an effort to improve my own productivity around this process, I started looking for ways to automate the verification tasks. This would enable me to participate in more votes and be a “better” PMC member.

This led to building a serverless web application to run all the verification tasks remotely, which is now hosted at https://apache.jamesthom.as. This tool uses Apache OpenWhisk (provided by IBM Cloud Functions), which means the project is being used to verify future releases of itself! I’ve also open-sourced the code to provide an example of how to use the platform for automating tasks like this.

With this tool and others listed above, verifying new Apache OpenWhisk releases has never been easier!