James Thomas

Notes on software.

Monitoring Dashboards With Kibana for IBM Cloud Functions

Following all the events from the World Cup can be hard. So many matches, so many goals. Rather than manually refreshing BBC Football to check the scores, I decided to created a Twitter bot that would automatically tweet out each goal.

The Twitter bot runs on IBM Cloud Functions. It is called once a minute to check for new goals, using the alarm trigger feed. If new goals have been scored, it calls another action to send the tweet messages.

Once it was running, I need to ensure it was working correctly for the duration of the tournament. Using the IBM Cloud Logging service, I built a custom monitoring dashboard to help to me recognise and diagnose issues.

The dashboard showed counts for successful and failed activations, when they occurred and a list of failed activations. If issues have occurred, I can retrieve the failed activation identifiers and investigate further.

Let’s walk through the steps used to create this dashboard to help you create custom visualisations for serverless applications running on IBM Cloud Functions…

IBM Cloud Logging

IBM Cloud Logging can be accessed using the link on the IBM Cloud Functions dashboard. This will open the logging service for the current organisation and space.

All activation records and application logs are automatically forwarded to the logging service by IBM Cloud Functions.

Log Message Fields

Activation records and application log messages have a number of common record fields.

  • activationId_str - activation identifier for log message.
  • timestamp - log draining time.
  • @timestamp - message ingestion time.
  • action_str - fully qualified action name

Log records for different message types are identified using the type field. This is either activation_record or user_logs for IBM Cloud Functions records.

Activation records have the following custom fields.

  • duration_int - activation duration in milliseconds
  • status_str - activation status response (non-zero for errors)
  • message - activation response returned from action
  • time_date - activation record start time
  • end_date - activation record end time

Applications log lines, written to stdout or stderr, are forwarded as individual records. One application log line per record. Log message records have the following custom fields.

  • message - single application log line output
  • stream_str - log message source, either stdout or stderr
  • time_date - timestamp parsed from application log line

Finding Log Messages For One Activation

Use this query string in the ”Discover tab to retrieve all logs messages from a particular activation.

1
activationId_str: <ACTIVATION_ID>

Search queries are executed against log records within a configurable time window.

Monitoring Dashboard

This is the monitoring dashboard I created. It contains visualisations showing counts for successful and failed activations, histograms of when they occurred and a list of the recent failed activation identifiers.

It allows me to quickly review the previous 24 hours activations for issues. If there are notable issues, I can retrieve the failed activation identifiers and investigate further.

Before being able to create the dashboard, I needed to define two resources: saved searches and visualisations.

Saved Searches

Kibana supports saving and referring to search queries from visualisations using explicit names.

Using saved searches with visualisations, rather than explicit queries, removes the need to manually update visualisations’ configuration when queries change.

This dashboard uses two custom queries in visualisations. Queries are needed to find activation records from both successful and failed invocations.

  • Create a new “Saved Search” named “activation records (success)” using the following search query.
1
type: activation_record AND status_str: 0
  • Create a new “Saved Search” named “activation records (failed)” using the following search query.
1
type: activation_record AND NOT status_str: 0

The status_str field is set to a non-zero value for failures. Using the type field ensures log messages from other sources are excluded from the results.

Indexed Fields

Before referencing log record fields in visualisations, those fields need to be indexed correctly. Use these instructions to verify activation records fields are available.

  • Check IBM Cloud Functions logs are available in IBM Cloud Logging using the ”Discover” tab.
  • Click the “⚙️ (Management)” menu item on the left-hand drop-down menu in IBM Cloud Logging.
  • Click the ”Index Patterns” link.
  • Click the 🔄 button to refresh the field list.

Visualisations

Three types of visualisation are used on the monitoring dashboard. Metric displays are used for the activation counts, vertical bar charts for the activation times and a data table to list failed activations.

Visualisations can be created by opening the “Visualize” menu item and select a new visualisation type under the “Create New Visualization” menu.

Create five different visualisations, using the instructions below, before moving on to create the dashboard.

Activation Counts

Counts for successful and failed activations are displayed as singular metric values.

  • Select the “Metric” visualisation from the visualisation type list.
  • Use the “activation records (success)” saved search as the data source.
  • Ensure the Metric Aggregation is set to “Count”
  • Set the “Font Size” under the Options menu to 120pt.
  • Save the visualisation as “Activation Counts (Success)”

  • Repeat this process to create the failed activation count visualisation.
  • Use the “activation records (failed)” saved search as the data source.
  • Save the visualisation as “Activation Counts (Failed)”.

Activation Times

Activation counts over time, for successful and failed invocations, are displayed in vertical bar charts.

  • Select the “Vertical bar chart” visualisation from the visualisation type list.
  • Use the “activation records (success)” saved search as the data source.
  • Set the “Custom Label” to Invocations
  • Add an “X-Axis” bucket type under the Buckets section.
  • Choose “Date Histogram” for the aggregation, “@timestamp” for the field and “Minute” for the interval.
  • Save the visualisation as “Activation Times (Success)”

  • Repeat this process to create the failed activation times visualisation.
  • Use the “activation records (failed)” saved search as the data source.
  • Save the visualisation as “Activation Times (Failed)”

Failed Activations List

Activation identifiers for failed invocations are shown using a data table.

  • Select the “Data table” visualisation from the visualisation type list.
  • Use the “activation records (failed)” saved search as the data source.
  • Add a “Split Rows” bucket type under the Buckets section.
  • Choose “Date Histogram” for the aggregation, “@timestamp” for the field and “Second” for the interval.
  • Add a “sub-bucket” with the “Split Rows” type.
  • Set sub aggregation to “Terms”, field to “activationId_str” and order by “Term”.
  • Save the visualisation as “Errors Table”

Creating the dashboard

Having created the individual visualisations components, the monitoring dashboard can be constructed.

  • Click the “Dashboard” menu item from the left-and menu panel.
  • Click the “Add” button to import visualisations into the current dashboard.
  • Add each of the five visualisations created above.

Hovering the mouse cursor over visualisations will reveal icons for moving and re-sizing.

  • Re-order the visualisations into the following rows:
    • Activations Metrics
    • Activation Times
    • Failed Activations List
  • Select the “Last 24 hours” time window, available from the relative time ranges menu.
  • Save the dashboard as ”Cloud Functions Monitoring”. Tick the ”store time with dashboard” option.

Having saved the dashboard with time window, re-opening the dashboard will show our visualisations with data for the previous 24 hours. This dashboard can be used to quickly review recent application issues.

Conclusion

Monitoring serverless applications is crucial to diagnosing issues on serverless platforms.

IBM Cloud Functions provides automatic integration with the IBM Cloud Logging service. All activation records and application logs from serverless applications are automatically forwarded as log records. This makes it simple to build custom monitoring dashboards using these records for serverless applications running on IBM Cloud Functions.

Using this service with World Cup Twitter bot allowed me to easily monitor the application for issues. This was much easier than manually retrieving and reviewing activation records using the CLI!

Debugging Node.js OpenWhisk Actions

Debugging serverless applications is one of the most challenging issues developers face when using serverless platforms. How can you use debugging tools without any access to the runtime environment?

Last week, I worked out how to expose the Node.js debugger in the Docker environment used for the application runtime in Apache OpenWhisk.

Using the remote debugging service, we can set breakpoints and step through action handlers live, rather than just being reliant on logs and metrics to diagnose bugs.

So, how does this work?

Let’s find out more about how Apache OpenWhisk executes serverless functions…

Background

Apache OpenWhisk is the open-source serverless platform which powers IBM Cloud Functions. OpenWhisk uses Docker containers to create isolated runtime environments for executing serverless functions.

Containers are started on-demand as invocation requests arrive. Serverless function source files are dynamically injected into the runtime and executed for each invocation. Between invocations, containers are paused and kept in a cache for re-use with further invocations.

The benefit of using an open-source serverless platform is that the build files used to create runtime images are also open-source. OpenWhisk also automatically builds and publishes all runtime images externally on Docker Hub. Running containers using these images allows us to simulate the remote serverless runtime environment.

Runtime Images

All OpenWhisk runtime images are published externally on Docker Hub.

Runtime images start a HTTP server which listens on port 8080. This HTTP server must implement two API endpoints (/init & /run) accepting HTTP POST requests. The platform uses these endpoints to initialise the runtime with action code and then invoke the action with event parameters.

More details on the API endpoints can be found in this blog post on creating Docker-based actions.

Node.js Runtime Image

This repository contains the source code used to create Node.js runtime environment image.

https://github.com/apache/incubator-openwhisk-runtime-nodejs

Both Node.js 8 and 6 runtimes are built from a common base image. This base image contains an Express.js server which handles the platform API requests. The app.js file containing the server is executed when the containers starts.

JavaScript code is injected into the runtime using the /init API. Actions created from source code are dynamically evaluated to instantiate the code in the runtime. Actions created from zip files are extracted into a temporary directory and imported as a Node.js module.

Once instantiated, actions are executed using the /run API. Event parameters are come from the request body. Each time a new request is received, the server calls the action handler with event parameters. Returned values are serialised as the JSON body in the API response.

Starting Node.js Runtime Containers

Use this command to start the Node.js runtime container locally.

1
$ docker run -it -p 8080:8080 openwhisk/action-nodejs-v8

Once the container has started, port 8080 on localhost will be mapped to the HTTP service exposed by the runtime environment. This can be used to inject serverless applications into the runtime environment and invoke the serverless function handler with event parameters.

Node.js Remote Debugging

Modern versions of the Node.js runtime have a command-line flag (--inspect) to expose a remote debugging service. This service runs a WebSocket server on localhost which implements the Chrome DevTools Protocol.

1
2
$ node --inspect index.js
Debugger listening on 127.0.0.1:9229.

External tools can connect to this port to provide debugging capabilities for Node.js code.

Docker images for the OpenWhisk Node.js runtimes use the following command to start the internal Node.js process. Remote debugging is not enabled by default.

1
node --expose-gc app.js

Docker allows containers to override the default image start command using a command line argument.

This command will start the OpenWhisk Node.js runtime container with the remote debugging service enabled. Binding the HTTP API and WebSocket ports to the host machine allows us to access those services remotely.

1
docker run -p 8080:8080 -p 9229:9229 -it openwhisk/action-nodejs-v8 node --inspect=0.0.0.0:9229 app.js

Once a container from the runtime image has started, we can connect our favourite debugging tools…

Chrome Dev Tools

To connect Chrome Dev Tools to the remote Node.js debugging service, follow these steps.

Chrome Dev Tools is configured to open a connection on port 9229 on localhost. If the web socket connection succeeds, the debugging target should be listed in the “Remote Target” section.

  • Click the ”Open dedicated DevTools for Node” link.

In the “Sources” panel the JavaScript files loaded by the Node.js process are available.

Setting breakpoints in the runner.js file will allow you to halt execution for debugging upon invocations.

VSCode

Visual Studio Code supports remote debugging of Node.js code using the Chrome Dev Tools protocol. Follow these steps to connect the editor to the remote debugging service.

  • Click the menu item ”Debug -> Add Configuration
  • Select the ”Node.js: Attach to Remote Program” from the Intellisense menu.
  • Edit the default configuration to have the following values.
1
2
3
4
5
6
7
8
{
  "type": "node",
  "request": "attach",
  "name": "Attach to Remote",
  "address": "127.0.0.1",
  "port": 9229,
  "localRoot": "${workspaceFolder}"
}

  • Choose the new ”attach to remote” debugging profile and click the Run button.

The ”Loaded Scripts” window will show all the JavaScript files loaded by the Node.js process.

Setting breakpoints in the runner.js file will allow you to halt execution for debugging upon invocations.

Breakpoint Locations

Here are some useful locations to set breakpoints to catch errors in your serverless functions for the OpenWhisk Node.js runtime environments.

Initialisation Errors - Source Actions

If you are creating OpenWhisk actions from JavaScript source files, the code is dynamically evaluated during the /init request at this location. Putting a breakpoint here will allow you to catch errors thrown during that eval() call.

Initialisation Errors - Binary Actions

If you are creating OpenWhisk actions from a zip file containing JavaScript modules, this location is where the archive is extracted in the runtime filesystem. Putting a breakpoint here will catch errors from the extraction call and runtime checks for a valid JavaScript module.

This code is where the JavaScript module is imported once it has been extracted. Putting a breakpoint here will catch errors thrown importing the module into the Node.js environment.

Action Handler Errors

For both source file and zipped module actions, this location is where the action handler is invoked on each /run request. Putting a breakpoint here will catch errors thrown from within action handlers.

Invoking OpenWhisk Actions

Once you have attached the debugger to the remote Node.js process, you need to send the API requests to simulate the platform invocations. Runtime containers use separate HTTP endpoints to import the action source code into the runtime environment (/init) and then fire the invocation requests (/run).

Generating Init Request Body - Source Files

If you are creating OpenWhisk actions from JavaScript source files, send the following JSON body in the HTTP POST to the /init endpoint.

1
2
3
4
5
6
{
  "value": {
    "main": "<FUNCTION NAME IN SOURCE FILE>",
    "code": "<INSERT SOURCE HERE>"
  }
}

code is the JavaScript source to be evaluated which contains the action handler. main is the function name in the source file used for the action handler.

Using the jq command-line tool, we can create the JSON body for the source code in file.js.

1
$ cat file.js | jq -sR  '{value: {main: "main", code: .}}'

Generating Init Request Body - Zipped Modules

If you are creating OpenWhisk actions from a zip file containing JavaScript modules, send the following JSON body in the HTTP POST to the /init endpoint.

1
2
3
4
5
6
7
{
  "value": {
    "main": "<FUNCTION NAME ON JS MODULE>",
    "code": "<INSERT BASE64 ENCODED STRING FROM ZIP FILE HERE>",
    "binary": true
  }
}

code must be a Base64 encoded string for the zip file. main is the function name returned in the imported JavaScript module to call as the action handler.

Using the jq command-line tool, we can create the JSON body for the zip file in action.zip.

1
$ base64 action.zip | tr -d '\n' | jq -sR '{value: {main: "main", binary: true, code: .}}'

Sending Init Request

The HTTPie tool makes it simple to send HTTP requests from the command-line.

Using this tool, the following command will initialise the runtime container with an OpenWhisk action.

1
2
3
4
5
6
$ http post localhost:8080/init < init.json
HTTP/1.1 200 OK
...
{
    "OK": true
}

If this HTTP request returns without an error, the action is ready to be invoked.

No further initialisation requests are needed unless you want to modify the action deployed.

Generating Run Request Body

Invocations of the action handler functions are triggered from a HTTP POST to the /run API endpoint.

Invocations parameters are sent in the JSON request body, using a JSON object with a value field.

1
2
3
4
5
6
{
  "value": {
    "some-param-name": "some-param-value",
    "another-param-name": "another-param-value",
  }
}

Sending Run Request

Using the HTTPie tool, the following command will invoke the OpenWhisk action.

1
2
3
4
5
6
$ http post localhost:8080/run < run.json
HTTP/1.1 200 OK
...
{
    "msg": "Hello world"
}

Returned values from the action handler are serialised as the JSON body in the HTTP response. Issuing further HTTP POST requests to the /run endpoint allows us to re-invoke the action.

Conclusion

Lack of debugging tools is one of the biggest complaints from developers migrating to serverless platforms.

Using an open-source serverless platform helps with this problem, by making it simple to run the same containers locally that are used for the platform’s runtime environments. Debugging tools can then be started from inside these local environments to simulate remote access.

In this example, this approach was used to enable the remote debugging service from the OpenWhisk Node.js runtime environment. The same approach could be used for any language and debugging tool needing local access to the runtime environment.

Having access to the Node.js debugger is huge improvement when debugging challenging issues, rather than just being reliant on logs and metrics collected by the platform.

Binding IAM Services to IBM Cloud Functions

Binding service credentials to actions and packages is a much better approach to handling authentication credentials in IBM Cloud Functions, than manually updating (and maintaining) default parameters 🔐.

IBM Cloud Functions supports binding credentials from IAM-based and Cloud Foundry provisioned services.

Documentation and blog posts demonstrating service binding focuses on traditional platform services, created using the Cloud Foundry service broker. As IBM Cloud integrates IAM across the platform, more platform services will migrate to use the IAM service for managing authentication credentials.

How do we bind credentials for IAM-based services to IBM Cloud Functions? 🤔

Binding IAM-based services to IBM Cloud Functions works the same as traditional platform services, but has some differences in how to retrieve details needed for the service bind command.

Let’s look at how this works…

Binding IAM Credentials

Requirements

Before binding an IAM-based service to IBM Cloud Functions, the following conditions must be met.

You will need the following information to bind a service credentials.

  • Service name.
  • (Optional) Instance name.
  • (Optional) Credentials identifier.

Using the CLI

Use the ibmcloud wsk service bind command to bind service credentials to actions or packages.

1
bx wsk service bind <SERVICE_NAME> <ACTION|PACKAGE> --instance <INSTANCE> --keyname <KEY>

This command supports the following (optional) flags: --instance and --keyname.

If the instance and/or key names are not specified, the CLI uses the first instance and credentials returned from the system for the service identifier.

Accessing from actions

Credentials are stored as default parameters on the action or package.

The command uses a special parameter name (__bx_creds) to store all credentials. Individual service credentials are indexed using the service name.

1
2
3
4
5
6
7
8
{
   "__bx_creds":{
      "service-name":{
         "apikey":"<API_KEY>",
         ...
      }
   }
}

Default parameters are automatically merged into the request parameters during invocations.

Common Questions

How can I tell whether a service instance uses IAM-based authentication?

Running the ibmcloud resource service-instances command will return the IAM-based service instances provisioned.

Cloud Foundry provisioned services are available using a different command: ibmcloud service list.

Both service types can be bound using the CLI but the commands to retrieve the necessary details are different.

How can I find the service name for an IAM-based service instance?

Run the ibmcloud resource service-instance <INSTANCE_NAME> command.

Service names are shown as the Service Name: field value.

How can I list available service credentials for an IAM-based service instance?

Use the ibmcloud resource service-keys --instance-name <NAME> command.

Replace the <NAME> value with the service instance returned from the ibmcloud service list command.

How can I manually retrieve IAM-based credentials for an instance?

Use the ibmcloud resource service-key <CREDENTIALS_NAME> command.

Replace the <CREDENTIALS_NAME> value with credential names returned from the ibmcloud service service-keys command.

How can I create new service credentials?

Credentials can be created through the service management page on IBM Cloud.

You can also use the CLI to create credentials using the ibmcloud resource service-key-create command. This command needs a name for the credentials, IAM role and service instance identifier.

Example - Cloud Object Storage

Having explained how to bind IAM-based services to IBM Cloud Functions, let’s look at an example….

Cloud Object Storage is the service used to manage files for serverless applications on IBM Cloud. This service supports the newer IAM-based authentication service.

Let’s look at how to bind authentication credentials for an instance of this service to an action.

Using the CLI, we can check an instance of this service is available…

1
2
3
4
5
$ ibmcloud resource service-instances
Retrieving service instances in resource group default..
OK
Name                     Location   State    Type               Tags
my-cos-storage           global     active   service_instance

In this example, we have a single instance of IBM Cloud Object Storage provisioned as my-cos-storage.

Retrieving instance details will show us the service name to use in the service binding command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ibmcloud resource service-instance my-cos-storage
Retrieving service instance my-cos-storage in resource group default..
OK

Name:                  my-cos-storage
ID:                    crn:v1:bluemix:public:cloud-object-storage:global:<GUID>:
GUID:                  <GUID>
Location:              global
Service Name:          cloud-object-storage
Service Plan Name:     lite
Resource Group Name:   default
State:                 active
Type:                  service_instance
Tags:

The IBM Cloud Object Storage service name is cloud-object-storage.

Before we can bind service credentials, we need to verify service credentials are available for this instance.

1
2
3
4
5
$ ibmcloud resource service-keys --instance-name my-cos-storage
Retrieving service keys in resource group default...
OK
Name                     State    Created At
serverless-credentials   active   Tue Jun  5 09:11:06 UTC 2018

This instance has a single service key available, named serverless-credentials.

Retrieving the service key details shows us the API secret for this credential.

1
2
3
4
5
6
7
8
9
10
11
$ ibmcloud resource service-key serverless-credentials
Retrieving service key serverless-credentials in resource group default...
OK

Name:          serverless-credentials
ID:            <ID>
Created At:    Tue Jun  5 09:11:06 UTC 2018
State:         active
Credentials:
               ...
               apikey:                   <SECRET_API_KEY_VALUE>

apikey denotes the secret API key used to authenticate calls to the service API.

Having retrieved the service name, instance identifier and available credentials, we can use these values to bind credentials to an action.

1
2
$ bx wsk service bind cloud-object-storage params --instance my-cos-storage --keyname serverless-credentials
Credentials 'serverless-credentials' from 'cloud-object-storage' service instance 'my-cos-storage' bound to 'params'.

Retrieving action details shows default parameters bound to an action. These will now include the API key for the Cloud Object Storage service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ bx wsk action get params
ok: got action params
{
  ...
  "parameters": [{
    "key": "__bx_creds",
    "value": {
      "cloud-object-storage": {
        "apikey": "<API_KEY_SECRET>",
        ...
      }
    }
  }]
}

Under the __bx_creds default parameter, there is a cloud-object-storage property with the API key amongst other service credential values.

Using Cloud Object Storage From IBM Cloud Functions (Node.js)

How do you manage files for a serverless application? 🤔

Previous blog posts discussed this common problem and introduced the most popular solution, using a cloud-based object storage service. 👏👏👏

Object stores provide elastic storage in the cloud, with a billing model which charges for capacity used. These services are the storage solution for serverless applications, which do not have access to a traditional file system. 👍

I’m now going to demonstrate how to use IBM Cloud Object Storage from IBM Cloud Functions.

This blog post will show you…

  • How to provision IBM Cloud Object Storage and create authentication tokens.
  • How use client libraries to access IBM Cloud Object Storage from IBM Cloud Functions.
  • Example serverless functions for common use-cases, e.g uploading files.

Code examples in this blog post will focus on the Node.js runtime.

Instructions on service provisioning and authentication credentials are relevant for any runtime.

IBM Cloud Accounts and Storage Services

IBM Cloud Object Storage is available to all IBM Cloud users.

IBM Cloud has three different account types: lite, pay-as-you-go or subscription.

Lite Accounts

Lite accounts do not require a credit card to register and do not expire after a limited time period.

Numerous platform services, including Cloud Object Storage, provide free resources for lite account users. IBM Cloud Object Storage’s free resource tier comes the following monthly limits.

  • Store 25GB of new data.
  • Issue 20,000 GET and 2,000 PUT requests.
  • Use 10GB of public bandwidth.

Lite tier usage supports all resiliency and storage class options but are limited to a single service instance.

Users can sign up for a free “Lite” account here. Please follow the instructions to install the IBM Cloud CLI.

Pay-as-you-Go & Subscription Accounts

Lite accounts can be upgraded to Pay-As-You-Go or Subscription accounts. Upgraded accounts still have access to the free tiers provided in Lite accounts. Users with Pay-As-You-Go or Subscriptions accounts can access services and tiers not included in the Lite account.

Benefits of the additional service tiers for IBM Cloud Object Storage include unlimited instances of the object storage service. Costs are billed according to usage per month. See the pricing page for more details: https://www.ibm.com/cloud-computing/bluemix/pricing-object-storage#s3api

Provisioning IBM Cloud Object Storage

IBM Cloud Object Storage can be provisioned through the IBM Cloud service catalog.

From the Service Details page, follow these instructions to provision a new instance.

  • Give the service an identifying name.
  • Leave the resource group as ”default”.
  • Click the “Create” button.

Once the service has been provisioned, it will be shown under the “Services” section of the IBM Cloud Dashboard. IBM Cloud Object Storage services are global services and not bound to individual regions.

  • Click the service instance from the dashboard to visit the service management page.

Once the service has been provisioned, we need to create authentication credentials for external access…

Service Credentials

Service credentials for IBM Cloud Object Storage use IBM Cloud’s IAM service.

I’m just going to cover the basics of using IAM with Cloud Object Storage. Explaining all the concepts and capabilities of the IAM service would need a separate (and lengthy) blog post!

Auto-Binding Service Credentials

IBM Cloud Functions can automatically provision and bind service credentials to actions.

This feature is supported through the IBM Cloud CLI command: bx wsk service bind.

Bound service credentials are stored as default action parameters. Default parameters are automatically included as request parameters for each invocation.

Using this approach means users do not have to manually provision and manage service credentials. 👍

Service credentials provisioned in this manner use the following configuration options:

  • IAM Role: Manager
  • Optional Configuration Parameters: None.

If you need to use different configuration options, you will have to manually provision service credentials.

Manually Creating Credentials

  • Select the ”Service Credentials” menu item from the service management page.
  • Click the “New credential” button.

Fill in the details for the new credentials.

  • Choose an identifying name for the credentials.
  • Select an access role. Access roles define which operations applications using these credentials can perform. Permissions for each role are listed in the documentation.

    Note: If you want to make objects publicly accessible make sure you use the manager permission.

  • Leave the Service ID unselected.

If you need HMAC service keys, which are necessary for generating presigned URLs, use the following inline configuration parameters before. Otherwise, leave this field blank.

1
{"HMAC": true}
  • Click the “Add” button.

🔐 Credentials shown in this GIF were deleted after the demo (before you get any ideas…) 🔐

Once created, new service credentials will be shown in the credentials table.

IBM Cloud Object Storage API

Cloud Object Storage exposes a HTTP API for interacting with buckets and files.

This API implements the same interface as AWS S3 API.

Service credentials created above are used to authenticate requests to the API endpoints. Full details on the API operations are available in the documentation.

HTTP Endpoints

IBM Cloud Object Storage’s HTTP API is available through region-based endpoints.

When creating new buckets to store files, the data resiliency for the bucket (and therefore the files within it) is based upon the endpoint used for the bucket create operation.

Current endpoints are listed in the external documentation and available through an external API: https://cos-service.bluemix.net/endpoints

Choosing an endpoint

IBM Cloud Functions is available in the following regions: US-South, United Kingdom and Germany.

Accessing Cloud Object Storage using regional endpoints closest to the Cloud Functions application region will result in better application performance.

IBM Cloud Object Storage lists public and private endpoints for each region (and resiliency) choice. IBM Cloud Functions only supports access using public endpoints.

In the following examples, IBM Cloud Functions applications will be hosted in the US-South region. Using the US Regional endpoint for Cloud Object Storage will minimise network latency when using the service from IBM Cloud Functions.

This endpoint will be used in all our examples: s3-api.us-geo.objectstorage.softlayer.net

Client Libraries

Rather than manually creating HTTP requests to interact with the Cloud Object Storage API, client libraries are available.

IBM Cloud Object Storage publishes modified versions of the Node.js, Python and Java AWS S3 SDKs, enhanced with IBM Cloud specific features.

Both the Node.js and Python COS libraries are pre-installed in the IBM Cloud Functions runtime environments for those languages. They can be used without bundling those dependencies in the deployment package.

We’re going to look at using the JavaScript client library from the Node.js runtime in IBM Cloud Functions.

JavaScript Client Library

When using the JavaScript client library for IBM Cloud Object Storage, endpoint and authentication credentials need to be passed as configuration parameters.

1
2
3
4
5
6
7
8
9
const COS = require('ibm-cos-sdk');

const config = {
    endpoint: '<endpoint>',
    apiKeyId: '<api-key>',
    serviceInstanceId: '<resource-instance-id>',
};

const cos = new COS.S3(config);

Hardcoding configuration values within source code is not recommended. IBM Cloud Functions allows default parameters to be bound to actions. Default parameters are automatically passed into action invocations within the event parameters.

Default parameters are recommended for managing application secrets for IBM Cloud Functions applications.

Having provisioned the storage service instance, learnt about service credentials, chosen an access endpoint and understood how to use the client library, there’s one final step before we can start to creating functions…

Creating Buckets

IBM Cloud Object Storage organises files into a flat hierarchy of named containers, called buckets. Buckets can be created through the command-line, using the API or the web console.

Let’s create a new bucket, to store all files for our serverless application, using the web console.

  • Open the ”Buckets” page from the COS management page.
  • Click the ”Create Bucket” link.

  • Create a bucket name. Bucket names must be unique across the entire platform, rather than just your account.

  • Select the following configuration options
    • Resiliency: Cross Region
    • Location: us-geo
    • Storage class: Standard
  • Click the ”Create” button.

Once the bucket has been created, you will be taken back to the bucket management page.

Test Files

We need to put some test files in our new bucket. Download the following images files.

Using the bucket management page, upload these files to the new bucket.

Using Cloud Object Storage from Cloud Functions

Having created a storage bucket containing test files, we can start to develop our serverless application.

Let’s begin with a serverless function that returns a list of files within a bucket. Once this works, we will extend the application to support retrieving, removing and uploading files to a bucket. We can also show how to make objects publicly accessible and generate pre-signed URLs, allowing external clients to upload new content directly.

Separate IBM Cloud Functions actions will be created for each storage operation.

Managing Default Parameters

Serverless functions will need the bucket name, service endpoint and authentication parameters to access the object storage service. Configuration parameters will be bound to actions as default parameters.

Packages can be used to share configuration values across multiple actions. Actions created within a package inherit all default parameters stored on that package. This removes the need to manually configure the same default parameters for each action.

Let’s create a new package (serverless-files) for our serverless application.

1
2
$ bx wsk package create serverless-files
ok: created package serverless-files

Update the package with default parameters for the bucket name (bucket) and service endpoint (cos_endpoint).

1
2
$ bx wsk package update serverless-files -p bucket <MY_BUCKET_NAME> -p cos_endpoint s3-api.us-geo.objectstorage.softlayer.net
ok: updated package serverless-files

Did you notice we didn’t provide authentication credentials as default parameters?

Rather than manually adding these credentials, the CLI can automatically provision and bind them. Let’s do this now for the cloud-object-storage service…

  • Bind service credentials to the serverless-files package using the bx wsk service bind command.
1
2
$ bx wsk service bind cloud-object-storage serverless-files
Credentials 'cloud-fns-key' from 'cloud-object-storage' service instance 'object-storage' bound to 'serverless-files'.
  • Retrieve package details to check default parameters contain expected configuration values.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ bx wsk package get serverless-files
ok: got package serverless-files
{
    ...
    "parameters": [
        {
            "key": "bucket",
            "value": "<MY_BUCKET_NAME>"
        },
        {
            "key": "cos_endpoint",
            "value": "s3-api.us-geo.objectstorage.softlayer.net"
        },
        {
            "key": "__bx_creds",
            "value": {
                "cloud-object-storage": {
                    ...
                }
            }
        }
    ]
}

List Objects Within the Bucket

  • Create a new file (action.js) with the following contents.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
const COS = require('ibm-cos-sdk')

function cos_client (params) {
  const bx_creds = params['__bx_creds']
  if (!bx_creds) throw new Error('Missing __bx_creds parameter.')

  const cos_creds = bx_creds['cloud-object-storage']
  if (!cos_creds) throw new Error('Missing cloud-object-storage parameter.')

  const endpoint = params['cos_endpoint']
  if (!endpoint) throw new Error('Missing cos_endpoint parameter.')

  const config = {
    endpoint: endpoint,
    apiKeyId: cos_creds.apikey,
    serviceInstanceId: cos_creds.resource_instance_id
  }

  return new COS.S3(config);
}

function list (params) {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  const client = cos_client(params)

  return client.listObjects({ Bucket: params.bucket }).promise()
    .then(results => ({ files: results.Contents }))
}

This action retrieves the bucket name, service endpoint and authentication credentials from invocation parameters. Errors are returned if those parameters are missing.

  • Create a new package action from this source file with the following command.
1
2
$ bx wsk action create serverless-files/list-files actions.js --main list --kind nodejs:8
ok: created action list-files

The —main flag set the function name to call for each invocation. This defaults to main. Setting this to an explicit value allows us to use a single source file for multiple actions.

The —kind sets the action runtime. This optional flag ensures we use the Node.js 8 runtime rather than Node.js 6, which is the default for JavaScript actions. The IBM Cloud Object Storage client library is only included in the Node.js 8 runtime.

  • Invoke the new action to verify it works.
1
2
3
4
5
6
7
8
$ bx wsk action invoke serverless-files/list-files -r
{
    "files": [
        { "Key": "jumping pug.jpg", ... },
        { "Key": "pug blanket.jpg", ... },
        { "Key": "swimming pug.jpg", ... }
    ]
}

The action response should contain a list of the files uploaded before. 💯💯💯

Retrieve Object Contents From Bucket

Let’s add another action for retrieving object contents from a bucket.

  • Add a new function (retrieve) to the existing source file (action.js) with the following source code.
1
2
3
4
5
6
7
8
function retrieve (params) {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  if (!params.name) throw new Error("Missing name parameter.")
  const client = cos_client(params)

  return client.getObject({ Bucket: params.bucket, Key: params.name }).promise()
    .then(result => ({ body: result.Body.toString('base64') }))
}

Retrieving files needs a file name in addition to the bucket name. File contents needs encoding as a Base64 string to support returning in the JSON response returned by IBM Cloud Functions.

  • Create an additional action from this updated source file with the following command.
1
2
$ bx wsk action create serverless-files/retrieve-file actions.js --main retrieve --kind nodejs:8
ok: created action serverless-files/retrieve-file
  • Invoke this action to test it works, passing the parameter name for the file to retrieve.
1
2
3
4
$ bx wsk action invoke serverless-files/retrieve-file -r -p name "jumping pug.jpg"
{
    "body": "<BASE64 ENCODED STRING>"
}

If this is successful, a (very long) response body containing a base64 encoded image should be returned. 👍

Delete Objects From Bucket

Let’s finish this section by adding a final action that removes objects from our bucket.

  • Update the source file (actions.js) with this additional function.
1
2
3
4
5
6
7
function remove (params) {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  if (!params.name) throw new Error("Missing name parameter.")
  const client = cos_client(params)

  return client.deleteObject({ Bucket: params.bucket, Key: params.name }).promise()
}
  • Create a new action (remove-file) from the updated source file.
1
2
$ bx wsk action create serverless-files/remove-file actions.js --main remove --kind nodejs:8
ok: created action serverless-files/remove-file
  • Test this new action using it to remove a file from the bucket.
1
2
$ bx wsk action invoke serverless-files/remove-file -r -p name "jumping pug.jpg"
{}
  • Listing bucket files should now return two files, rather than three.
1
2
3
4
5
6
7
$ bx wsk action invoke serverless-files/list-files -r
{
    "files": [
        { "Key": "pug blanket.jpg", ... },
        { "Key": "swimming pug.jpg", ... }
    ]
}

Listing, retrieving and removing files using the client library is relatively simple. Functions just need to call the correct method passing the bucket and object name.

Let’s move onto a more advanced example, creating new files in the bucket from our action…

Create New Objects Within Bucket

File content will be passed into our action as Base64 encoded strings. JSON does not support binary data.

When creating new objects, we should set the MIME type. This is necessary for public access from web browsers, something we’ll be doing later on. Node.js libraries can calculate the correct MIME type value, rather than requiring this as an invocation parameter.

  • Update the source file (action.js) with the following additional code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
const mime = require('mime-types');

function upload (params) {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  if (!params.name) throw new Error("Missing name parameter.")
  if (!params.body) throw new Error("Missing object parameter.")

  const client = cos_client(params)
  const body = Buffer.from(params.body, 'base64')

  const ContentType = mime.contentType(params.name) || 'application/octet-stream'
  const object = {
    Bucket: params.bucket,
    Key: params.name,
    Body: body,
    ContentType
  }

  return client.upload(object).promise()
}

exports.upload = upload;

As this code uses an external NPM library, we need to create the action from a zip file containing source files and external dependencies.

  • Create a package.json file with the following contents.
1
2
3
4
5
6
7
{
  "name": "upload-files",
  "main": "actions.js",
  "dependencies": {
    "mime-types": "^2.1.18"
  }
}
  • Install external libraries in local environment.
1
2
$ npm install
added 2 packages in 0.804s
  • Bundle source file and dependencies into zip file.
1
2
3
4
$ zip -r upload.zip package.json actions.js node_modules
  adding: actions.js (deflated 72%)
  adding: node_modules/ (stored 0%)
  ...
  • Create a new action from the zip file.
1
2
$ bx wsk action create serverless-files/upload-file upload.zip --main upload --kind nodejs:8
ok: created action serverless-files/upload-file
  • Create the Base64-encoded string used to pass the new file’s content.
1
2
$ wget http://www.pugnow.com/wp-content/uploads/2016/04/fly-pug-300x300.jpg
$ base64 fly-pug-300x300.jpg > body.txt
  • Invoke the action with the file name and content as parameters.
1
$ bx wsk action invoke serverless-files/upload-file -r -p body $(cat body.txt) -p name "flying pug.jpg"

Object details should be returned if the file was uploaded correctly.

1
2
3
4
5
6
7
{
    "Bucket": "my-serverless-files",
    "ETag": "\"b2ae0fb61dc827c03d6920dfae58e2ba\"",
    "Key": "flying pug.jpg",
    "Location": "https://<MY_BUCKET_NAME>.s3-api.us-geo.objectstorage.softlayer.net/flying%20pug.jpg",
    "key": "flying pug.jpg"
}

Accessing the object storage dashboard shows the new object in the bucket, with the correct file name and size.

Having actions to create, delete and access objects within a bucket, what’s left to do? 🤔

Expose Public Objects From Buckets

Users can also choose to make certain objects within a bucket public. Public objects can be retrieved, using the external HTTP API, without any further authentication.

Public file access allows external clients to access files directly. It removes the need to invoke (and pay for) a serverless function to serve content. This is useful for serving static assets and media files.

Objects have an explicit property (x-amz-acl) which controls access rights. Files default to having this value set as private, meaning all operations require authentication. Setting this value to public-read will enable GET operations without authentication.

Files can be created with an explicit ACL property using credentials with the Writer or Manager role. Modifying ACL values for existing files is only supported using credentials with the Manager role.

  • Add the following source code to the existing actions file (action.js).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
function make_public (params) {
  return update_acl(params, 'public-read')
}

function make_private (params) {
  return update_acl(params, 'private')
}

function update_acl (params, acl) => {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  if (!params.name) throw new Error("Missing name parameter.")
  const client = cos_client(params)

  const options = {
    Bucket: params.bucket,
    Key: params.name,
    ACL: acl
  }

  return client.putObjectAcl(options).promise()
}
  • Create two new actions with the update source file.
1
2
3
4
$ bx wsk action create serverless-files/make-public actions.js --main make_public --kind nodejs:8
ok: created action serverless-files/make-public
$ bx wsk action create serverless-files/make-private actions.js --main make_private --kind nodejs:8
ok: created action serverless-files/make-private

Bucket objects use the following URL scheme: https://./

We have been using the following endpoint hostname: s3-api.us-geo.objectstorage.softlayer.net.

  • Checking the status code returned when accessing an existing object confirms it defaults to private.
1
2
3
$ curl -I https://<BUCKET_NAME>.s3-api.us-geo.objectstorage.softlayer.net/flying%20pug.jpg
HTTP/1.1 403 Forbidden
...
  • Invoke the make-public action to allow GET requests without authentication.
1
$ bx wsk action invoke serverless-files/make-public -r -p name "flying pug.jpg"
  • Retry file access using the external HTTP API. This time a 200 response is returned with the content.
1
2
3
4
$ curl -I https://<BUCKET_NAME>.s3-api.us-geo.objectstorage.softlayer.net/flying%20pug.jpg
HTTP/1.1 200 OK
Content-Type: image/jpeg
...

Having set an explicit content type for the file, opening this URL in a web browser will show the image.

  • Disable public access using the other new action.
1
bx wsk action invoke serverless-files/make-private -r -p name "flying pug.jpg"
  • Re-issue the curl request to the file location.
1
2
3
$ curl -I https://<BUCKET_NAME>.s3-api.us-geo.objectstorage.softlayer.net/flying%20pug.jpg
HTTP/1.1 403 Forbidden
...

HTTP requests to this file now return a 403 status. Authentication is required again. 🔑

In addition to allowing public read access we can go even further in allowing clients to interact with buckets…

Provide Direct Upload Access To Buckets

Cloud Object Storage provides a mechanism (presigned URLs) to generate temporary links that allow clients to interact with buckets without further authentication. Passing these links to clients means they can access to private objects or upload new files to buckets. Presigned URLs expire after a configurable time period.

Generating presigned URLs is only supported from HMAC authentication keys.

HMAC service credentials must be manually provisioned, rather than using the bx wsk service bind command. See above for instructions on how to do this.

  • Save provisioned HMAC keys into a file called credentials.json.

Let’s create an action that returns presigned URLs, allowing users to upload files directly. Users will call the action with a new file name. Returned URLs will support an unauthenticated PUT request for the next five minutes.

  • Create a new file called presign.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
'use strict';

const COS = require('ibm-cos-sdk');
const mime = require('mime-types');

function cos_client (params) {
  const creds = params.cos_hmac_keys
  if (!creds) throw new Error('Missing cos_hmac_keys parameter.')

  const endpoint = params.cos_endpoint
  if (!endpoint) throw new Error('Missing cos_endpoint parameter.')

  const config = {
    endpoint: endpoint,
    accessKeyId: creds.access_key_id,
    secretAccessKey: creds.secret_access_key
  }

  return new COS.S3(config);
}

function presign (params) {
  if (!params.bucket) throw new Error("Missing bucket parameter.")
  if (!params.name) throw new Error("Missing name parameter.")

  const client = cos_client(params)

  const options = {
    Bucket: params.bucket,
    Key: params.name,
    Expires: 300,
    ContentType: mime.contentType(params.name) || 'application/octet-stream'
  }

  return { url: client.getSignedUrl('putObject', options) }
}

exports.presign = presign;
  • Update the package.json file with the following contents.
1
2
3
4
5
6
7
{
  "name": "presign",
  "main": "presign.js",
  "dependencies": {
    "mime-types": "^2.1.18"
  }
}
  • Bundle source file and dependencies into zip file.
1
2
3
4
$ zip -r presign.zip package.json presign.js node_modules
  adding: actions.js (deflated 72%)
  adding: node_modules/ (stored 0%)
  ...
  • Create a new action from the zip file.
1
2
$ bx wsk action create serverless-files/presign presign.zip --main presign --kind nodejs:8 -P credentials.json
ok: created action serverless-files/presign
  • Invoke the action to return a presigned URL for a new file.
1
2
3
4
$ bx wsk action invoke serverless-files/presign -r -p name pug.jpg
{
    "url": "https://<BUCKET>.s3-api.us-geo.objectstorage.softlayer.net/pug.jpg?AWSAccessKeyId=<SECRET>&Content-Type=image%2Fjpeg&Expires=<TIME>&Signature=<KEY>"
}

Using this URL we can upload a new image without providing authentication credentials.

  • This curl command —upload-file will send a HTTP PUT, with image file as request body, to that URL.
1
$ curl --upload-file "my pug.jpg" <URL> --header "Content-Type: image/jpeg"

The HTTP request must include the correct “Content-Type” header. Use the value provided when creating the presigned URL. If these values do not match, the request will be rejected.

Exploring the objects in our bucket confirms we have uploaded a file! 🕺💃

Presigned URLs are a brilliant feature of Cloud Object Storage. Allowing users to upload files directly overcomes the payload limit for cloud functions. It also reduces the cost for uploading files, removing the cloud functions’ invocation cost.

conclusion

Object storage services are the solution for managing files with serverless applications.

IBM Cloud provides both a serverless runtime (IBM Cloud Functions) and an object storage service (IBM Cloud Object Store). In this blog post, we looked at how integrate these services to provide a file storage solution for serverless applications.

We showed you how to provision new COS services, create and manage authentication credentials, access files using a client library and even allow external clients to interact directly with buckets. Sample serverless functions using the Node.js runtime were also provided.

Do you have any questions, comments or issues about the content above? Please leave a comment below, find me on the openwhisk slack or send me a tweet.

File Storage for Serverless Applications

“Where do you store files without a server?”

…is the most common question I get asked during Q&A after one of my ”Introduction to Serverless Platforms” conference talks. Searching for this question online, this is the answer you will often find.

“Use an object store for file storage and access using the S3-compatible interface. Provide direct access to files by making buckets public and return pre-signed URLs for uploading content. Easy, right?”

Responding to people with this information often leads to the following response:

🤔🤔🤔

Developers who are not familiar with cloud platforms, can often understand the benefits and concepts behind serverless, but don’t know the other cloud services needed to replicate application services from traditional (or server-full) architectures.

In this blog post, I want to explain why we do not use the file system for files in serverless applications and introduce the cloud services used to handle this.

serverless runtime file systems

Serverless runtimes do provide access to a filesystem with a (small) amount of ephemeral storage.

Serverless application deployment packages are extracted into this filesystem prior to execution. Uploading files into the environment relies on them being included within the application package. Serverless functions can read, modify and create files within this local file system.

These temporary file systems come with the following restrictions…

  • Maximum application package size limits additional files that can be uploaded.
  • Serverless platforms usually limit total usable space to around 512MB.
  • Modifications to the file system are lost once the environment is not used for further invocations.
  • Concurrent executions of the same function use independent runtime environments and do not share filesystem storage.
  • There is no access to these temporary file systems outside the runtime environment.

All these limitations make the file system provided by serverless platforms unsuitable as a scalable storage solution for serverless applications.

So, what is the alternative?

object stores

Object stores manage data as objects, as opposed to other storage architectures like file systems which manage data as a file hierarchy. Object-storage systems allow retention of massive amounts of unstructured data, with simple retrieval and search capabilities.

https://en.wikipedia.org/wiki/Object_storage

Object stores provide “storage-as-a-service” solutions for cloud applications.

These services are used for file storage within serverless applications.

Unlike traditional block storage devices, data objects in object storage services are organised using flat hierarchies of containers, known as ”buckets”. Objects within buckets are identified by unique identifiers, known as ”keys”. Metadata can also be stored alongside data objects for additional context.

Object stores provide simple access to files by applications, rather than users.

advantages of an object store

scalable and elastic storage

Rather than having a disk drive, with a fixed amount of storage, object stores provide scalable and elastic storage for data objects. Users are charged based upon the amount of data stored, API requests and bandwidth used. Object stores are built to scale as storage needs grow towards the petabyte range.

simple http access

Object stores provide a HTTP-based API endpoint to interact with the data objects.

Rather than using a standard library methods to access the file system, which translates into system calls to the operating system, files are available over a standard HTTP endpoint.

Client libraries provide a simple interface for interacting with the remote endpoints.

expose direct access to files

Files stored in object storage can be made publicly accessible. Client applications can access files directly without needing to use an application backend as a proxy.

Special URLs can also be generated to provide temporary access to files for external clients. Clients can even use these URLs to directly upload and modify files. URLs are set to expire after a fixed amount of time.

ibm cloud object storage

IBM Cloud provides an object storage service called IBM Cloud Object Storage. This service provides the following features concerning resiliency, reliability and cost.

data resiliency

Buckets’ contents can be stored with the following automatic data resiliency choices.

  • Cross Region. Store data across three regions within a geographic area.
  • Regional. Store data in multiple data centres within a single geographic region.
  • Single Data Centre. Store data across multiple devices in a single data centre.

Cross Region is the best choice for ”regional concurrent access and highest availability”. Regional is used for “high availability and performance”. Single Data Centre is appropriate when “when data locality matters most”.

storage classes

Data access patterns can be used to save costs by choosing the appropriate storage class for data storage.

IBM Cloud Object Storage offers the following storage classes: Standard, Vault, Cold Vault, Flex.

Standard class is used for workloads with frequent data access. Vault and Cold Vault are used with infrequent data retrieval and data archiving workloads. Flex is a mixed storage class for workloads where access patterns are more difficult to predict.

costs

Storage class and data resiliency options are used to calculate the cost of service usage.

Storage is charged based upon the amount of data storage used, operational requests (GET, POST, PUT…) and outgoing public bandwidth.

Storage classes affect the price of data retrieval operations and storage costs. Storage classes used for archiving, e.g. cold vault, charge less for data storage and more for operational requests. Storage classes used for frequency access, e.g. standard, charge more for data storage and less for operational requests.

Higher resiliency data storage is more expensive than lower resiliency storage.

lite plan

IBM Cloud Object Storage provides a generous free tier (25GB storage per month, 5GB public bandwidth) for Lite account users. IBM Cloud Lite accounts provide perpetual access to a free set of IBM Cloud resources. Lite accounts do not expire after a time period or need a credit card to sign up.

conclusion

Serving files from serverless runtimes is often accomplished using object storage services.

Object stores provide a scalable and cost-effective service for managing files without using storage infrastructure directly. Storing files in an object store provides simple access from serverless runtimes and even allows the files to be made directly accessible to end users.

In the next blog posts, I’m going to show you how to set up IBM Cloud Object Storage and access files from serverless applications on IBM Cloud Functions. I’ll be demonstrating this approach for both the Node.js and Swift runtimes.

Configuring Alert Notifications Using Serverless Metrics

This blog post is the final part of a series on “Monitoring Serverless Applications Metrics”. See the introduction post for details and links to other posts.

In previous blog posts, we showed how to capture serverless metrics from IBM Cloud Functions, send those values into the IBM Cloud Monitoring service and build visualisation dashboards using Grafana.

Dashboards are a great way to monitor metrics but rely on someone watching them! We need a way to be alerted to issues without having to manually review dashboards.

Fortunately, IBM Cloud Monitoring service comes with an automatic alerting mechanism. Users configure rules that define metrics to monitor and expected values. When values fall outside normal ranges, alerts are sent using installed notification methods.

Let’s finish off this series on monitoring serverless applications by setting up a sample alert notification monitoring errors from our serverless applications…

Alerting in IBM Cloud Monitoring

IBM Cloud Monitoring service supports defining custom monitoring alerts. Users define rules to identify metric values to monitor and expected values. Alerts are triggered when metric values fall outside thresholds. Notification methods including email, webhooks and PagerDuty are supported.

Let’s set up a sample monitoring alert for IBM Cloud Functions applications.

We want to be notified when actions start to return error codes, rather than successful responses. The monitoring library already records boolean values for error responses from each invocation.

Creating monitoring alerts needs us to use the IBM Cloud Monitoring API.

Using the IBM Cloud Monitoring API needs authentication credentials and a space domain identifier. In a previous blog post, we showed how to retrieve these values.

Monitoring Rules API

Monitoring rules can be registered by sending a HTTP POST request to the /alert/rule endpoint.

Configuration parameters are included in the JSON body. This includes the metric query, threshold values and monitoring time window. Rules are connected to notification methods using notification identifiers.

This is an example rule configuration for monitoring errors from IBM Cloud Function applications.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "name": "ibm_cloud_functions",
  "description": "Monitor errors from all actions",
  "expression": "sumSeries(ibm.public.cloud-functions.<region>.<namespace>.*.*.error)",
  "enabled": true,
  "from": "-5min",
  "until": "now",
  "comparison": "above",
  "comparison_scope": "last",
  "error_level" : 10,
  "warning_level" : 1,
  "frequency": "1min",
  "dashboard_url": "https://metrics.ng.bluemix.net",
  "notifications": [
    "email_alert"
  ]
}

The expression parameter defines the query used to monitor values.

1
sumSeries(ibm.public.cloud-functions.<region>.<namespace>.*.*.error)

Error metric values use 0 for normal responses and 1 for errors. sumSeries adds up all error values recorded within the monitoring window.

Using a wildcard for the sixth field means all actions are monitored. Replacing this field value with an action name will restrict monitoring to just that action. Region and namespace templates need substituting with actual values for your application.

Threshold values for triggering alerts are defined using the warning_level and error_level parameters. Warning messages are triggered after a single action failure and error messages after ten failures.

Notification identifiers, registered using the API, are provided in the notifications field. Rules may include more than one notification identifiers.

Notifications API

Notifications can be registered by sending a HTTP POST request to the /alert/notification endpoint. Configuration parameters are included in the JSON body.

This is an example configuration for email notifications.

1
2
3
4
5
6
{
  "name": "email_alert",
  "type": "Email",
  "description" : "Email alerting notifications",
  "detail": "email@address.com"
}

Notifications are configured using the type parameter in the body. Valid values for this field include Email, Webhook or PagerDuty. The detail field is used to include the email address, webhook endpoint or PagerDuty API key. The name field is used to reference this notification method when setting up rules.

Setting up alerts for serverless errors

Creating an email notification

  • Create the notify.json file from the template above.
1
2
3
4
5
6
7
$ cat notify.json
{
  "name": "email_alert",
  "type": "Email",
  "description" : "Email alerting notifications",
  "detail": "your_email@address.com"
}
  • Send the following HTTP request using curl. Include scope and auth token values in the headers.
1
2
3
4
5
6
7
8
9
$ curl --request POST \
    --url https://metrics.ng.bluemix.net/v1/alert/notification \
    --header 'x-auth-scope-id: s-<YOUR_DOMAIN_SPACE_ID>' \
    --header 'x-auth-user-token: apikey <YOUR_API_KEY>' \
    --data @notify.json
{
  "status": 200,
  "message": "Created notification 'email_alert'"
}

Testing email notification

  • Sending the following HTTP request using curl to generate a test email.
1
2
3
4
$ curl --request POST \
    --url https://metrics.ng.bluemix.net/v1/alert/notification/test/email_alert \
    --header 'x-auth-scope-id: s-<YOUR_DOMAIN_SPACE_ID>' \
    --header 'x-auth-user-token: apikey <YOUR_API_KEY>'
  • This returns the test notification message which will be emailed to the address.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
    "status": 200,
    "message": "Triggered test for notification 'email_alert'",
    "content": {
      "rule_name": "test_rule_name",
      "description": "test_rule_description",
      "notification_name": "email_alert",
      "scope_id": "s-<YOUR_DOMAIN_SPACE_ID>",
      "expression": "test_rule_expression",
      "warning_level": "80",
      "error_level": "90.9",
      "dashboard_url": "https://metrics.ng.bluemix.net",
      "alert_messages": [
        {
          "target": "test_alert_target",
          "from_type": "OK",
          "to_type": "ERROR",
          "current_value": "95.0",
          "comparison": "above",
          "timestamp": "2018-01-25T12:36:05Z"
        }
      ]
    }
}
  • Check the email inbox to verify the message has arrived.

Create monitoring rule for errors

  • Create the rule.json file from the template above, replacing region and namespace values.

  • Send the following HTTP request using curl. Include scope and auth token values in the headers.

1
2
3
4
5
6
7
8
$ curl --request POST --url https://metrics.ng.bluemix.net/v1/alert/rule \
    --header 'x-auth-scope-id: s-<YOUR_DOMAIN_SPACE_ID>' \
    --header 'x-auth-user-token: apikey <YOUR_API_KEY>' \
    --data @rule.json
{
  "status": 200,
  "message": "Created rule 'ibm_cloud_functions'"
}

Testing alerts for serverless errors

Let’s generate some errors in a sample action to check the monitoring rule works.

Create failing action

  • Create a new Node.js library called “fails”.
1
$ mkdir fails && cd fails && npm init
  • Install the openwhisk-metrics library.
1
$ npm install openwhisk-metrics
  • Edit the index.js file to have the following source code.
1
2
3
4
5
6
7
const metrics = require('openwhisk-metrics')

const main = params => {
  return { error: 'Oh dear, this action failed...' }
}

exports.main = metrics(main)
1
2
3
4
5
$ zip -r action.zip *
  adding: index.js (deflated 22%)
  ...
$ bx wsk action create fails action.zip --kind nodejs:8
ok: created action fails
  • Invoke the action. Check the activation response is an error.
1
2
3
4
5
6
7
8
9
10
11
12
13
$ bx wsk action invoke fails -b
ok: invoked /_/fails with id cbee42f77c6543c6ae42f77c6583c6a7
{
  "activationId": "cbee42f77c6543c6ae42f77c6583c6a7",
  "response": {
    "result": {
      "error": "Oh dear, this action failed..."
    },
    "status": "application error",
    "success": false
  },
  ...
}

response.result.success should be false.

  • Update actions parameter for the metric-forwarder action to include the fails action name.
1
2
3
4
5
6
7
8
9
10
11
$ cat params.json
{
  "actions": ["fails"],
  "service": {
    "api_key": "<API_KEY>",
    "host": "metrics.ng.bluemix.net",
    "scope": "s-<SPACE_ID>"
  },
  "since": 1516894777975
}
$ wsk action update metric-forwarder -P params.json

Generate serverless errors

Invoking the fails action should now trigger an email notification. Let’s test this out and trace metrics values through the platform.

  • Fire an action invocation using the CLI.
1
2
3
4
$ wsk action invoke fails -b
bx wsk action invoke fails -b
ok: invoked /_/fails with id 524b27044fd84b6a8b27044fd84b6ad8
...
  • Review the activation logs to show the error metric was recorded.
1
2
3
$ wsk activation logs 524b27044fd84b6a8b27044fd84b6ad8
...
stdout: METRIC <namespace>.fails.524b27044fd84b6a8b27044fd84b6ad8.error 1 1516895270
  • Invoke the metric-forwarder action to push metric values into the IBM Cloud Monitoring service.
1
2
$ bx wsk action invoke metric-forwarder -b
ok: invoked /_/metric-forwarder with id 295c47f05ea042849c47f05ea08284f0
  • Review activation logs to verify metric values were retrieved.
1
2
3
4
5
6
7
$ bx wsk activation logs 295c47f05ea042849c47f05ea08284f0
2018-01-25T15:51:47.160135346Z stdout: actions being monitored: [ 'fails' ]
2018-01-25T15:51:47.160177305Z stdout: retrieving logs since: 1516894777975
2018-01-25T15:51:47.290529179Z stdout: found 11 metric values from 1 activations
2018-01-25T15:51:47.291234046Z stdout: saving to metrics service -> metrics.ng.bluemix.net
2018-01-25T15:51:48.232790321Z stdout: saving metrics to service took: 941.169ms
2018-01-25T15:51:48.233334982Z stdout: updating since parameter: 1516895270458
  • Use the IBM Cloud Monitoring dashboard to show the error has been recorded.

  • Check your email inbox for the message showing the error notification!

  • Using the Cloud Monitoring API, we can retrieve the notification history to show this message was sent.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ curl --request POST --url https://metrics.ng.bluemix.net/v1/alert/history \
    --header 'x-auth-scope-id: s-<YOUR_DOMAIN_SPACE_ID>' \
    --header 'x-auth-user-token: apikey <YOUR_API_KEY>'
[
  {
    "from_level": "OK",
    "metric_name": "sumSeries(ibm.public.cloud-functions.<region>.<namespace>.*.*.error)",
    "notification_names": [
      "email_alert"
    ],
    "rule_name": "ibm_cloud_functions",
    "timestamp": "2018-01-23T15:29:48Z",
    "to_level": "WARN",
    "value": 1
  }
]

Invoking the fails action more than ten times will trigger a second alert when the rule moves from warning to error thresholds.

Conclusion

IBM Cloud Monitoring service supports sending notification alerts based upon application metric values. Configuring notifications rules, based upon our serverless metrics, ensures we will be alerted immediately when issues occur with our serverless applications. Notifications can be sent over email, webhooks or using PagerDuty.

In this series on “Monitoring Serverless Application Metrics”, we have shown you how to monitor serverless applications using IBM Cloud. Starting with capturing runtime metrics from IBM Cloud Functions, we then showed how to forward metrics into the IBM Cloud Monitoring service. Once metric values were being recorded, visualisation dashboards were built to help diagnose and resolve application issues. Finally, we configured automatic alerting rules to notify us over email as soon as issues developed.

Serverless applications are not “No Ops”, but “Different Ops”. Monitoring runtime metrics is still crucial. IBM Cloud provides a comprehensive set of tools for monitoring cloud applications. Utilising these services, you can create a robust monitoring solution for IBM Cloud Functions applications.

Lessons From West Berkshire Action for Refugees

For the past two years, I’ve been involved with a local charity (West Berkshire Action For Refugees) set up in response to the refugee crisis.

Started with the aim to send a single collection of aid to refugees in Calais, the group ended up sending thousands of boxes of aid to refugees all over Europe. Campaigning for West Berkshire Council to participate in the UK’s resettlement scheme for Syrian refugees also led to multiple refugee families being resettled locally. The group now runs a volunteer-led integration programme to assist them upon arrival.

WBAR became a second-job (albeit one with no remuneration, employee benefits or time off 😉) for me and many other members of the group. Running a charitable organisation solely with volunteers, working around full-time jobs, families and other commitments, has innate challenges. We never had enough time or resources to implement all the ideas we came up with!

Before Christmas, I stepped down as a trustee and from all other official roles within the group. Having been involved for over two years, from the first donation drive to becoming a registered charity, I was ready for a break.

Since stepping down, I’ve been thinking about the lessons we learnt about running a charitable organisation, staffed solely by volunteers, with minimal resources.

If I had to do it all again, here’s what I wished I’d known from the beginning…

managing volunteers

what did we learn?

Volunteers are the lifeblood of small charitable organisations. Organising a systematic process for recruiting volunteers is crucial to building a sustainable charitable organisation. Growing the size and scope of an organisation, without growing the volunteer base, is a strategy for burn out.

background

WBAR started during a period of intense media coverage on the ”refugee crisis”. People understood how dire the situation was and were desperate to help. The group was inundated with offers of assistance. The biggest challenge was simply responding to all these messages.

We did not need to recruit volunteers, they came to us.

People asking to help would be invited to meet the core team at a committee meeting. If they turned up, someone would hopefully try to find an opportunity for them within the group.

This process was not ideal for the following reasons…

  • Volunteer enquiries would often get missed by a core team busy with other activities.
  • Lacking a process for registering enquiries, it was difficult to coordinate and track the status of those people we did follow up with.
  • Committee meetings were also not the best place to “on-board” new volunteers.

Initially, with a constant stream of new volunteers coming forward, this was not an issue.

Skip forward twelve months…

When the media focus on the refugee crisis (predictably) disappeared, so did the number of people contacting the group to help. When the number of new volunteers shrank, the group’s activities did not…

As we took on more responsibilities, the more acute the need for new volunteers became, but the less time we had to focus on recruitment.

Eventually, due to existing volunteers stepping down or not being able to take on those extra roles, it became critical to actively recruit new volunteers, rather than passively waiting for them to come to us.

This was the point at which having an existing volunteer recruitment system would have been a huge benefit.

If we had been formally registering volunteer enquiries, including interviewing people to record their skills and availability, filling new roles would have been a quick and efficient process.

Without this database of potential volunteers, we were reliant on posting messages on social media asking for new volunteers. This caused a significant delay in staffing new roles whilst we waited for people to see and respond to the messages. Finding new volunteers felt slow and inefficient.

This issue led the group to appoint a formal volunteer coordinator. The coordinator is responsible for running a continual recruitment process and managing all volunteer enquiries. This ensures there is a recurring pipeline of potential recruits.

What should we have done differently?

Focused on volunteer recruitment before it became an acute issue.

Set up a systematic process for handling volunteer enquiries. Record all details of people contacting the group to build a pipeline of new recruits. Work on an outbound recruitment programme, with the group actively seeking volunteers for identified roles. Don’t be reliant on volunteers finding us.

using facebook

what did we learn?

Facebook is the new Excel. It has become the default platform for all online activities. Charitable causes are no different. Private groups and messenger enable distributed collaboration between remote volunteers.

Facebook as a collaboration tool struggles once groups reach a certain size. Moving to more appropriate tools, rather than continuing to work around the challenges, will be a necessity longer-term.

background

During the summer of 2015, when the refugee crisis became a front-page story, people all over the United Kingdom started collecting donations for Calais and other refugee camps across Europe.

Facebook became the platform for connecting volunteers throughout the country.

There were hundreds of public (and private) Facebook groups relating to the crisis. From pages for groups collecting aid in different towns across the country or finding organisations working in refugee camps needing aid to those offering lifts and accommodation for people volunteering in refugee camps.

There was a Facebook group for everything.

WBAR started when the founder created one of these pages in August 2015. The following week, I stumbled across the page whilst looking for a local group to offer assistance too. Sending a private message to the founder, I asked if there was anything I could do to help. Little did I know that the group would become a significant part of my life for the next two years…

This page became our main communication tool and grew to having over one thousand followers. Whether asking for donations, finding volunteers or highlighting our activities, Facebook made it simple to reach huge numbers of people within the local community with minimal effort or expense.

Facebook also became the default solution for coordinating group activities and volunteers.

There was private Facebook group for the core volunteers, committee members and trustees. Volunteers would post in the group with updates on activities, requests for assistance or other issues. Threads on posts were used to communicate between volunteers. Tagging members in posts would highlight where certain volunteers were needed to assist. Groups also allowed sharing files and other documents between members.

Facebook has a number of advantages over other (more appropriate) tools for digital collaboration.

  • Everyone is on Facebook. No-one has to sign up for an account with a new platform.
  • People understand how to use the site. No training is needed to on-board new members.
  • Facebook has web and mobile clients for all platforms. People can access the site using devices they prefer.
  • People already spend an average of fifty minutes a day on Facebook(!).

Making volunteers sign up for a new platform, learn how to use it and then remember to check daily for messages, would dramatically decrease engagement and collaboration.

But the limits of Facebook as a collaboration platform for the volunteers began to show as the organisation grew in size and scope.

These included, but were not limited to, the following issues…

  • Group posts with multiple levels of threaded comments are difficult to follow. It’s not obvious which comments are new without re-reading every comment.
  • Finding historical posts or specific comments often felt impossible. Facebook search did not support complex filtering operations. Manually scrolling through all items in the group was often the only way to find specific items.
  • Group notifications were often lost in the morass of other alerts people received. Facebook would not let you subscribe to notifications from specific groups or posts. Volunteers had to manually remind each other of notifications using other tools.

Spending my work time collaborating with distributed teams in open-source, I often wished we were using a Github project with issues, milestones and markdown support!

There is a plethora of more suitable collaboration tools on the Internet. However, new tools come with a “cognitive burden” on volunteers. Registration, training, device support and others issues need to be balanced against the benefits from using more appropriate platforms.

What should we have done differently?

Investigated additional tools to support distributed work flows between remote volunteers. Once the limitations of Facebook became apparent, rather than working around them, we should have worked out a better solution.

charitable incorporated organisations

what did we learn?

Becoming an official charitable organisation is inevitable as you grow in size and scope.

Registering with the charity commission is not a simple or fast process. It’s impossible to know how long your application will take to be confirmed. If registration is prerequisite for other activities, this can leave you in stuck for an indeterminable amount of time.

background

After twelve months, it became clear the group needed to register as an official charitable organisation. Registration opened up opportunities in applying to trusts for grants, became a requirement for projects we wanted to start and insurance purposes.

West Berks Action For Refugees chose to incorporate as a Charitable Incorporated Organisation (CIO).

Launched in 2012, CIOs were a new form of charitable organisation, with lighter regulation and reporting requirements. CIOs are administered by the Charity Commission, who have sole responsibility for their formation and registration. This reduces the administrative burden by not having to additionally register and report to Companies House, like Charitable Companies.

Registering a CIO with the Charity Commission was supposed to be an easy and efficient process. Unfortunately, cuts in Government funding has led to severe resource issues at the Charity Commission. Recent news indicated registration times were currently around three months.

It took West Berks Action For Refugees nearly six months to register as a CIO. This caused enormous problems for the group.

Opening an official bank account with The Co-Operative Bank required us to have registration confirmed. Until we had a bank account, it was difficult to receive official donations to the group. Other organisations often used cheques for donations, with the group’s name as the recipient. These were unable to be received without an official bank account.

Once the charity registration did come through, The Co-Operative Bank still took another six months to open the account!

Group activities also began to need insurance policies. For example, public liability insurance was a requirement for even the smallest public event, like a cake sale in the church hall. Insurers do provide specialist charity insurance but only for registered organisations. These policies were unavailable to us until the registration came through.

CIOs were set up to make registering a charitable organisation a quick and efficient process. Unfortunately, due to Government funding cuts, this is no longer the case. Whilst waiting for our registration to come through, the group had numerous challenges that we were unable to do anything about.

what should we have done differently?

Looked for a community bank account, that didn’t require being a registered charitable organisation. This would have resolved issues we faced processing donations.

Chosen a different charity bank account provider. The Co-Operative Bank were incredibly slow to process the account opening and have an awful online banking site for business accounts. I’ve heard similar complaints from other groups. Would not recommend!

governance & decision-making

what did we learn?

Organisations need to have an appropriate level of governance for their size and scope. Formal governance structures are a requirement for registered charitable organisations. Trustees need to have oversight on activities and keep a formal record of decisions.

Moving from an informal to a formal decision making process can lead to resistance from volunteers. It might appear that this added “bureaucracy” unnecessary slows down decision making.

background

The charity started as a small group of volunteers working on a single activity, collecting donations for refugees in Calais. People volunteered when they had time. Communication and coordination between volunteers happened in an ad-hoc sense.

An informal decision making process was a natural governance model for a group of that size and scope. When the group changed, in activities and responsibilities, the governance model needed to reflect that.

This started as a committee meeting every six weeks. Volunteers would attend to bring issues for the wider group to resolve. With people still working independently, this was often the only time people would regularly see each other.

This meeting was crucial to keeping the group running smoothly. Over time, we expanded the meeting to use a more formal process, with an explicit agenda and reports from the sub-committees. Minutes were noted to keep an official record of the meeting and provide an overview to those unable to attend.

There was often a tension between the formal decision-making process and the volunteers. People often wanted a decision on an issue immediately, rather than waiting for the next meeting. There was a pressure to make important decisions outside of the committee meetings. People were used to the informal decision making process we had started with.

Volunteers sometimes failed to engage with the new governance structure. People not attending meetings or sending reports into the group was a regular issue. Decisions would become repeatedly postponed, due to missing reports or non-attendance of members involved. This undermined the effectiveness of the governance structure, leading to further resistance.

Setting up a formal decision making process and governance structure for the charity was a legal requirement of incorporating as a CIO. The group needed a transparent decision making process, along with a formal record of decisions. However, moving away from an informal and ad-hoc decision making process did seem, to some people, like unnecessary bureaucracy and a burden on an already stretched group of volunteers.

What should we have done differently?

Moved earlier to use a more formal governance model. Officially documented the governance structure and decision making process. Explained to all volunteers how decisions need to be made within the group and the rationale for this approach.

Starting OpenWhisk in Sixty Seconds

Apache OpenWhisk is an open-source serverless platform. Developers can use hosted instances from IBM, deploy to any infrastructure provider and even run it locally.

Developers often use a local instance of the platform during development. Deploying to a local instance is faster than the cloud. It also provides access runtime environments to debug issues and allows development without an Internet connection. Production applications are still run on IBM Cloud Functions.

But OpenWhisk provides numerous options for starting the platform, including running the platform services directly, using container management tools like Kubernetes and Mesos or starting a pre-configured virtual machine with Vagrant.

So, what’s easiest?

OpenWhisk Devtools.

Using this project, the platform can be started on any machine with Docker Compose in around sixty seconds. Before we explain how this works, let’s show the steps needed to spin up the platform using the project.

openwhisk in around sixty seconds…

Do you have Docker with Compose support installed? If not, follow the instructions here.

Start the platform with the following commands.

1
2
3
$ git clone git@github.com:apache/incubator-openwhisk-devtools.git
$ cd incubator-openwhisk-devtools/docker-compose
$ make quick-start

Having cloned the repository, creating the local instance only takes around sixty seconds! 💯

1
2
3
4
5
$ time make quick-start &>/dev/null

real    1m10.128s
user    0m1.709s
sys     0m1.258s

Platform services will be running as containers on the host after initialisation.

1
2
3
4
5
6
7
8
9
10
11
12
$ docker ps --format "{{.ID}}: {{.Names}} {{.Image}}" 
17c5d31e2c20: wsk0_60_prewarm_nodejs6         (openwhisk/nodejs6action:latest)
0eace484289c: wsk0_59_prewarm_nodejs6         (openwhisk/nodejs6action:latest)
1be725d8767c: openwhisk_apigateway_1          (adobeapiplatform/apigateway:1.1.0)
641cbabeb790: openwhisk_kafka-topics-ui_1     (landoop/kafka-topics-ui:0.9.3)
f52c25dbadd9: openwhisk_controller_1          (openwhisk/controller)
8f0c6aa14ccc: openwhisk_invoker_1             (openwhisk/invoker)
d5274194f842: openwhisk_kafka-rest_1          (confluentinc/cp-kafka-rest:3.3.1)
40a1585f64bb: openwhisk_kafka_1               (wurstmeister/kafka:0.11.0.1)
b0b0f75c6fdb: openwhisk_db_1                  (couchdb:1.6)
a7449c2edc4d: openwhisk_zookeeper_1           (zookeeper:3.4)
178abe09b793: openwhisk_redis_1               (redis:2.8)

…and that’s it!

testing it out

setting up CLI tool

OpenWhisk provides a CLI tool for interacting with the platform. The quick-start command automatically writes account credentials for the local instance into the CLI configuration file. Using the CLI tool to print current configuration values shows the platform endpoint set as the local machine ip or hostname.

If you don’t have the CLI tool already installed, the project downloads the binary to the following location: devtools/docker-compose/openwhisk-master/bin/wsk

1
2
$ wsk property get | grep host
whisk API host        localhost

The local instance is configured with a single user account (guest) with these credentials. Administrative credentials are stored in this configuration file.

creating sample actions

With the CLI configured correctly, you can create and invoke an action on the local platform instance.

  • Create a new file called hello.js with the following contents.
1
2
3
4
function main (params) {
  var name = params.name || 'World'
  return { payload: 'Hello, ' + name + '!' }
}
  • Create a new action called hello from the local hello.js file.
1
2
$ wsk action create hello hello.js
ok: created action hello
  • List the actions registered on the platform.
1
2
3
$ wsk action list
actions
/guest/hello                                                           private nodejs:6
  • Invoke the action, blocking until it has finished.
1
2
3
4
$ wsk action invoke -r hello -p name James
{
  "payload": "Hello, James!"
}
  • Retrieve the activation record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ wsk activation list
activations
1d16d13fdbba4fdc96d13fdbba7fdc76 hello
$ wsk activation get 1d16d13fdbba4fdc96d13fdbba7fdc76
ok: got activation 1d16d13fdbba4fdc96d13fdbba7fdc76
{
  "namespace": "guest",
  "name": "hello",
  "version": "0.0.1",
  "subject": "guest",
  "activationId": "1d16d13fdbba4fdc96d13fdbba7fdc76",
  "start": 1516364811767,
  "end": 1516364811769,
  "duration": 2,
  "response": {
    "status": "success",
    "statusCode": 0,
    "success": true,
    "result": {
      "payload": "Hello, James!"
    }
  },
  ...
}

how does it work?

Apache OpenWhisk is an open-source serverless platform, composed of microservices written in Scala. Custom runtime code is bundled and managed as platform services using Docker. The platform also uses numerous external open-source projects, including CouchDB, Kafka, Zookeeper and Redis. Existing public images are used to pull those dependencies into the project.

Apache OpenWhisk automatically builds and publishes images for custom runtime services on Docker Hub. This means containers can be started from public images, rather than having to build them manually. Using the existing images dramatically reduces the start-up time for getting the project running locally.

“Devtools” uses Docker Compose to start a local instance of the platform. Docker Compose coordinates starting containers in the correct order, injects runtime configuration properties and link services using local networking interfaces.

tips & tricks

Once you have a local instance of the platform running, there are a few things you can do to make it easier to use during development…

switching CLI between platform instances

Using a local instance of the platform for testing and development, you will still want to deploy production applications to IBM Cloud Functions. Switching between these platform instances relies on updating the CLI configuration file with the correct authorisation credentials each time.

Rather than manually updating the default configuration file each time, an environment variable (WSK_CONFIG_FILE) can be used to choose a different configuration file. Keep credentials for different instances in separate files. Use the environment parameter to quickly switch platform instances.

1
$ WSK_CONFIG_FILE=~/.wskprops-local wsk

Using a shell alias to expose a new command to do this automatically makes this even easier.

1
alias wsk-local='WSK_CONFIG_FILE=~/.wskprops-local wsk -i'

The -i flag is used because the local platform instance uses a self-signed SSL certificate.

view platform details in database

CouchDB is the platform data store. This stores all installed actions, save activation records and other runtime properties. Accessing this database can be helpful to diagnose issues directly.

CouchDB comes with a administration web application called ”Futon”. This can read and modify database documents, run view queries and check configuration parameters.

Open this URL to access Futon for the local platform instance: http://localhost:5984/_utils/

Docker is configured to forward networking traffic from local port 5984 to the same port on the container.

read platform logs

System logs, generated from platform containers, are stored in the ~/tmp/openwhisk directory.

Logs from the invoker are stored in the invoker/logs/invoker-local_logs.log file. Logs from the controller are stored in the controller/logs/controller-local_logs.log file. All other container logs are stored in the same docker-compose.log file.

Searching these files with the activation id will allow you to find all platform logs for that action invocation.

find the runtime container for an activation

Running the platform locally means you can access runtime environments used to execute actions. This can help diagnosing and debugging application errors.

Finding the runtime container used for an invocation relies on having the activation identifier. Using this value, search the controller logs for the following log message.

1
[InvokerReactive] <namespace/action_id> <user> <activation_id>

This is then followed by a log message with the container identifier used for that invocation.

1
[DockerContainer] sending initialization to ContainerId(<CONTAINER_ID>)

Using docker exec you can then access the runtime environment to poke around!

1
$ docker exec -it <CONTAINER_ID> /bin/bash

install shared packages

On IBM Cloud Functions, the /whisk.system/ namespace contains shared packages for common utilities and external feed providers. These packages are not installed by default on the platform created by devtools.

These packages are available in the following repositories.

Follow the instructions in the repositories to make these available on your local version of the platform.

Pull requests have been opened to automate installing these packages in the devtools project.

conclusion

Serverless platforms often get criticised as having a poor “developer experience” for people used to traditional application servers. Having no access to the environments running your code can make development and testing challenging. Debugging issues through console logs feels like a step backwards.

Using an open-source serverless platform means you can actually run the entire platform locally. Using a local instance during development improves deployment times, provides access to runtime environments and allows you to work offline.

Apache OpenWhisk has numerous options for deploying the platform, including virtual machines, kubernetes or native containers. Whilst the platform is not the simplest to set-up manually, the devtools project bootstraps a pre-configured version by spinning up a local instance using Docker Compose. This is the easiest approach for most developers to have a local instance of the platform.

Visualising Serverless Metrics With Grafana Dashboards

This blog post is part three of a series on “Monitoring Serverless Applications Metrics”. See the introduction post for details and links to other posts.

Having configured collecting serverless metrics from IBM Cloud Functions (Apache OpenWhisk) applications, monitoring incoming metric values will alert us to problems.

IBM Cloud Monitoring provides a Grafana-based service to help with this.

Grafana is an open source metric analytics & visualization
suite. It is most commonly used for visualizing time series data for
infrastructure and application analytics.

Serverless metrics can be monitored in real-time using custom Grafana dashboards.

Let’s review a few Grafana basics before we start setting up the serverless monitoring dashboards…

Grafana Basics

Metric Data Values

Metrics data collected through the IBM Cloud Monitoring Service uses the following label format.

1
ibm.public.cloud-functions.$region.$namespace.$action.$activation.$labels

Templated variables ($varname) are replaced during collection by the monitoring library.

  • $region - Geographic region for IBM Cloud Functions instance.
  • $namespace - User namespace containing monitored actions.
  • $activation - Activation identifier associated with metric values.
  • $labels - One or more labels to identify metric data, e.g. time.duration

Metric values must be rational numbers. IBM Cloud Monitoring does not support other data types.

Templates

When defining metric queries, hardcoding values for region, namespace or action names does not scale when monitoring multiple serverless applications. Developers would need to replicate and maintain the same dashboards for every application.

Grafana uses template variables to resolve this problem.

Templates allow users to define a variable identifier with a user-defined value. Identifiers can be used in metric queries instead of hardcoded values. Changing template values automatically updates queries.

Common Tasks

How to create a new dashboard?

  • Open the dashboard menu by clicking the drop-down menu.
  • Click the “Create New” button.

How to set the dashboard name?

  • Select the “Manage Dashboard” menu option.
  • Click “Settings” to open the dashboard options panel.
  • Change the “General -> Details -> Name” configuration value.

How to set dashboard template variables?

  • Select the “Manage Dashboard” menu option.
  • Click “Templating” to open the templating variables configuration panel.
  • Click “New” button to define template variables.

  • Fill in the name field with the template identifier.
  • Select “IBM Cloud Monitoring” as the data source.
  • Fill in the query field with chosen metric query.

How to add new row to dashboard?

  • Click the “Add Row” button beneath the last row.

How to add new chart to row?

  • Hover over the options menu on the right-hand side of the row.
  • Select the “Add Panel” menu item.
  • Choose a chart type from the panel menu.

How to set and display row name?

  • Hover over the options menu on the right-hand side of the row.
  • Select the “Row Options” menu item.
  • Fill in the “Title” field. Click the “Show” checkbox.

How to edit chart parameters?

  • Click the panel title to open the panel options dialog.
  • Select the “Edit” button.
  • Graph options dialog opens below the chart panel.

How to choose time range for metric values?

  • Click the clock icon on the right-hand side of the menu bar.
  • Define time ranges manually or by selecting options from the “Quick Ranges” examples.
  • Auto-update can be enabled using the “Refresh” drop-down menu.

Dashboards

Having introduced some of the basics around using Grafana, we can now start to create dashboards.

tldr: want to set these dashboards up without following all the instructions?

Here are the completed JSON configuration files for the Grafana dashboards below. Remember to create the necessary template variables.

Overview Dashboard

This is an example of the first dashboard we want to create.

The dashboard provides information on actions invocations, errors, durations and other high-level metrics. It gives an overview of the performance of serverless applications within a region and workspace.

setup

  1. Create a new dashboard named “Overview”.
  2. Set the following template variables.
    • $region => ibm.public.cloud-functions.*
    • $namespace => ibm.public.cloud-functions.$region.*

Once the dashboard is created, we can add the first row showing action invocation counts.

invocations graph

This dashboard row will contain a single graph, using a bar chart of action invocation frequencies over time intervals. Stacking, rather than overlaying, chart values makes it easier to identify counts per action.

How can we calculate total invocations from the metric values?

One approach is to convert all metric values for a chosen label to a constant value of 1. This can be achieved using the scale() and offset() functions. Adding these constant values will return a count of the invocations recorded.

Let’s implement this now…

  • Set and display default row name as “Invocations”.
  • Add new “Graph” chart to row.
  • Configure metric query for chart:
1
2
ibm.public.cloud-functions.$region.$namespace.*.*.error
.scale(0).offset(1).groupByNode(5, sum)
  • Set the following options to true.
    • Legend->Options->Show
    • Display->Draw Modes->Bars
    • Display->Stacking & Null value->Stack

invocation types

This next dashboard row will show counts for different invocation types. Counts will be shown for total, successful, failed and cold start invocations.

Calculating the sum for all invocations recorded will use the same “scale & offset” trick explained above. Cold start and error totals can be calculated by simply summing the individual metric values. Successful invocation counts can be created by offsetting and scaling error values by -1 before summing.

all count

  • Add a new row.
  • Set and display default row name as “Invocation Types”.
  • Add a new “Single Stat” chart to row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.scale(0).offset(1).sumSeries()
  • Set the following options.
    • General -> Info -> Title = All
    • Options -> Value -> Stat = total
    • Options -> Coloring -> Background = true
    • Options -> Coloring -> Thresholds = 0,100000

success count

  • Duplicate the “All” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace…error.offset(-1).scale(-1).sumSeries()
  • Set the following options.
    • General -> Info -> Title = Success
    • Options -> Coloring -> Colors = Make green the last threshold colour.
    • Options -> Coloring -> Thresholds = 0,0

errors count

  • Duplicate the “Success” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Errors
    • Options-> Coloring -> Colors = Make red the last threshold colour.

cold start count

  • Duplicate the “Errors” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.coldstart.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Cold Start
    • Options-> Coloring -> Colors = Make blue the last threshold colour.

invocation durations

This row will contain counts for the total, mean and range of all invocations.

Duration is recorded as a metric value for each invocation. Grafana provides functions to calculate mean and range values from existing data series.

total duration

  • Add a new row.
  • Set and display default row name as “Invocation Durations”.
  • Add a new “Single Stat” chart to row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.time.duration.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Total
    • Options -> Value -> Stat = total
    • Options -> Value -> Unit = milliseconds
    • Options -> Coloring -> Background = true
    • Options -> Coloring -> Thresholds = 100000000,100000000
    • Options -> Coloring -> Colors = Make grey the first threshold colour.

average duration

  • Duplicate the “Total” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.time.duration.averageSeries()
  • Set the following options.
    • General -> Info -> Title = Average
    • Options -> Value -> Stat = avg

range duration

  • Duplicate the “Average” chart in the row.
  • Set the following options.
    • General -> Info -> Title = Range
    • Options -> Value -> Stat = range

invocation details table

Tables will show invocation details per action in this row. Invocation counts, errors recorded and duration statistics are shown in separate tables.

all invocations table

  • Add a new row.
  • Set and display row name as “Invocations Per Action”.
  • Add a “Table” panel to the row.
  • Configure metric query for chart:
1
2
ibm.public.cloud-functions.$region.$namespace.*.*.error
.scale(0).offset(1).groupByNode(5, sum)
  • Set the following options.
    • General -> Info -> Title = Invocations (All)
    • Options -> Data -> Table Transform = Time series aggregations
    • Options -> Data -> Columns = Total
    • Options -> Column Styles -> Decimals = 0

error invocations table

  • Duplicate the “Invocations (All)” chart in the row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.groupByNode(5, sum)
  • Set the following options.
    • General -> Info -> Title = Invocations (Errors)

duration statistics table

  • Duplicate the “Invocations (Errors)” chart in the row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.groupByNode(5, avg)
  • Set the following options.
    • General -> Info -> Title = Invocations (Duration)
    • Options -> Data -> Columns = Avg, Min, Max
    • Options -> Column Styles -> Decimals = Milliseconds
    • Options -> Column Styles -> Decimals = 2

Having finished all the charts for the overview dashboard, it should look like the example above.

Let’s move onto the second dashboard, which will give us more in-depth statistics for individual actions…

Action Dashboard

This is an example of the second dashboard we want to create.

The dashboard provides information on specific action application metrics. It includes more detailed statistics including duration percentiles, memory and cpu usage. This provides more context to help diagnosing issues for individual actions.

setup

  • Create a new dashboard named “Action Details”.
  • Set the following template variables.
    • $region => ibm.public.cloud-functions.*
    • $namespace => ibm.public.cloud-functions.$region.*
    • $actions => ibm.public.cloud-functions.$region.$namespace.<action>

Replace <action> with the name of an action you are monitoring.

invocations

Action invocations are shown this first dashboard row. Bar charts display successful versus failed invocations and cold versus warm starts.

Failed invocations and cold starts are recorded as metric values. Using the scale() and offset() functions allows us to calculate successful invocations and warm starts from these properties.

  • Set and display default row name as “Invocations”.
  • Add new “Graph” chart to row.
  • Configure two metric queries for the chart:
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.error
.scale(0).offset(1).groupByNode(5, sum).alias(success)
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.error
.groupByNode(5, sum).alias(failure)
  • Set the following options to true.
    • Legend->Options->Show
    • Display->Draw Modes->Bars

invocation types

This row replicates the “Invocation Types” row from the “Overview” dashboard.

Repeat the instructions from the above to create this row here.

Metric query settings must use the action template identifier rather than a wildcard value.

invocation durations

This row uses an extended version of the durations row from the “Overview” dashboard. In addition to total and average durations, minimum and maximum are also included.

Repeat the instructions from above to add the “Total” and “Average” panels.

Metric query settings must use the action template identifier rather than a wildcard value.

minimum duration

  • Duplicate the “Total” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration.minSeries()
  • Set the following options.
    • General -> Info -> Title = Min
    • Options -> Value -> Stat = min

maximum duration

  • Duplicate the “Minimum” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration.maxSeries()
  • Set the following options.
    • General -> Info -> Title = Min
    • Options -> Value -> Stat = max

percentiles graph

  • Add a “Table” panel to the row.
  • Configure this metric query for the chart:
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration
.percentileOfSeries(50, false).aliasByNode(5).alias($actions 50th percentile)
  • Duplicate this query three times, replacing 50 with 90, 95 and 99.
  • Set the following options.
    • General -> Info -> Title = Durations (Percentiles)
    • Axes -> Left Y -> Unit = Milliseconds
    • Legend -> Options -> Show = True
    • Legend -> Values -> Avg = True
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Null Value = connected

cpu usage

CPU usage for the Node.js process is recorded with two metric values, user and system time.

  • Add a new row.
  • Set and display row name as “CPU Usage”.
  • Add new “Graph” panel to row.
  • Configure two metric queries for the chart.
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.cpu.user
.groupByNode(5, avg).alias(user-time)
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.cpu.system
.groupByNode(5, avg).alias(system-time)
  • Set the following options.
    • Axes -> Left Y -> Unit = Microseconds
    • Legend -> Values -> Avg = true
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Stack = true
    • Display -> Stacking & Null value -> Null Value = connected

memory usage

Memory usage for the Node.js process is recorded with multiple values, including heap used & total, external and rss.

  • Add a new row.
  • Set and display row name as “Memory Usage”.
  • Add new “Graph” panel to row.
  • Configure four metric queries for the chart using this template.
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.*.memory.<label>
.groupByNode(5, avg).alias(<label>)

Replace <label> with following options: external, rss, heapUsed & heapTotal.

  • Set the following options.
    • Axes -> Left Y -> Unit = bytes
    • Legend -> Values -> Avg = true
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Stack = true
    • Display -> Stacking & Null value -> Null Value = connected

Having finished all the charts for the action details example, you should now have dashboards which look like the examples above! 📈📊📉

conclusion

Once you are collecting application metrics for IBM Cloud Functions (Apache OpenWhisk) applications, you need to be able to monitor metric values in real-time.

Grafana dashboards, hosted by the IBM Cloud Monitoring service, are a perfect solution for this problem. Building custom dashboards allows us to monitor incoming data values live.

In the next blog post, we’re going to finish off this series by looking at setting up automatic alerts based upon the metric values…

Capturing Runtime Metrics for OpenWhisk Applications

This blog post is part one of a series on “Monitoring Serverless Applications Metrics”. See the introduction post for details and links to other posts.

Serverless platforms pose a unique challenge for collecting application metrics. Runtime environments are ephemeral, existing only to process requests.

Using a background process to capture and export metrics to an external service is impossible. With such a restricted runtime environment, we have to look at other options… 🤔

exporting serverless runtime metrics

console logs with alarm trigger

Apache OpenWhisk captures console logs written to stdout or stderr by actions. Logs are available in activation records through the platform API.

Runtime metrics written to the console will be saved in the activation records.

An additional action, triggered from the alarm package, can be scheduled to collect these values from the logs and forward to the metrics service.

This approach is simple and does not interfere with request processing. However, it does add a delay to the metric values being available in the collection service. This delay is dependent on the schedule chosen for the alarm trigger feed.

send before returning

Another option is to instrument the serverless functions to automatically collect and push metrics to the metric service before returning from the function.

The function will have to wait for the external metrics service to respond before returning.

This method pushes metric values into the collection service in real-time. There is no waiting for the background collection action to run. The disadvantage of this approach is that it adds a delay to each request. This delay is dependent on the response time from the collection service.

capturing runtime metrics

Metric values will have to be captured using a runtime library, due to the restrictions on running background processes.

The library should automatically capture registered metrics during each invocation. Values will be forwarded to the collection service using the configured forwarding method.

openwhisk-metrics

There is a Node.js library to ease the process of capturing runtime metrics for OpenWhisk actions.

https://github.com/jthomas/openwhisk-metrics

Node.js actions are wrapped with a proxy to automate recording metrics during invocations.

Metric values for cpu, memory, time, error and coldstart are collected by default. It supports adding custom metric collectors.

usage

Wrap action handlers with the metrics library.

1
2
3
4
5
6
7
const metrics = require('openwhisk-metrics')

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

Metrics values are logged to stdout for each invocation of the serverless function.

1
2
3
4
5
6
7
8
9
10
11
METRIC <workspace>.<action_name>.<activation>.memory.rss 53018624 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.heapTotal 34463744 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.heapUsed 16955224 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.external 987361 1512489781
METRIC <workspace>.<action_name>.<activation>.error 0 1512489781
METRIC <workspace>.<action_name>.<activation>.coldstart 0 1512489781
METRIC <workspace>.<action_name>.<activation>.cpu.user 177 1512489781
METRIC <workspace>.<action_name>.<activation>.cpu.system 2 1512489781
METRIC <workspace>.<action_name>.<activation>.time.start 1511605588388 1512489781
METRIC <workspace>.<action_name>.<activation>.time.end 1511605588468 1512489781
METRIC <workspace>.<action_name>.<activation>.time.duration 80 1512489781

Once you are collecting metrics, you need a monitoring service to forward them to…

monitoring service

We’re going to look at collecting and visualising metrics using the IBM Cloud Monitoring Service.

Use the IBM® Cloud Monitoring service to expand your collection and retention capabilities when working with metrics, and to be able to define rules and alerts that notify you of conditions that require attention.

IBM Cloud Monitoring service comes with a metric collection service, Grafana-based visualisation dashboard and an alerting system.

Let’s demonstrate how to use the approaches listed above for exporting metrics to the IBM Cloud Monitoring service.

There’s an additional Node.js library to integrate the OpenWhisk Metrics library with the IBM Cloud Monitoring Service. This can forward metrics in real-time or batches (using a schedule action from a timer).

provisioning

An instance of the service must be provisioned before being used.

The service is available in multiple regions. Choose the same region as the IBM Cloud Functions instance for best performance.

Instances can be provisioned through the IBM Cloud dashboard or the IBM Cloud CLI tool.

1
 $ bx cf create-service Monitoring lite my_monitoring_svc

For more details on provisioning instances of this service, please see the documentation: https://console.bluemix.net/docs/services/cloud-monitoring/how-to/provision.html#provision

authentication

IBM Cloud Monitoring supports the following authentication methods: API Key, UAA Token or IAM Token.

An API key will be used to provide authentication credentials in the examples below.

Keys can be created through the IBM Cloud dashboard or the IBM Cloud CLI tool.

1
$ bx iam api-key-create metrics-key -d "API Key For Serverless Metrics"

Note: The API key is only shown at the time of creation. If the API key is lost, you must create a new API key.

For more details on creating API keys, please see the documentation here: https://console.bluemix.net/docs/services/cloud-monitoring/security/auth_api_key.html#auth_api_key

space domain

The GUID of the account space is also required to use the metrics service.

Retrieve this value using the IBM Cloud CLI tool. Replace SpaceName with the name of the space.

1
bx iam space SpaceName --guid

The GUID for the space is returned.

1
2
$ bx iam space dev --guid
667fadfc-jhtg-1234-9f0e-cf4123451095

Note: Space GUIDs must be prefixed with s- when being using with the monitoring service.

1
"667fadfc-jhtg-1234-9f0e-cf4123451095" => "s-667fadfc-jhtg-1234-9f0e-cf4123451095"

example (real-time forwarding)

Let’s start with an example of using real-time forwarding of metrics values. Metric values will be automatically collected by the runtime library. Before each invocation finishes, the library will send the values to the external collection service.

This example assumes you already have the CLI tool for IBM Cloud Functions installed and configured. If you need to do this, please follow the instructions here.

create new directory

1
2
$ mkdir hello-world
$ cd hello-world

initialise npm package

1
$ npm init -y

install libraries

1
$ npm install openwhisk-metrics cloud-functions-metrics-service

update action handler source

Create a file called index.js with following code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const metrics = require('openwhisk-metrics')
const service = require('cloud-functions-metrics-service')

metrics.service = service.client({
  host: 'metrics.<???>.bluemix.net',
  scope: 's-<???>',
  api_key: '<???>'
})

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

Replace the host, scope and api_key parameters custom values. Host names for the regional monitoring service endpoints are listed here.

Space GUIDs must be prefixed with s- to identify a space in the scope parameter.

create new action

1
2
$ zip -r action.zip index.js package.json node_modules
$ wsk action create hello-world --kind nodejs:8 action.zip

invoke action

1
2
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 74add543b3b94bbbadd543b3b9dbbb17

use api to list metrics

Use curl to manually list the ingested metric labels for IBM Cloud Functions.

Replace the <???> values in the command with configuration values from above.

1
2
3
4
5
6
7
8
9
$ curl --url 'https://metrics.<???>.bluemix.net/v1/metrics/list?query=ibm.public.cloud-functions.*.*.*.*' --header 'x-auth-scope-id: s-<???>' --header 'X-Auth-User-Token: apikey <???>'
[{
  "leaf": 0,
  "context": {},
  "text": "72df4bc809c04fae9f4bc809c01fae77",
  "expandable": 1,
  "id": "ibm.public.cloud-functions.*.*.*.72df4bc809c04fae9f4bc809c01fae77",
  "allowChildren": 1
}]

Each activation identifier should be listed as a label value in the metrics service.

example (batch forwarding)

If we don’t want to add a (short) delay to each invocation, metric values can be forwarded asynchronously. An explicit action will be created to forward metric values from invocation logs. This action will be triggered on a periodic schedule using the alarm trigger feed.

This example assumes you already have the CLI tool for IBM Cloud Functions installed and configured. If you need to do this, please follow the instructions here.

create sample action

1
2
3
4
$ mkdir hello-world
$ cd hello-world
$ npm init -y
$ npm install openwhisk-metrics

Create a file called index.js with the following code.

1
2
3
4
5
6
7
const metrics = require('openwhisk-metrics')

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

deploy and test sample action

Package and deploy hello-world action.

1
2
$ zip -r action.zip index.js package.json node_modules
$ wsk action create hello-world --kind nodejs:8 action.zip

Metric values are written to the console for each invocation.

1
2
3
4
5
6
7
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 28da39d219df436a9a39d219df036a30
$ wsk activation logs 28da39d219df436a9a39d219df036a30
2017-12-18T14:38:50.751615113Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.cpu.user 0 1513607930
2017-12-18T14:38:50.751672372Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.cpu.system 0 1513607930
2017-12-18T14:38:50.751685034Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.time.start 1513607930749 1513607930
...

create metric-forwarder action

Clone the project repository and install NPM dependencies.

1
2
3
$ git clone https://github.com/jthomas/cloud-functions-metrics-service
$ cd cloud-functions-metrics-service
$ npm install

Update action configuration file (config.json) with the following parameter values.

1
2
3
4
5
6
7
8
{
  "actions": ["hello-world"],
  "service": {
    "host": "metrics.<???>.bluemix.net",
    "scope": "s-<???>",
    "api_key": "<???>"
  }
}

Replace the <???> fields in the configuration file from values from above.

Package metric-forwarder action.

1
$ zip -r action.zip index.js package.json lib node_modules

deploy metric-forwarder action

Create new metric-forwarder action from deployment package and configuration file.

1
$ wsk action create metric-forwarder --kind nodejs:8 action.zip --param-file config.json

Create trigger feed for alarm package to run metric-forwarder on periodic schedule.

1
2
3
$ wsk trigger create interval \
  --feed /whisk.system/alarms/interval \
  --param minutes 1

Bind trigger to action using rule.

1
$ wsk rule create forward-metrics-on-interval interval metric-forwarder

invoke sample action

1
2
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 28da39d219df436a9a39d219df036a30

This will generate activation records containing metric values. When the interval trigger is next fired, metric values from these records forwarded to the collection service.

Logs from the metric-forwarder action will show the activation records that have been retrieved.

1
$ wsk activation poll metric-forwarder

use api to list metrics

Use curl to manually list the ingested metric labels for IBM Cloud Functions.

Replace the <???> values in the command with configuration values from above.

1
2
3
4
5
6
7
8
9
$ curl --url 'https://metrics.<???>.bluemix.net/v1/metrics/list?query=ibm.public.cloud-functions.*.*.*.*' --header 'x-auth-scope-id: s-<???>' --header 'x-auth-user-token: apikey <???>'
[{
  "leaf": 0,
  "context": {},
  "text": "72df4bc809c04fae9f4bc809c01fae77",
  "expandable": 1,
  "id": "ibm.public.cloud-functions.*.*.*.28da39d219df436a9a39d219df036a30",
  "allowChildren": 1
}]

Each activation identifier should be listed as a label value in the metrics service.

next steps

Metric values from our serverless applications are now being collected by the IBM Cloud Monitoring service. 👏👏👏

Applications metrics are automatically recorded by the runtime library for each invocation. Metric values are exported to the monitoring service in real-time or using a timed action to forward in batches.

Setting up monitoring dashboards from the collected values will allow us to identify and resolve issues with our serverless applications. In the next blog post, we’ll look using Grafana to visualise metric values being collected…