James Thomas

Notes on software.

Lessons From West Berkshire Action for Refugees

For the past two years, I’ve been involved with a local charity (West Berkshire Action For Refugees) set up in response to the refugee crisis.

Started with the aim to send a single collection of aid to refugees in Calais, the group ended up sending thousands of boxes of aid to refugees all over Europe. Campaigning for West Berkshire Council to participate in the UK’s resettlement scheme for Syrian refugees also led to multiple refugee families being resettled locally. The group now runs a volunteer-led integration programme to assist them upon arrival.

WBAR became a second-job (albeit one with no remuneration, employee benefits or time off ๐Ÿ˜‰) for me and many other members of the group. Running a charitable organisation solely with volunteers, working around full-time jobs, families and other commitments, has innate challenges. We never had enough time or resources to implement all the ideas we came up with!

Before Christmas, I stepped down as a trustee and from all other official roles within the group. Having been involved for over two years, from the first donation drive to becoming a registered charity, I was ready for a break.

Since stepping down, I’ve been thinking about the lessons we learnt about running a charitable organisation, staffed solely by volunteers, with minimal resources.

If I had to do it all again, here’s what I wished I’d known from the beginningโ€ฆ

managing volunteers

what did we learn?

Volunteers are the lifeblood of small charitable organisations. Organising a systematic process for recruiting volunteers is crucial to building a sustainable charitable organisation. Growing the size and scope of an organisation, without growing the volunteer base, is a strategy for burn out.

background

WBAR started during a period of intense media coverage on the ”refugee crisis”. People understood how dire the situation was and were desperate to help. The group was inundated with offers of assistance. The biggest challenge was simply responding to all these messages.

We did not need to recruit volunteers, they came to us.

People asking to help would be invited to meet the core team at a committee meeting. If they turned up, someone would hopefully try to find an opportunity for them within the group.

This process was not ideal for the following reasonsโ€ฆ

  • Volunteer enquiries would often get missed by a core team busy with other activities.
  • Lacking a process for registering enquiries, it was difficult to coordinate and track the status of those people we did follow up with.
  • Committee meetings were also not the best place to “on-board” new volunteers.

Initially, with a constant stream of new volunteers coming forward, this was not an issue.

Skip forward twelve monthsโ€ฆ

When the media focus on the refugee crisis (predictably) disappeared, so did the number of people contacting the group to help. When the number of new volunteers shrank, the group’s activities did notโ€ฆ

As we took on more responsibilities, the more acute the need for new volunteers became, but the less time we had to focus on recruitment.

Eventually, due to existing volunteers stepping down or not being able to take on those extra roles, it became critical to actively recruit new volunteers, rather than passively waiting for them to come to us.

This was the point at which having an existing volunteer recruitment system would have been a huge benefit.

If we had been formally registering volunteer enquiries, including interviewing people to record their skills and availability, filling new roles would have been a quick and efficient process.

Without this database of potential volunteers, we were reliant on posting messages on social media asking for new volunteers. This caused a significant delay in staffing new roles whilst we waited for people to see and respond to the messages. Finding new volunteers felt slow and inefficient.

This issue led the group to appoint a formal volunteer coordinator. The coordinator is responsible for running a continual recruitment process and managing all volunteer enquiries. This ensures there is a recurring pipeline of potential recruits.

What should we have done differently?

Focused on volunteer recruitment before it became an acute issue.

Set up a systematic process for handling volunteer enquiries. Record all details of people contacting the group to build a pipeline of new recruits. Work on an outbound recruitment programme, with the group actively seeking volunteers for identified roles. Don’t be reliant on volunteers finding us.

using facebook

what did we learn?

Facebook is the new Excel. It has become the default platform for all online activities. Charitable causes are no different. Private groups and messenger enable distributed collaboration between remote volunteers.

Facebook as a collaboration tool struggles once groups reach a certain size. Moving to more appropriate tools, rather than continuing to work around the challenges, will be a necessity longer-term.

background

During the summer of 2015, when the refugee crisis became a front-page story, people all over the United Kingdom started collecting donations for Calais and other refugee camps across Europe.

Facebook became the platform for connecting volunteers throughout the country.

There were hundreds of public (and private) Facebook groups relating to the crisis. From pages for groups collecting aid in different towns across the country or finding organisations working in refugee camps needing aid to those offering lifts and accommodation for people volunteering in refugee camps.

There was a Facebook group for everything.

WBAR started when the founder created one of these pages in August 2015. The following week, I stumbled across the page whilst looking for a local group to offer assistance too. Sending a private message to the founder, I asked if there was anything I could do to help. Little did I know that the group would become a significant part of my life for the next two yearsโ€ฆ

This page became our main communication tool and grew to having over one thousand followers. Whether asking for donations, finding volunteers or highlighting our activities, Facebook made it simple to reach huge numbers of people within the local community with minimal effort or expense.

Facebook also became the default solution for coordinating group activities and volunteers.

There was private Facebook group for the core volunteers, committee members and trustees. Volunteers would post in the group with updates on activities, requests for assistance or other issues. Threads on posts were used to communicate between volunteers. Tagging members in posts would highlight where certain volunteers were needed to assist. Groups also allowed sharing files and other documents between members.

Facebook has a number of advantages over other (more appropriate) tools for digital collaboration.

  • Everyone is on Facebook. No-one has to sign up for an account with a new platform.
  • People understand how to use the site. No training is needed to on-board new members.
  • Facebook has web and mobile clients for all platforms. People can access the site using devices they prefer.
  • People already spend an average of fifty minutes a day on Facebook(!).

Making volunteers sign up for a new platform, learn how to use it and then remember to check daily for messages, would dramatically decrease engagement and collaboration.

But the limits of Facebook as a collaboration platform for the volunteers began to show as the organisation grew in size and scope.

These included, but were not limited to, the following issuesโ€ฆ

  • Group posts with multiple levels of threaded comments are difficult to follow. It’s not obvious which comments are new without re-reading every comment.
  • Finding historical posts or specific comments often felt impossible. Facebook search did not support complex filtering operations. Manually scrolling through all items in the group was often the only way to find specific items.
  • Group notifications were often lost in the morass of other alerts people received. Facebook would not let you subscribe to notifications from specific groups or posts. Volunteers had to manually remind each other of notifications using other tools.

Spending my work time collaborating with distributed teams in open-source, I often wished we were using a Github project with issues, milestones and markdown support!

There is a plethora of more suitable collaboration tools on the Internet. However, new tools come with a “cognitive burden” on volunteers. Registration, training, device support and others issues need to be balanced against the benefits from using more appropriate platforms.

What should we have done differently?

Investigated additional tools to support distributed work flows between remote volunteers. Once the limitations of Facebook became apparent, rather than working around them, we should have worked out a better solution.

charitable incorporated organisations

what did we learn?

Becoming an official charitable organisation is inevitable as you grow in size and scope.

Registering with the charity commission is not a simple or fast process. It’s impossible to know how long your application will take to be confirmed. If registration is prerequisite for other activities, this can leave you in stuck for an indeterminable amount of time.

background

After twelve months, it became clear the group needed to register as an official charitable organisation. Registration opened up opportunities in applying to trusts for grants, became a requirement for projects we wanted to start and insurance purposes.

West Berks Action For Refugees chose to incorporate as a Charitable Incorporated Organisation (CIO).

Launched in 2012, CIOs were a new form of charitable organisation, with lighter regulation and reporting requirements. CIOs are administered by the Charity Commission, who have sole responsibility for their formation and registration. This reduces the administrative burden by not having to additionally register and report to Companies House, like Charitable Companies.

Registering a CIO with the Charity Commission was supposed to be an easy and efficient process. Unfortunately, cuts in Government funding has led to severe resource issues at the Charity Commission. Recent news indicated registration times were currently around three months.

It took West Berks Action For Refugees nearly six months to register as a CIO. This caused enormous problems for the group.

Opening an official bank account with The Co-Operative Bank required us to have registration confirmed. Until we had a bank account, it was difficult to receive official donations to the group. Other organisations often used cheques for donations, with the group’s name as the recipient. These were unable to be received without an official bank account.

Once the charity registration did come through, The Co-Operative Bank still took another six months to open the account!

Group activities also began to need insurance policies. For example, public liability insurance was a requirement for even the smallest public event, like a cake sale in the church hall. Insurers do provide specialist charity insurance but only for registered organisations. These policies were unavailable to us until the registration came through.

CIOs were set up to make registering a charitable organisation a quick and efficient process. Unfortunately, due to Government funding cuts, this is no longer the case. Whilst waiting for our registration to come through, the group had numerous challenges that we were unable to do anything about.

what should we have done differently?

Looked for a community bank account, that didn’t require being a registered charitable organisation. This would have resolved issues we faced processing donations.

Chosen a different charity bank account provider. The Co-Operative Bank were incredibly slow to process the account opening and have an awful online banking site for business accounts. I’ve heard similar complaints from other groups. Would not recommend!

governance & decision-making

what did we learn?

Organisations need to have an appropriate level of governance for their size and scope. Formal governance structures are a requirement for registered charitable organisations. Trustees need to have oversight on activities and keep a formal record of decisions.

Moving from an informal to a formal decision making process can lead to resistance from volunteers. It might appear that this added “bureaucracy” unnecessary slows down decision making.

background

The charity started as a small group of volunteers working on a single activity, collecting donations for refugees in Calais. People volunteered when they had time. Communication and coordination between volunteers happened in an ad-hoc sense.

An informal decision making process was a natural governance model for a group of that size and scope. When the group changed, in activities and responsibilities, the governance model needed to reflect that.

This started as a committee meeting every six weeks. Volunteers would attend to bring issues for the wider group to resolve. With people still working independently, this was often the only time people would regularly see each other.

This meeting was crucial to keeping the group running smoothly. Over time, we expanded the meeting to use a more formal process, with an explicit agenda and reports from the sub-committees. Minutes were noted to keep an official record of the meeting and provide an overview to those unable to attend.

There was often a tension between the formal decision-making process and the volunteers. People often wanted a decision on an issue immediately, rather than waiting for the next meeting. There was a pressure to make important decisions outside of the committee meetings. People were used to the informal decision making process we had started with.

Volunteers sometimes failed to engage with the new governance structure. People not attending meetings or sending reports into the group was a regular issue. Decisions would become repeatedly postponed, due to missing reports or non-attendance of members involved. This undermined the effectiveness of the governance structure, leading to further resistance.

Setting up a formal decision making process and governance structure for the charity was a legal requirement of incorporating as a CIO. The group needed a transparent decision making process, along with a formal record of decisions. However, moving away from an informal and ad-hoc decision making process did seem, to some people, like unnecessary bureaucracy and a burden on an already stretched group of volunteers.

What should we have done differently?

Moved earlier to use a more formal governance model. Officially documented the governance structure and decision making process. Explained to all volunteers how decisions need to be made within the group and the rationale for this approach.

Starting OpenWhisk in Sixty Seconds

Apache OpenWhisk is an open-source serverless platform. Developers can use hosted instances from IBM, deploy to any infrastructure provider and even run it locally.

Developers often use a local instance of the platform during development. Deploying to a local instance is faster than the cloud. It also provides access runtime environments to debug issues and allows development without an Internet connection. Production applications are still run on IBM Cloud Functions.

But OpenWhisk provides numerous options for starting the platform, including running the platform services directly, using container management tools like Kubernetes and Mesos or starting a pre-configured virtual machine with Vagrant.

So, what’s easiest?

OpenWhisk Devtools.

Using this project, the platform can be started on any machine with Docker Compose in around sixty seconds. Before we explain how this works, let’s show the steps needed to spin up the platform using the project.

openwhisk in around sixty secondsโ€ฆ

Do you have Docker with Compose support installed? If not, follow the instructions here.

Start the platform with the following commands.

1
2
3
$ git clone git@github.com:apache/incubator-openwhisk-devtools.git
$ cd incubator-openwhisk-devtools/docker-compose
$ make quick-start

Having cloned the repository, creating the local instance only takes around sixty seconds! ๐Ÿ’ฏ

1
2
3
4
5
$ time make quick-start &>/dev/null

real    1m10.128s
user    0m1.709s
sys     0m1.258s

Platform services will be running as containers on the host after initialisation.

1
2
3
4
5
6
7
8
9
10
11
12
$ docker ps --format "{{.ID}}: {{.Names}} {{.Image}}" 
17c5d31e2c20: wsk0_60_prewarm_nodejs6         (openwhisk/nodejs6action:latest)
0eace484289c: wsk0_59_prewarm_nodejs6         (openwhisk/nodejs6action:latest)
1be725d8767c: openwhisk_apigateway_1          (adobeapiplatform/apigateway:1.1.0)
641cbabeb790: openwhisk_kafka-topics-ui_1     (landoop/kafka-topics-ui:0.9.3)
f52c25dbadd9: openwhisk_controller_1          (openwhisk/controller)
8f0c6aa14ccc: openwhisk_invoker_1             (openwhisk/invoker)
d5274194f842: openwhisk_kafka-rest_1          (confluentinc/cp-kafka-rest:3.3.1)
40a1585f64bb: openwhisk_kafka_1               (wurstmeister/kafka:0.11.0.1)
b0b0f75c6fdb: openwhisk_db_1                  (couchdb:1.6)
a7449c2edc4d: openwhisk_zookeeper_1           (zookeeper:3.4)
178abe09b793: openwhisk_redis_1               (redis:2.8)

โ€ฆand that’s it!

testing it out

setting up CLI tool

OpenWhisk provides a CLI tool for interacting with the platform. The quick-start command automatically writes account credentials for the local instance into the CLI configuration file. Using the CLI tool to print current configuration values shows the platform endpoint set as the local machine ip or hostname.

If you don’t have the CLI tool already installed, the project downloads the binary to the following location: devtools/docker-compose/openwhisk-master/bin/wsk

1
2
$ wsk property get | grep host
whisk API host        localhost

The local instance is configured with a single user account (guest) with these credentials. Administrative credentials are stored in this configuration file.

creating sample actions

With the CLI configured correctly, you can create and invoke an action on the local platform instance.

  • Create a new file called hello.js with the following contents.
1
2
3
4
function main (params) {
  var name = params.name || 'World'
  return { payload: 'Hello, ' + name + '!' }
}
  • Create a new action called hello from the local hello.js file.
1
2
$ wsk action create hello hello.js
ok: created action hello
  • List the actions registered on the platform.
1
2
3
$ wsk action list
actions
/guest/hello                                                           private nodejs:6
  • Invoke the action, blocking until it has finished.
1
2
3
4
$ wsk action invoke -r hello -p name James
{
  "payload": "Hello, James!"
}
  • Retrieve the activation record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ wsk activation list
activations
1d16d13fdbba4fdc96d13fdbba7fdc76 hello
$ wsk activation get 1d16d13fdbba4fdc96d13fdbba7fdc76
ok: got activation 1d16d13fdbba4fdc96d13fdbba7fdc76
{
  "namespace": "guest",
  "name": "hello",
  "version": "0.0.1",
  "subject": "guest",
  "activationId": "1d16d13fdbba4fdc96d13fdbba7fdc76",
  "start": 1516364811767,
  "end": 1516364811769,
  "duration": 2,
  "response": {
    "status": "success",
    "statusCode": 0,
    "success": true,
    "result": {
      "payload": "Hello, James!"
    }
  },
  ...
}

how does it work?

Apache OpenWhisk is an open-source serverless platform, composed of microservices written in Scala. Custom runtime code is bundled and managed as platform services using Docker. The platform also uses numerous external open-source projects, including CouchDB, Kafka, Zookeeper and Redis. Existing public images are used to pull those dependencies into the project.

Apache OpenWhisk automatically builds and publishes images for custom runtime services on Docker Hub. This means containers can be started from public images, rather than having to build them manually. Using the existing images dramatically reduces the start-up time for getting the project running locally.

“Devtools” uses Docker Compose to start a local instance of the platform. Docker Compose coordinates starting containers in the correct order, injects runtime configuration properties and link services using local networking interfaces.

tips & tricks

Once you have a local instance of the platform running, there are a few things you can do to make it easier to use during developmentโ€ฆ

switching CLI between platform instances

Using a local instance of the platform for testing and development, you will still want to deploy production applications to IBM Cloud Functions. Switching between these platform instances relies on updating the CLI configuration file with the correct authorisation credentials each time.

Rather than manually updating the default configuration file each time, an environment variable (WSK_CONFIG_FILE) can be used to choose a different configuration file. Keep credentials for different instances in separate files. Use the environment parameter to quickly switch platform instances.

1
$ WSK_CONFIG_FILE=~/.wskprops-local wsk

Using a shell alias to expose a new command to do this automatically makes this even easier.

1
alias wsk-local='WSK_CONFIG_FILE=~/.wskprops-local wsk -i'

The -i flag is used because the local platform instance uses a self-signed SSL certificate.

view platform details in database

CouchDB is the platform data store. This stores all installed actions, save activation records and other runtime properties. Accessing this database can be helpful to diagnose issues directly.

CouchDB comes with a administration web application called ”Futon”. This can read and modify database documents, run view queries and check configuration parameters.

Open this URL to access Futon for the local platform instance: http://localhost:5984/_utils/

Docker is configured to forward networking traffic from local port 5984 to the same port on the container.

read platform logs

System logs, generated from platform containers, are stored in the ~/tmp/openwhisk directory.

Logs from the invoker are stored in the invoker/logs/invoker-local_logs.log file. Logs from the controller are stored in the controller/logs/controller-local_logs.log file. All other container logs are stored in the same docker-compose.log file.

Searching these files with the activation id will allow you to find all platform logs for that action invocation.

find the runtime container for an activation

Running the platform locally means you can access runtime environments used to execute actions. This can help diagnosing and debugging application errors.

Finding the runtime container used for an invocation relies on having the activation identifier. Using this value, search the controller logs for the following log message.

1
[InvokerReactive] <namespace/action_id> <user> <activation_id>

This is then followed by a log message with the container identifier used for that invocation.

1
[DockerContainer] sending initialization to ContainerId(<CONTAINER_ID>)

Using docker exec you can then access the runtime environment to poke around!

1
$ docker exec -it <CONTAINER_ID> /bin/bash

install shared packages

On IBM Cloud Functions, the /whisk.system/ namespace contains shared packages for common utilities and external feed providers. These packages are not installed by default on the platform created by devtools.

These packages are available in the following repositories.

Follow the instructions in the repositories to make these available on your local version of the platform.

Pull requests have been opened to automate installing these packages in the devtools project.

conclusion

Serverless platforms often get criticised as having a poor “developer experience” for people used to traditional application servers. Having no access to the environments running your code can make development and testing challenging. Debugging issues through console logs feels like a step backwards.

Using an open-source serverless platform means you can actually run the entire platform locally. Using a local instance during development improves deployment times, provides access to runtime environments and allows you to work offline.

Apache OpenWhisk has numerous options for deploying the platform, including virtual machines, kubernetes or native containers. Whilst the platform is not the simplest to set-up manually, the devtools project bootstraps a pre-configured version by spinning up a local instance using Docker Compose. This is the easiest approach for most developers to have a local instance of the platform.

Visualising Serverless Metrics With Grafana Dashboards

This blog post is part three of a series on “Monitoring Serverless Applications Metrics”. See the introduction post for details and links to other posts.

Having configured collecting serverless metrics from IBM Cloud Functions (Apache OpenWhisk) applications, monitoring incoming metric values will alert us to problems.

IBM Cloud Monitoring provides a Grafana-based service to help with this.

Grafana is an open source metric analytics & visualization
suite. It is most commonly used for visualizing time series data for
infrastructure and application analytics.

Serverless metrics can be monitored in real-time using custom Grafana dashboards.

Let’s review a few Grafana basics before we start setting up the serverless monitoring dashboardsโ€ฆ

Grafana Basics

Metric Data Values

Metrics data collected through the IBM Cloud Monitoring Service uses the following label format.

1
ibm.public.cloud-functions.$region.$namespace.$action.$activation.$labels

Templated variables ($varname) are replaced during collection by the monitoring library.

  • $region - Geographic region for IBM Cloud Functions instance.
  • $namespace - User namespace containing monitored actions.
  • $activation - Activation identifier associated with metric values.
  • $labels - One or more labels to identify metric data, e.g. time.duration

Metric values must be rational numbers. IBM Cloud Monitoring does not support other data types.

Templates

When defining metric queries, hardcoding values for region, namespace or action names does not scale when monitoring multiple serverless applications. Developers would need to replicate and maintain the same dashboards for every application.

Grafana uses template variables to resolve this problem.

Templates allow users to define a variable identifier with a user-defined value. Identifiers can be used in metric queries instead of hardcoded values. Changing template values automatically updates queries.

Common Tasks

How to create a new dashboard?

  • Open the dashboard menu by clicking the drop-down menu.
  • Click the “Create New” button.

How to set the dashboard name?

  • Select the “Manage Dashboard” menu option.
  • Click “Settings” to open the dashboard options panel.
  • Change the “General -> Details -> Name” configuration value.

How to set dashboard template variables?

  • Select the “Manage Dashboard” menu option.
  • Click “Templating” to open the templating variables configuration panel.
  • Click “New” button to define template variables.

  • Fill in the name field with the template identifier.
  • Select “IBM Cloud Monitoring” as the data source.
  • Fill in the query field with chosen metric query.

How to add new row to dashboard?

  • Click the “Add Row” button beneath the last row.

How to add new chart to row?

  • Hover over the options menu on the right-hand side of the row.
  • Select the “Add Panel” menu item.
  • Choose a chart type from the panel menu.

How to set and display row name?

  • Hover over the options menu on the right-hand side of the row.
  • Select the “Row Options” menu item.
  • Fill in the “Title” field. Click the “Show” checkbox.

How to edit chart parameters?

  • Click the panel title to open the panel options dialog.
  • Select the “Edit” button.
  • Graph options dialog opens below the chart panel.

How to choose time range for metric values?

  • Click the clock icon on the right-hand side of the menu bar.
  • Define time ranges manually or by selecting options from the “Quick Ranges” examples.
  • Auto-update can be enabled using the “Refresh” drop-down menu.

Dashboards

Having introduced some of the basics around using Grafana, we can now start to create dashboards.

tldr: want to set these dashboards up without following all the instructions?

Here are the completed JSON configuration files for the Grafana dashboards below. Remember to create the necessary template variables.

Overview Dashboard

This is an example of the first dashboard we want to create.

The dashboard provides information on actions invocations, errors, durations and other high-level metrics. It gives an overview of the performance of serverless applications within a region and workspace.

setup

  1. Create a new dashboard named “Overview”.
  2. Set the following template variables.
    • $region => ibm.public.cloud-functions.*
    • $namespace => ibm.public.cloud-functions.$region.*

Once the dashboard is created, we can add the first row showing action invocation counts.

invocations graph

This dashboard row will contain a single graph, using a bar chart of action invocation frequencies over time intervals. Stacking, rather than overlaying, chart values makes it easier to identify counts per action.

How can we calculate total invocations from the metric values?

One approach is to convert all metric values for a chosen label to a constant value of 1. This can be achieved using the scale() and offset() functions. Adding these constant values will return a count of the invocations recorded.

Let’s implement this nowโ€ฆ

  • Set and display default row name as “Invocations”.
  • Add new “Graph” chart to row.
  • Configure metric query for chart:
1
2
ibm.public.cloud-functions.$region.$namespace.*.*.error
.scale(0).offset(1).groupByNode(5, sum)
  • Set the following options to true.
    • Legend->Options->Show
    • Display->Draw Modes->Bars
    • Display->Stacking & Null value->Stack

invocation types

This next dashboard row will show counts for different invocation types. Counts will be shown for total, successful, failed and cold start invocations.

Calculating the sum for all invocations recorded will use the same “scale & offset” trick explained above. Cold start and error totals can be calculated by simply summing the individual metric values. Successful invocation counts can be created by offsetting and scaling error values by -1 before summing.

all count

  • Add a new row.
  • Set and display default row name as “Invocation Types”.
  • Add a new “Single Stat” chart to row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.scale(0).offset(1).sumSeries()
  • Set the following options.
    • General -> Info -> Title = All
    • Options -> Value -> Stat = total
    • Options -> Coloring -> Background = true
    • Options -> Coloring -> Thresholds = 0,100000

success count

  • Duplicate the “All” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespaceโ€ฆerror.offset(-1).scale(-1).sumSeries()
  • Set the following options.
    • General -> Info -> Title = Success
    • Options -> Coloring -> Colors = Make green the last threshold colour.
    • Options -> Coloring -> Thresholds = 0,0

errors count

  • Duplicate the “Success” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Errors
    • Options-> Coloring -> Colors = Make red the last threshold colour.

cold start count

  • Duplicate the “Errors” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.coldstart.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Cold Start
    • Options-> Coloring -> Colors = Make blue the last threshold colour.

invocation durations

This row will contain counts for the total, mean and range of all invocations.

Duration is recorded as a metric value for each invocation. Grafana provides functions to calculate mean and range values from existing data series.

total duration

  • Add a new row.
  • Set and display default row name as “Invocation Durations”.
  • Add a new “Single Stat” chart to row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.time.duration.sumSeries()
  • Set the following options.
    • General -> Info -> Title = Total
    • Options -> Value -> Stat = total
    • Options -> Value -> Unit = milliseconds
    • Options -> Coloring -> Background = true
    • Options -> Coloring -> Thresholds = 100000000,100000000
    • Options -> Coloring -> Colors = Make grey the first threshold colour.

average duration

  • Duplicate the “Total” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.time.duration.averageSeries()
  • Set the following options.
    • General -> Info -> Title = Average
    • Options -> Value -> Stat = avg

range duration

  • Duplicate the “Average” chart in the row.
  • Set the following options.
    • General -> Info -> Title = Range
    • Options -> Value -> Stat = range

invocation details table

Tables will show invocation details per action in this row. Invocation counts, errors recorded and duration statistics are shown in separate tables.

all invocations table

  • Add a new row.
  • Set and display row name as “Invocations Per Action”.
  • Add a “Table” panel to the row.
  • Configure metric query for chart:
1
2
ibm.public.cloud-functions.$region.$namespace.*.*.error
.scale(0).offset(1).groupByNode(5, sum)
  • Set the following options.
    • General -> Info -> Title = Invocations (All)
    • Options -> Data -> Table Transform = Time series aggregations
    • Options -> Data -> Columns = Total
    • Options -> Column Styles -> Decimals = 0

error invocations table

  • Duplicate the “Invocations (All)” chart in the row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.groupByNode(5, sum)
  • Set the following options.
    • General -> Info -> Title = Invocations (Errors)

duration statistics table

  • Duplicate the “Invocations (Errors)” chart in the row.
  • Configure metric query for chart:
1
ibm.public.cloud-functions.$region.$namespace.*.*.error.groupByNode(5, avg)
  • Set the following options.
    • General -> Info -> Title = Invocations (Duration)
    • Options -> Data -> Columns = Avg, Min, Max
    • Options -> Column Styles -> Decimals = Milliseconds
    • Options -> Column Styles -> Decimals = 2

Having finished all the charts for the overview dashboard, it should look like the example above.

Let’s move onto the second dashboard, which will give us more in-depth statistics for individual actions…

Action Dashboard

This is an example of the second dashboard we want to create.

The dashboard provides information on specific action application metrics. It includes more detailed statistics including duration percentiles, memory and cpu usage. This provides more context to help diagnosing issues for individual actions.

setup

  • Create a new dashboard named “Action Details”.
  • Set the following template variables.
    • $region => ibm.public.cloud-functions.*
    • $namespace => ibm.public.cloud-functions.$region.*
    • $actions => ibm.public.cloud-functions.$region.$namespace.<action>

Replace <action> with the name of an action you are monitoring.

invocations

Action invocations are shown this first dashboard row. Bar charts display successful versus failed invocations and cold versus warm starts.

Failed invocations and cold starts are recorded as metric values. Using the scale() and offset() functions allows us to calculate successful invocations and warm starts from these properties.

  • Set and display default row name as “Invocations”.
  • Add new “Graph” chart to row.
  • Configure two metric queries for the chart:
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.error
.scale(0).offset(1).groupByNode(5, sum).alias(success)
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.error
.groupByNode(5, sum).alias(failure)
  • Set the following options to true.
    • Legend->Options->Show
    • Display->Draw Modes->Bars

invocation types

This row replicates the “Invocation Types” row from the “Overview” dashboard.

Repeat the instructions from the above to create this row here.

Metric query settings must use the action template identifier rather than a wildcard value.

invocation durations

This row uses an extended version of the durations row from the “Overview” dashboard. In addition to total and average durations, minimum and maximum are also included.

Repeat the instructions from above to add the “Total” and “Average” panels.

Metric query settings must use the action template identifier rather than a wildcard value.

minimum duration

  • Duplicate the “Total” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration.minSeries()
  • Set the following options.
    • General -> Info -> Title = Min
    • Options -> Value -> Stat = min

maximum duration

  • Duplicate the “Minimum” chart in the row.
  • Change the metric query for this chart:
1
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration.maxSeries()
  • Set the following options.
    • General -> Info -> Title = Min
    • Options -> Value -> Stat = max

percentiles graph

  • Add a “Table” panel to the row.
  • Configure this metric query for the chart:
1
2
ibm.public.cloud-functions.$region.$namespace.$action.*.time.duration
.percentileOfSeries(50, false).aliasByNode(5).alias($actions 50th percentile)
  • Duplicate this query three times, replacing 50 with 90, 95 and 99.
  • Set the following options.
    • General -> Info -> Title = Durations (Percentiles)
    • Axes -> Left Y -> Unit = Milliseconds
    • Legend -> Options -> Show = True
    • Legend -> Values -> Avg = True
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Null Value = connected

cpu usage

CPU usage for the Node.js process is recorded with two metric values, user and system time.

  • Add a new row.
  • Set and display row name as “CPU Usage”.
  • Add new “Graph” panel to row.
  • Configure two metric queries for the chart.
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.cpu.user
.groupByNode(5, avg).alias(user-time)
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.cpu.system
.groupByNode(5, avg).alias(system-time)
  • Set the following options.
    • Axes -> Left Y -> Unit = Microseconds
    • Legend -> Values -> Avg = true
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Stack = true
    • Display -> Stacking & Null value -> Null Value = connected

memory usage

Memory usage for the Node.js process is recorded with multiple values, including heap used & total, external and rss.

  • Add a new row.
  • Set and display row name as “Memory Usage”.
  • Add new “Graph” panel to row.
  • Configure four metric queries for the chart using this template.
1
2
ibm.public.cloud-functions.$region.$namespace.$actions.*.memory.<label>
.groupByNode(5, avg).alias(<label>)

Replace <label> with following options: external, rss, heapUsed & heapTotal.

  • Set the following options.
    • Axes -> Left Y -> Unit = bytes
    • Legend -> Values -> Avg = true
    • Display -> Draw Modes = Lines & Points
    • Display -> Stacking & Null value -> Stack = true
    • Display -> Stacking & Null value -> Null Value = connected

Having finished all the charts for the action details example, you should now have dashboards which look like the examples above! ๐Ÿ“ˆ๐Ÿ“Š๐Ÿ“‰

conclusion

Once you are collecting application metrics for IBM Cloud Functions (Apache OpenWhisk) applications, you need to be able to monitor metric values in real-time.

Grafana dashboards, hosted by the IBM Cloud Monitoring service, are a perfect solution for this problem. Building custom dashboards allows us to monitor incoming data values live.

In the next blog post, we’re going to finish off this series by looking at setting up automatic alerts based upon the metric values…

Capturing Runtime Metrics for OpenWhisk Applications

This blog post is part one of a series on “Monitoring Serverless Applications Metrics”. See the introduction post for details and links to other posts.

Serverless platforms pose a unique challenge for collecting application metrics. Runtime environments are ephemeral, existing only to process requests.

Using a background process to capture and export metrics to an external service is impossible. With such a restricted runtime environment, we have to look at other optionsโ€ฆ ๐Ÿค”

exporting serverless runtime metrics

console logs with alarm trigger

Apache OpenWhisk captures console logs written to stdout or stderr by actions. Logs are available in activation records through the platform API.

Runtime metrics written to the console will be saved in the activation records.

An additional action, triggered from the alarm package, can be scheduled to collect these values from the logs and forward to the metrics service.

This approach is simple and does not interfere with request processing. However, it does add a delay to the metric values being available in the collection service. This delay is dependent on the schedule chosen for the alarm trigger feed.

send before returning

Another option is to instrument the serverless functions to automatically collect and push metrics to the metric service before returning from the function.

The function will have to wait for the external metrics service to respond before returning.

This method pushes metric values into the collection service in real-time. There is no waiting for the background collection action to run. The disadvantage of this approach is that it adds a delay to each request. This delay is dependent on the response time from the collection service.

capturing runtime metrics

Metric values will have to be captured using a runtime library, due to the restrictions on running background processes.

The library should automatically capture registered metrics during each invocation. Values will be forwarded to the collection service using the configured forwarding method.

openwhisk-metrics

There is a Node.js library to ease the process of capturing runtime metrics for OpenWhisk actions.

https://github.com/jthomas/openwhisk-metrics

Node.js actions are wrapped with a proxy to automate recording metrics during invocations.

Metric values for cpu, memory, time, error and coldstart are collected by default. It supports adding custom metric collectors.

usage

Wrap action handlers with the metrics library.

1
2
3
4
5
6
7
const metrics = require('openwhisk-metrics')

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

Metrics values are logged to stdout for each invocation of the serverless function.

1
2
3
4
5
6
7
8
9
10
11
METRIC <workspace>.<action_name>.<activation>.memory.rss 53018624 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.heapTotal 34463744 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.heapUsed 16955224 1512489781
METRIC <workspace>.<action_name>.<activation>.memory.external 987361 1512489781
METRIC <workspace>.<action_name>.<activation>.error 0 1512489781
METRIC <workspace>.<action_name>.<activation>.coldstart 0 1512489781
METRIC <workspace>.<action_name>.<activation>.cpu.user 177 1512489781
METRIC <workspace>.<action_name>.<activation>.cpu.system 2 1512489781
METRIC <workspace>.<action_name>.<activation>.time.start 1511605588388 1512489781
METRIC <workspace>.<action_name>.<activation>.time.end 1511605588468 1512489781
METRIC <workspace>.<action_name>.<activation>.time.duration 80 1512489781

Once you are collecting metrics, you need a monitoring service to forward them toโ€ฆ

monitoring service

We’re going to look at collecting and visualising metrics using the IBM Cloud Monitoring Service.

Use the IBMยฎ Cloud Monitoring service to expand your collection and retention capabilities when working with metrics, and to be able to define rules and alerts that notify you of conditions that require attention.

IBM Cloud Monitoring service comes with a metric collection service, Grafana-based visualisation dashboard and an alerting system.

Let’s demonstrate how to use the approaches listed above for exporting metrics to the IBM Cloud Monitoring service.

There’s an additional Node.js library to integrate the OpenWhisk Metrics library with the IBM Cloud Monitoring Service. This can forward metrics in real-time or batches (using a schedule action from a timer).

provisioning

An instance of the service must be provisioned before being used.

The service is available in multiple regions. Choose the same region as the IBM Cloud Functions instance for best performance.

Instances can be provisioned through the IBM Cloud dashboard or the IBM Cloud CLI tool.

1
 $ bx cf create-service Monitoring lite my_monitoring_svc

For more details on provisioning instances of this service, please see the documentation: https://console.bluemix.net/docs/services/cloud-monitoring/how-to/provision.html#provision

authentication

IBM Cloud Monitoring supports the following authentication methods: API Key, UAA Token or IAM Token.

An API key will be used to provide authentication credentials in the examples below.

Keys can be created through the IBM Cloud dashboard or the IBM Cloud CLI tool.

1
$ bx iam api-key-create metrics-key -d "API Key For Serverless Metrics"

Note: The API key is only shown at the time of creation. If the API key is lost, you must create a new API key.

For more details on creating API keys, please see the documentation here: https://console.bluemix.net/docs/services/cloud-monitoring/security/auth_api_key.html#auth_api_key

space domain

The GUID of the account space is also required to use the metrics service.

Retrieve this value using the IBM Cloud CLI tool. Replace SpaceName with the name of the space.

1
bx iam space SpaceName --guid

The GUID for the space is returned.

1
2
$ bx iam space dev --guid
667fadfc-jhtg-1234-9f0e-cf4123451095

Note: Space GUIDs must be prefixed with s- when being using with the monitoring service.

1
"667fadfc-jhtg-1234-9f0e-cf4123451095" => "s-667fadfc-jhtg-1234-9f0e-cf4123451095"

example (real-time forwarding)

Let’s start with an example of using real-time forwarding of metrics values. Metric values will be automatically collected by the runtime library. Before each invocation finishes, the library will send the values to the external collection service.

This example assumes you already have the CLI tool for IBM Cloud Functions installed and configured. If you need to do this, please follow the instructions here.

create new directory

1
2
$ mkdir hello-world
$ cd hello-world

initialise npm package

1
$ npm init -y

install libraries

1
$ npm install openwhisk-metrics cloud-functions-metrics-service

update action handler source

Create a file called index.js with following code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const metrics = require('openwhisk-metrics')
const service = require('cloud-functions-metrics-service')

metrics.service = service.client({
  host: 'metrics.<???>.bluemix.net',
  scope: 's-<???>',
  api_key: '<???>'
})

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

Replace the host, scope and api_key parameters custom values. Host names for the regional monitoring service endpoints are listed here.

Space GUIDs must be prefixed with s- to identify a space in the scope parameter.

create new action

1
2
$ zip -r action.zip index.js package.json node_modules
$ wsk action create hello-world --kind nodejs:8 action.zip

invoke action

1
2
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 74add543b3b94bbbadd543b3b9dbbb17

use api to list metrics

Use curl to manually list the ingested metric labels for IBM Cloud Functions.

Replace the <???> values in the command with configuration values from above.

1
2
3
4
5
6
7
8
9
$ curl --url 'https://metrics.<???>.bluemix.net/v1/metrics/list?query=ibm.public.cloud-functions.*.*.*.*' --header 'x-auth-scope-id: s-<???>' --header 'X-Auth-User-Token: apikey <???>'
[{
  "leaf": 0,
  "context": {},
  "text": "72df4bc809c04fae9f4bc809c01fae77",
  "expandable": 1,
  "id": "ibm.public.cloud-functions.*.*.*.72df4bc809c04fae9f4bc809c01fae77",
  "allowChildren": 1
}]

Each activation identifier should be listed as a label value in the metrics service.

example (batch forwarding)

If we don’t want to add a (short) delay to each invocation, metric values can be forwarded asynchronously. An explicit action will be created to forward metric values from invocation logs. This action will be triggered on a periodic schedule using the alarm trigger feed.

This example assumes you already have the CLI tool for IBM Cloud Functions installed and configured. If you need to do this, please follow the instructions here.

create sample action

1
2
3
4
$ mkdir hello-world
$ cd hello-world
$ npm init -y
$ npm install openwhisk-metrics

Create a file called index.js with the following code.

1
2
3
4
5
6
7
const metrics = require('openwhisk-metrics')

const main = params => {
  return { message: "Hello World" }
}

module.exports.main = metrics(main)

deploy and test sample action

Package and deploy hello-world action.

1
2
$ zip -r action.zip index.js package.json node_modules
$ wsk action create hello-world --kind nodejs:8 action.zip

Metric values are written to the console for each invocation.

1
2
3
4
5
6
7
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 28da39d219df436a9a39d219df036a30
$ wsk activation logs 28da39d219df436a9a39d219df036a30
2017-12-18T14:38:50.751615113Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.cpu.user 0 1513607930
2017-12-18T14:38:50.751672372Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.cpu.system 0 1513607930
2017-12-18T14:38:50.751685034Z stdout: METRIC user@host_dev.hello-world.28da39d219df436a9a39d219df036a30.time.start 1513607930749 1513607930
...

create metric-forwarder action

Clone the project repository and install NPM dependencies.

1
2
3
$ git clone https://github.com/jthomas/cloud-functions-metrics-service
$ cd cloud-functions-metrics-service
$ npm install

Update action configuration file (config.json) with the following parameter values.

1
2
3
4
5
6
7
8
{
  "actions": ["hello-world"],
  "service": {
    "host": "metrics.<???>.bluemix.net",
    "scope": "s-<???>",
    "api_key": "<???>"
  }
}

Replace the <???> fields in the configuration file from values from above.

Package metric-forwarder action.

1
$ zip -r action.zip index.js package.json lib node_modules

deploy metric-forwarder action

Create new metric-forwarder action from deployment package and configuration file.

1
$ wsk action create metric-forwarder --kind nodejs:8 action.zip --param-file config.json

Create trigger feed for alarm package to run metric-forwarder on periodic schedule.

1
2
3
$ wsk trigger create interval \
  --feed /whisk.system/alarms/interval \
  --param minutes 1

Bind trigger to action using rule.

1
$ wsk rule create forward-metrics-on-interval interval metric-forwarder

invoke sample action

1
2
$ wsk action invoke hello-world -b
ok: invoked /_/hello-world with id 28da39d219df436a9a39d219df036a30

This will generate activation records containing metric values. When the interval trigger is next fired, metric values from these records forwarded to the collection service.

Logs from the metric-forwarder action will show the activation records that have been retrieved.

1
$ wsk activation poll metric-forwarder

use api to list metrics

Use curl to manually list the ingested metric labels for IBM Cloud Functions.

Replace the <???> values in the command with configuration values from above.

1
2
3
4
5
6
7
8
9
$ curl --url 'https://metrics.<???>.bluemix.net/v1/metrics/list?query=ibm.public.cloud-functions.*.*.*.*' --header 'x-auth-scope-id: s-<???>' --header 'x-auth-user-token: apikey <???>'
[{
  "leaf": 0,
  "context": {},
  "text": "72df4bc809c04fae9f4bc809c01fae77",
  "expandable": 1,
  "id": "ibm.public.cloud-functions.*.*.*.28da39d219df436a9a39d219df036a30",
  "allowChildren": 1
}]

Each activation identifier should be listed as a label value in the metrics service.

next steps

Metric values from our serverless applications are now being collected by the IBM Cloud Monitoring service. ๐Ÿ‘๐Ÿ‘๐Ÿ‘

Applications metrics are automatically recorded by the runtime library for each invocation. Metric values are exported to the monitoring service in real-time or using a timed action to forward in batches.

Setting up monitoring dashboards from the collected values will allow us to identify and resolve issues with our serverless applications. In the next blog post, we’ll look using Grafana to visualise metric values being collectedโ€ฆ

Monitoring Serverless Applications Metrics

Serverless applications are not “No Ops”, despite the meme. ๐Ÿ™„

Monitoring runtime metrics is still crucial for serverless applications. Identifying, diagnosing and resolving issues in production requires data on execution durations, errors thrown, resource usage statistics amongst other metrics.

IBM Cloud has a comprehensive series of tools for monitoring application metrics. In this blog post series, we’re going to look at using these tools to monitor metrics from serverless applications. ๐Ÿ’ฏ๐Ÿ’ฏ๐Ÿ’ฏ

overview

We’ll start with understanding how to capture metrics from IBM Cloud Functions (Apache OpenWhisk) applications. Then we’ll look at methods for automatically sending metric values into the IBM Cloud Monitoring service. Once metric data is being captured, we’ll explore Grafana to create dashboards to monitor data values in real-time. Finally, we’ll look at configuring alerting services to notify us when issues occur.

Here’s a sneak peak of some of the dashboards we’ll be creating from serverless application metricsโ€ฆ

blog posts

Here are the links to the blog posts in the seriesโ€ฆ

You can start now with the first blog post, other blog posts will follow soonโ€ฆ.

tldr?

pssst. want to set this up without reading all those boring words and stuff?

Here are the links to the open-source libraries used to set this up. Follow the instructions in the repositories to enable capturing and forwarding metrics to the monitoring service.

Here are the JSON configuration files for the Grafana dashboards.

https://gist.github.com/jthomas/47466693e88d0407c239535624222e6b

Import these files to create new dashboards and add the following template variables.

  • $region => ibm.public.cloud-functions.*
  • $namespace => ibm.public.cloud-functions.$region.*
  • $actions => ibm.public.cloud-functions.$region.$namespace.*

Openwhisk Logstash Forwarder

Kibana Dashboard

Debugging serverless applications in production is often reliant on application logs, due to having no access to the runtime environment. No SSHing into the machine and attaching a debugger to a process or using strace to dump system calls.

Storing, searching and analysing serverless application logs is crucial to diagnosing and fixing bugs on serverless platforms.

The ”ELK Stack” has become a popular solution for managing applications logs. Combining three open-source projects (ElasticSearch, Logstash and Kibana), this solution provides a scalable platform for importing, storing and searching application logs.

How can we use the ELK stack to manage logs for serverless applications running on Apache OpenWhisk?

ELK and OpenWhisk

In traditional application runtimes, like a VM or a Docker container, a background agent is used to automatically forward application and system logs to the ingestion service for the ELK stack.

However, serverless applications run in an ephemeral environment. Runtimes are instantiated on-demand per request and destroyed after the function returns. These runtimes do not support the use of background agents.

One solution for this is the custom OpenWhisk plugin for Logstash. This plugin polls the platform for new logs and automatically ingests them into ElasticSearch.

But what if you are using a hosted ELK service that does not support installing custom plugins?

OpenWhisk Logstash Forwarder

OpenWhisk Logstash Forwarder” is designed for this scenario. It can ingest logs into ElasticSearch using standard Logstash input plugins.

https://github.com/jthomas/openwhisk-logstash-forwarder

This project contains an OpenWhisk action which acts as a “serverless” version of the logstash-forwarder agent. When the action executes, it retrieves all new logs from a user-provided list of actions to monitor. Log messages are pushed into Logstash using the Lumberjack protocol.

The action is connected to an alarm trigger feed with a customisable schedule. This event source will ensure all logs are forwarded on a regular schedule.

Demo

Demo

In this example, the developer has the serverless logstash forwarder agent deployed in their workspace. The agent is configured to monitor logs from the forecast action. The alarm trigger feed is connected to the monitoring action and runs once per minute.

Invoking the forecast action generates log messages to be ingested.

When the alarm trigger feed next fires, the monitoring action is executed. It retrieves log messages generated by new forecast activations and pushes those logs into the configured ELK instance.

Opening Kibana and refreshing the monitoring dashboard, new log messages are shown as individual documents. Selecting the individual documents shows the log message contents with activation record details.

Source Code

The source code for this project is now available on Github:

https://github.com/jthomas/openwhisk-logstash-forwarder

See the installation instructions for how to deploy this project on an OpenWhisk platform.

This project needs an instance of OpenWhisk platform and an ELK-stack service accessible on a public IP address.

This project can be deployed using The Serverless Framework or the OpenWhisk CLI.

Advanced Openwhisk Alarm Schedules

Apache OpenWhisk supports a cron-based alarm package for invoking serverless functions on a fixed schedule, e.g. every 5 minutes, every day at 5PM, once a week.

Scheduled events allow functions to be invoked for background processes or batch operations, like processing logs generated in the past 24 hours.

Using a cron-based schedule pattern, running functions once a minute, every two hours or 5pm on Mondays is simple, but what about more complex schedule patterns? ๐Ÿค”

What if we need toโ€ฆ

  • โฐ Fire a single one-off event at a specific time in the future?
  • โฐ Fire events a fixed period of time from an action finishing?
  • โฐ Fire events on an irregular schedule?

It is possible to implement all these examples with a few tricksโ€ฆ ๐Ÿคนโ€โ™‚๏ธ๐Ÿคนโ€โ™‚๏ธ๐Ÿคนโ€โ™‚๏ธ.

Before we dive into the details, let’s review how the alarm feed provider worksโ€ฆ

Alarm Trigger Feeds

OpenWhisk triggers are connected to external event sources using feed providers.

Feed providers listen to event sources, like message queues, firing triggers with event parameters as external events occur.

There are a number of pre-installed feed providers in the whisk.system namespace. This includes the alarms package which includes a feed provider (/whisk.system/alarms/alarm).

1
2
3
4
5
$ wsk package get /whisk.system/alarms --summary
package /whisk.system/alarms: Alarms and periodic utility
   (parameters: *apihost, *cron, *trigger_payload)
 feed   /whisk.system/alarms/alarm: Fire trigger when alarm occurs
   (parameters: none defined)

feed parameters

The following parameters are used to configure the feed provider.

  • cron - crontab syntax used to configure timer schedule.
  • trigger_payload - event parameters to fire trigger with.
  • maxTriggers - maximum number of triggers to fire (-1 for no limit).

cron is the parameter which controls when triggers will be fired. It uses the cron syntax to specify the schedule expression.

cron schedule format

Cron schedule values are a string containing sections for the following time fields. Field values can be integers or patterns including wild cards.

1
2
3
4
5
6
7
8
9
# โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ second (0 - 59, optional & defaults to 0)
# โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ minute (0 - 59)
# โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ hour (0 - 23)
# โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of month (1 - 31)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ month (0 - 11)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of week (0 - 6) (Sunday to Saturday)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# * * * * * *  

NOTE: Month field starts from 0 not 1, 0 is January with December being 11. Day of week also starts from 0. Sunday is first day of the week.

The second field is a non-standard cron field and does not need to be used. The Node.js module used to parse the cron schedules supports a value with five or six fields.

crontab examples

Here are some example patternsโ€ฆ

  • */10 * * * * * - run every 10 seconds
  • * * * * * - run every minute
  • 0 * * * * - run every hour
  • 0 */2 * * * - run every two hours
  • 30 11 * * 1-5 - run Monday to Friday at 11:30AM
  • 0 0 1 * * - run at midnight the first day of the month

https://crontab.guru/ is an online editor for generating cron schedule expressions.

Creating Alarm Triggers

Using the wsk cli triggers can be created using the alarm feed. Schedule and event parameters are passed in using command-line arguments (-p name value).

1
2
3
4
$ wsk trigger create periodic --feed /whisk.system/alarms/alarm -p cron '* * * * * *' -p trigger_payload '{"hello":"world"}'
ok: invoked /whisk.system/alarms/alarm with id 42ca80fbe7cf47318a80fbe7cff73177
...
ok: created trigger feed periodic

Trigger invocations are recorded in the activation records.

1
2
3
4
5
6
7
8
9
10
11
$ wsk activation list periodic
activations
d55d15297781474b9d15297781974b92 periodic
...
$ wsk activation get d55d15297781474b9d15297781974b92
ok: got activation d55d15297781474b9d15297781974b92
{
    "namespace": "user@host.com_dev",
    "name": "periodic",
    ...
}

Deleting the trigger will automatically remove the trigger from the alarm scheduler.

1
2
3
4
$ wsk delete periodic
ok: invoked /whisk.system/alarms/alarm with id 44e8fc5e76c64175a8fc5e76c6c175dd
...
ok: deleted trigger periodic

Programmatic Creation

The OpenWhisk JavaScript library can also register and remove triggers with feed providers.

1
2
3
4
5
6
7
8
const params = {cron: '* * * * * *', trigger_payload: {"hello":"world"}}
const name = '/whisk.system/alarms/alarm'
const trigger = 'periodic'
ow.feeds.create({name, trigger, params}).then(package => {
  console.log('alarm trigger feed created', package)
}).catch(err => {
  console.error('failed to create alarm trigger', err)
})

Triggers must already exist before registering with the feed provider using the client library.

Using the client library provides a mechanism for actions to dynamically set up scheduled events.

Advanced Examples

Having reviewed how the alarm feed works, let’s look at some more advanced use-cases for the schedulerโ€ฆ

Schedule one-off event at a specific time in the future

Creating one-off events, that fire at a specific date and time, is possible using the cron and maxTriggers parameters together.

Using the minute, hour, day of the month and month fields in the cron parameter, the schedule can be configured to run once a year. The day of the week field will use the wildcard value.

Setting the maxTriggers parameter to 1, the trigger is removed from the scheduler after firing.

happy new year example

What if we want to fire an event when the New Year starts?

Here’s the cron schedule for 00:00 on January 1st.

1
2
3
4
5
6
7
8
# โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ minute (0 - 59)
# โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ hour (0 - 23)
# โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of month (1 - 31)
# โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ month (0 - 11)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of week (0 - 6) (Sunday to Saturday)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# 0 0 1 0 *

Here are the cli commands to set up a trigger to run at 01/01/2018 @ 00:00 to celebrate the new year.

1
2
3
4
$ wsk trigger create new_year --feed /whisk.system/alarms/alarm -p cron '0 0 1 0 *' -p maxTriggers 1 -p trigger_payload '{"message":"Happy New Year!"}'
ok: invoked /whisk.system/alarms/alarm with id 754bec0a58b944a68bec0a58b9f4a6c1
...
ok: created trigger new_year

Firing events a fixed period of time from an action finishing

Imagine you want to run an action on a loop, with a 60 second delay between invocations. Start times for future invocations are dependent on the finishing time of previous invocations. This means we can’t use the alarm feed with a fixed schedule like ’* * * * *’.

Instead we’ll schedule the first invocation as a one-off event and then have the action re-schedule itself using the JavaScript client library!

action code

Here’s the sample JavaScript code for an action which does thatโ€ฆ.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
const openwhisk = require('openwhisk');

function calculateSchedule() {
  const now = new Date()
  const seconds = now.getSeconds()
  const nextMinute = (now.getMinutes() + 1) % 60
  return `${seconds} ${nextMinute} * * * *`
}

function main(params) {
  const ow = openwhisk();
  const params = {cron: calculateSchedule(), maxTriggers: 1}
  console.log(params)
  return ow.feeds.delete({name: '/whisk.system/alarms/alarm', trigger: 'delay'}).then(() => {
    console.log('delay trigger feed deleted.')
    return ow.feeds.create({name: '/whisk.system/alarms/alarm', trigger: 'delay', params: params})
    }).then(result => {
     console.log('delay trigger feed created.')
    })
    .catch(err => {
      console.error('failed to create/delete delay trigger', err)
      console.log("ERROR", err.error.response.result)
    })
}

setting up

  • Create an action called reschedule with code from above.
1
2
$ wsk action create reschedule reschedule.js
ok: created action reschedule
  • Create a trigger (delay) using the alarm feed, set to run in the next 60 seconds.
1
2
3
4
$ wsk trigger create delay --feed /whisk.system/alarms/alarm -p cron '* * * * * *'
ok: invoked /whisk.system/alarms/alarm with id b3da4de5726b41679a4de5726b0167c8
...
ok: created trigger delay
  • Connect the action (reschedule) to the trigger (delay) with a rule (reschedule_delay).
1
2
$ wsk rule create reschedule_delay delay reschedule
ok: created rule reschedule_delay

This action will continue to re-schedule itself indefinitely.

Stop this infinite loop by disabling or removing the rule connecting the action to the trigger.

1
2
$ wsk rule disable reschedule_delay
ok: disabled rule reschedule_delay

Firing events on an irregular schedule

How can you schedule events to occur from a predictable but irregular pattern, e.g. sending a daily message to users at sunrise?

Sunrise happens at a different time each morning. This schedule cannot be defined using a static cron-based pattern.

Using the same approach as above, where actions re-schedule triggers at runtime, events can created to follow an irregular schedule.

sunrise times

This external API provides the sunrise times for a location. Retrieving the sunrise times for tomorrow during execution will provide the date time used to re-schedule the action.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
    "results": {
        "astronomical_twilight_begin": "5:13:40 AM",
        "astronomical_twilight_end": "6:48:52 PM",
        "civil_twilight_begin": "6:14:23 AM",
        "civil_twilight_end": "5:48:09 PM",
        "day_length": "10:40:26",
        "nautical_twilight_begin": "5:43:50 AM",
        "nautical_twilight_end": "6:18:42 PM",
        "solar_noon": "12:01:16 PM",
        "sunrise": "6:41:03 AM",
        "sunset": "5:21:29 PM"
    },
    "status": "OK"
}

action code

Here’s the sample JavaScript action that will re-schedule itself at sunrise.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
const openwhisk = require('openwhisk');
const request = require('request-promise');

function getNextSunrise(lat, lng, when) {
  const options = {
    uri: 'https://api.sunrise-sunset.org/json',
    qs: { lat: lat, lng: lng, when: when },
    json: true
  }

  return request(options)
    .then(result => result.results.sunrise)
}

function calculateSchedule(sunrise) {
  console.log('Next sunrise:', sunrise)
  const sections = sunrise.split(':')
  const hour = sections[0], minute = sections[1]
  return `${minute} ${hour} * * *`
}

function scheduleSunriseEvent (sunrise) {
  const ow = openwhisk();
  const params = {cron: sunrise, maxTriggers: 1}
  return ow.feeds.delete({name: '/whisk.system/alarms/alarm', trigger: 'sunrise'}).then(() => {
    console.log('trigger feed deleted.')
    return ow.feeds.create({name: '/whisk.system/alarms/alarm', trigger: 'sunrise', params: params})
  }).then(result => {
    console.log('trigger feed created.')
  })
  .catch(err => {
    console.error('failed to create/delete trigger', err)
    console.log("ERROR", err.error.response.result)
  })
}

function main(params) {
  console.log('GOOD MORNING!')

  return getNextSunrise(params.lat, params.lng, 'tomorrow')
    .then(calculateSchedule)
    .then(scheduleSunriseEvent)
}

setting up

  • Create an action called wake_up with code from above. lat and lng parameters define location for sunrise.
1
2
$ wsk action create wake_up wake_up.js -p lat 51.50 -p lng -0.076
ok: created action wake_up
  • Create a trigger (sunrise) with the alarm feed, scheduled for the next sunrise.
1
2
3
4
$ wsk trigger create sunrise --feed /whisk.system/alarms/alarm -p cron '03 41 06 * * *'
ok: invoked /whisk.system/alarms/alarm with id 606dafe276f24400adafe276f2240082
...
ok: created trigger sunrise
  • Connect the action (wake_up) to the trigger (sunrise) with a rule (wake_up_at_sunrise).
1
2
$ wsk rule create wake_up_at_sunrise sunrise wake_up
ok: created rule wake_up_at_sunrise

Checking the activation logs the following morning will show the trigger being fired, which invokes the action, which re-schedules the one-off event! ๐ŸŒ…๐ŸŒ…๐ŸŒ…

Caveats

Here’s a few issues you might encounter using the alarm feed that I ran intoโ€ฆ.

  • Month field in cron schedule starts from zero not one. January is 0, December is 11.
  • Day of the week field starts from zero. First day of the week is Sunday, not Monday.
  • Feeds cannot be updated with a new schedule once created. Feeds must be deleted before being re-created to use a different schedule.

Future Plans

Extending the alarm feed to support even more features and improve the developer experience is in-progress. There are a number of Github issues in the official OpenWhisk repository around this work.

If you have feature requests, discover bugs with the feed or have other suggestions, please comment on the existing issues or open new ones.

Conclusion

Scheduled events are a necessary feature of serverless cloud platforms. Due to the ephemeral nature of runtime environments, scheduling background tasks must be managed by the platform.

In Apache OpenWhisk, the alarm feed allows static events to be generated on a customisable schedule. Using a cron-based schedule pattern, running functions once a minute, every two hours or 5pm on Mondays, is simple but what about more complex schedule patterns?

Using the cron and maxTriggers parameters with the OpenWhisk client library, much more advanced event schedules can be utilised within the platform. In the examples above, we looked at how to schedule one-off events, events using a predictable but irregular schedule and how actions can re-schedule events at runtime. ๐Ÿ’ฏ๐Ÿ’ฏ๐Ÿ’ฏ

Large Applications on OpenWhisk

OpenWhisk supports creating actions from archive files containing source files and project dependencies.

The maximum code size for the action is 48MB.

Applications with lots of third-party modules, native libraries or external tools may be soon find themselves running into this limit. Node.js libraries are notorious for having large amounts of dependencies.

What if you need to deploy an application larger than this limit to OpenWhisk?

Previous solutions used Docker support in OpenWhisk to build a custom Docker image per action. Source files and dependencies are built into a public image hosted on Docker Hub.

This approach overcomes the limit on deployment size but means application source files will be accessible on Docker Hub. This is not an issue for building samples or open-source projects but not realistic for most applications.

So, using an application larger than this limit requires me to make my source files public? ๐Ÿค”

There’s now a better solution! ๐Ÿ‘๐Ÿ‘๐Ÿ‘

OpenWhisk supports creating actions from an archive file AND a custom Docker image.

If we build a custom Docker runtime which includes shared libraries, those dependencies don’t need including in the archive file. Private source files will still be bundled in the archive and injected at runtime.

Reducing archive file sizes also improves deployment times.

Let’s look at an exampleโ€ฆ

Using Machine Learning Libraries on OpenWhisk

Python is a popular language for machine learning and data science. Libraries like pandas, scikit-learn and numpy provide all the tools. Serverless computing is becoming a good choice for machine learning microservices.

OpenWhisk supports Python 2 and 3 runtimes.

Popular libraries like flask, requests and beautifulsoup are available as global packages. Additional packages can be imported using virutalenv during invocations.

Python Machine Learning Libraries

Python packages can be used in OpenWhisk using virtualenv. Developers install the packages locally and include the virutalenv folder in the archive for deployment.

Machine Learning libraries often use numerous shared libraries and compile native dependencies for performance. This can lead to hundreds of megabytes of dependencies.

Setting up a new virtualenv folder and installing pandas leads to an environment with nearly 100MB of dependencies.

1
2
3
4
5
6
7
8
9
$ virtualenv env
$ source env/bin/activate
$ pip install pandas
...
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.13.1 pandas-0.20.3 python-dateutil-2.6.1 pytz-2017.2 six-1.10.0
$ du -h
...
84M   . <-- FOLDER SIZE ๐Ÿ˜ฑ

Bundling these libraries within an archive file will not be possible due to the file size limit.

Custom OpenWhisk Runtime Images

Overcoming this limit can be achieved using a custom runtime image. The runtime will pre-install additional libraries during the build process and make them available during invocations.

OpenWhisk uses Docker for the runtime containers. Source files for the images are available on Github under the core folder. Here’s the Dockerfile for the Python runtime: https://github.com/apache/incubator-openwhisk/blob/master/core/pythonAction/Dockerfile.

Images for OpenWhisk runtimes are also available on Docker Hub under the OpenWhisk organisation.

Docker supports building new images from a parent image using the FROM directive. Inheriting from the existing runtime images means the Dockerfile for the new runtime only has to contain commands for installing extra dependencies.

Let’s build a new Python runtime which includes those libraries as shared packages.

Building Runtimes

Let’s create a new Dockerfile which installs additional packages into the OpenWhisk Python runtime.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
FROM openwhisk/python3action

# lapack-dev is available in community repo.
RUN echo "http://dl-4.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories

# add package build dependencies
RUN apk add --no-cache \
        g++ \
        lapack-dev \
        gfortran

# add python packages
RUN pip install \
    numpy \
    pandas \
    scipy \
    sklearn

Running the Docker build command will create a new image with these extra dependencies.

1
2
3
4
5
6
7
$ docker build -t python_ml_runtime .
Sending build context to Docker daemon  83.01MB
Step 1/4 : FROM openwhisk/python3action
 ---> 46388e726fae
...
Successfully built cfc14a93863e
Successfully tagged python_ml_runtime:latest

Hosting images on Docker Hub requires registering a (free) account @ https://hub.docker.com/

Create a new tag from the python_ml_runtime image containing the Docker Hub username.

1
$ docker tag python_ml_runtime <YOUR_USERNAME>/python_ml_test

Push the image to Docker Hub to make it available to OpenWhisk.

1
$ docker push <YOUR_USERNAME>/python_ml_test

Testing It Out

Create a new Python file (main.py) with the following contents:

1
2
3
4
5
6
7
8
9
10
11
12
import numpy
import pandas
import sklearn
import scipy

def main(params):
    return {
        "numpy": numpy.__version__,
        "pandas": pandas.__version__,
        "sklearn": sklearn.__version__,
        "scipy": scipy.__version__
    }

Create a new OpenWhisk action using the Docker image from above and source file.

1
2
$ wsk action create lib-versions --docker <YOUR_USERNAME>/openwhisk_python_ml main.py
ok: created action lib-versions

Invoke the action to verify the modules are available and return the versions.

1
2
3
4
5
6
7
$ wsk action invoke lib-versions --result
{
    "numpy": "1.13.1",
    "pandas": "0.20.3",
    "scipy": "0.19.1",
    "sklearn": "0.18.2"
}

Yass. It works. ๐Ÿ’ƒ๐Ÿ•บ

Serverless Machine Learning here we comeโ€ฆ. ๐Ÿ˜‰

Conclusions

Using custom runtimes with private source files is an amazing feature of OpenWhisk. It enables developers to run larger applications on the platform but also enables lots of other use cases. Almost any runtime, library or tool can now be used from the platform.

Here are some examples of where this approach could be usedโ€ฆ

  • Installing global libraries to reduce archive file size under 48MB and speed up deployments.
  • Upgrading language runtimes, i.e. using Node.js 8 instead of 6.
  • Adding native dependencies or command-line tools to the environment, e.g. ffmpeg.

Building new runtimes is really simple using pre-existing base images published on Dockerhub.

The possibilities are endless!

Creating Swift Binaries for OpenWhisk

In the previous blog post, we explained how to write Serverless Swift functions using OpenWhisk actions.

Swift sources files are compiled into a binary by the platform before processing requests.

This compilation process adds a delay on the invocation time for “cold” runtimes. If the action has not been invoked for a while, the system is under heavy load or multiple invocations are received in parallel, a new runtime will need to be initialised.

Pre-compiled binaries can be deployed to remove this delay. Binaries must be compiled for the correct platform architecture and support execution through the OpenWhisk runtime.

There is now a Swift package to make the process of building pre-compiled binaries much easier.

Let’s have a look at how this worksโ€ฆ

Swift Packages

Swift introduced a package manager in Swift 3.0. The package manager integrates with the build system to “automate the process of downloading, compiling, and linking dependencies”.

Swift uses a manifest file (Packages.swift) to define package properties including dependencies.

Example Swift Package

Here’s an example manifest file from a sample package with external dependencies.

1
2
3
4
5
6
7
8
9
10
11
12
import PackageDescription

let package = Package(
    name: "DeckOfPlayingCards",
    targets: [],
    dependencies: [
        .Package(url: "https://github.com/apple/example-package-fisheryates.git",
                 majorVersion: 1),
        .Package(url: "https://github.com/apple/example-package-playingcard.git",
                 majorVersion: 1),
    ]
)

Packages are referenced through a URL which resolves to a Git repository. Semantic versioning conventions are used to control the package version installed.

External packages are downloaded, compiled and linked in the project during the build process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ swift build
Fetching https://github.com/apple/example-package-deckofplayingcards.git
Fetching https://github.com/apple/example-package-fisheryates.git
Fetching https://github.com/apple/example-package-playingcard.git
Cloning https://github.com/apple/example-package-fisheryates.git
Resolving https://github.com/apple/example-package-fisheryates.git at 2.0.3
Cloning https://github.com/apple/example-package-playingcard.git
Resolving https://github.com/apple/example-package-playingcard.git at 3.0.2
Cloning https://github.com/apple/example-package-deckofplayingcards.git
Resolving https://github.com/apple/example-package-deckofplayingcards.git at 3.0.3
Compile Swift Module 'PlayingCard' (3 sources)
Compile Swift Module 'FisherYates' (2 sources)
Compile Swift Module 'DeckOfPlayingCards' (1 sources)
Compile Swift Module 'Dealer' (1 sources)
Linking ./.build/debug/Dealer
$

OpenWhiskAction Package

OpenWhiskAction is a Swift package for registering Swift functions as OpenWhisk actions.

It bundles the Swift source files used to implement the runtime handler for OpenWhisk as a library. Using this package means developers do not have to manually import those source files into their projects.

usage

This package exposes a public function (OpenWhiskAction ) that should be called with a function reference (([String: Any]) -> [String: Any])) as a named parameter (main). The callback will be executed with the invocation parameters. Returned values will be serialised as the invocation response.

1
2
3
4
5
6
7
8
9
10
11
import OpenWhiskAction

func hello(args: [String:Any]) -> [String:Any] {
    if let name = args["name"] as? String {
      return [ "greeting" : "Hello \(name)!" ]
    } else {
      return [ "greeting" : "Hello stranger!" ]
    }
}

OpenWhiskAction(main: hello)

example

Let’s show an example of using the package to build a pre-compiled Swift action for OpenWhisk.

create new package

Create a new directory and use the swift package init command to generate the boilerplate package.

1
2
3
4
5
6
7
8
9
10
11
12
$ mkdir Action
$ cd Action/
$ swift package init
Creating library package: Action
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/Action.swift
Creating Tests/
Creating Tests/LinuxMain.swift
Creating Tests/ActionTests/
Creating Tests/ActionTests/ActionTests.swift

add package dependency

Add the OpenWhiskAction package as a dependency to the manifest file (Package.swift).

1
2
3
4
5
6
7
8
import PackageDescription

let package = Package(
    name: "Action",
    dependencies: [
        .Package(url: "https://github.com/jthomas/OpenWhiskAction.git", majorVersion: 0)
    ]
)

write serverless function

Create a new main.swift file under the Sources directory containing the following source code.

1
2
3
4
5
6
7
8
9
10
11
import OpenWhiskAction

func hello(args: [String:Any]) -> [String:Any] {
    if let name = args["name"] as? String {
      return [ "greeting" : "Hello \(name)!" ]
    } else {
      return [ "greeting" : "Hello stranger!" ]
    }
}

OpenWhiskAction(main: hello)

Swift’s build process will produce an executable if the package contains a main.swift file. That file will be compiled as the package binary.

compiling with docker

OpenWhisk Swift actions use a custom Docker image as the runtime environment. Compiling application binaries from this image will ensure it is compatible with the platform runtime.

This command will run the swift build command within a container from this image. The host filesystem is mounted into the container at /swift-package. Binaries and other build artifacts will be available in ./.build/release/ after the command has executed.

1
docker run --rm -it -v $(pwd):/swift-package openwhisk/action-swift-v3.1.1 bash -e -c "cd /swift-package && swift build -v -c release"

deploying to openwhisk

OpenWhisk actions can be created from a zip file containing action artifacts. The zip file will be expanded prior to execution. In the Swift environment, the Swift binary executed by the platform should be at ./.build/release/Action.

If an action is deployed from a zip file which contains this file, the runtime will execute this binary rather than compiling a new binary from source code within the zip file.

1
2
3
4
5
6
7
8
$ zip action.zip .build/release/Action
  adding: .build/release/Action (deflated 67%)
$ wsk action create swift-action --kind swift action.zip
ok: created action swift-action
$ wsk action invoke --blocking --result -p name "Bernie Sanders" swift-action
{
    "greeting": "Hello Bernie Sanders!"
}

Using With The Serverless Framework

As shown in the previous blog post, The Serverless Framework supports the Swift runtime. Actions can either be created from Swift source files or pre-compiled binaries.

This example project demonstrates how to integrate pre-compiled binaries into a serverless framework application.

example project

The project contains two Swift source files under the Sources directory. Using the main.swift file name means these files will be compiled into separate binaries under the .build/release directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ tree .
.
โ”œโ”€โ”€ Package.swift
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ Sources
โ”‚ย ย  โ”œโ”€โ”€ hello
โ”‚ย ย  โ”‚ย ย  โ””โ”€โ”€ main.swift
โ”‚ย ย  โ””โ”€โ”€ welcome
โ”‚ย ย      โ””โ”€โ”€ main.swift
โ”œโ”€โ”€ package.json
โ””โ”€โ”€ serverless.yml

3 directories, 6 files

The package manifest (Package.swift) contains the OpenWhiskAction dependency.

serverless.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
service: swift-packages

provider:
  name: openwhisk
  runtime: swift

functions:
  hello:
    handler: .build/release/hello
  welcome:
    handler: .build/release/welcome

custom:
  scripts:
    hooks:
      'package:initialize': npm run-script compile
plugins:
  - serverless-openwhisk
  - serverless-plugin-scripts

This configuration file describes two actions (hello and welcome) using the swift runtime. The handler property for those actions refers to a binary, produced by the build process, rather than source file.

compile during deployment

Before using serverless deploy command to create our application, we need to compile binaries for the OpenWhisk runtime.

Manually running the Swift build command before each deployment is cumbersome and error-prone.

Let’s automate this processโ€ฆ

Using NPM’s scripts feature, the project exports a new command npm run-script compile which triggers the build process using the OpenWhisk Docker runtime for Swift.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "name": "openwhisk-swift-package-with-precompiled-binaries",
  "version": "1.0.0",
  "description": "Swift packages and pre-compiled binaries on OpenWhisk.",
  "main": "handler.js",
  "scripts": {
    "postinstall": "npm link serverless-openwhisk",
    "compile": "docker run --rm -it -v $(pwd):/swift-package openwhisk/action-swift-v3.1.1 bash -e -c 'cd /swift-package && swift build -v -c release'"
  },
  "keywords": [
    "serverless",
    "openwhisk"
  ],
  "dependencies": {
    "serverless-plugin-scripts": "^1.0.2"
  }
}

The serverless-plugin-scripts plugin provides a mechanism for running shell commands when framework commands are executed. Using this plugin we can use the package:initialize event to execute the npm run-script compile command.

1
2
3
4
custom:
  scripts:
    hooks:
      'package:initialize': npm run-script compile

The package:initialize event is fired when the serverless deploy command executes.

Swift binaries will be compiled prior to deployment without any manual steps from the developer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ serverless deploy
> openwhisk-swift-package-with-precompiled-binaries@1.0.0 compile /Users/james/code/bluemix/serverless-examples/openwhisk-swift-precompiled-binaries
> docker run --rm -it -v $(pwd):/swift-package openwhisk/action-swift-v3.1.1 bash -e -c 'cd /swift-package && swift build -v -c release'
...
Serverless: Packaging service...
Serverless: Compiling Functions...
Serverless: Compiling API Gateway definitions...
Serverless: Compiling Rules...
Serverless: Compiling Triggers & Feeds...
Serverless: Deploying Functions...
Serverless: Deployment successful!

Service Information
platform:   openwhisk.ng.bluemix.net
namespace:  _
service:    swift-packages

actions:
swift-packages-dev-hello    swift-packages-dev-welcome
...
$ serverless invoke -f hello
{
    "greeting": "Hello stranger!"
}
$ serverless invoke -f welcome
{
    "greeting": "Welcome stranger!"
}

Conclusion

OpenWhisk supports creating Swift actions from source files and pre-compiled binaries. Using binaries reduces the startup time for “cold” environments. This is important for latency sensitive applications like API endpoints.

Swift binaries for OpenWhisk must be compiled for the correct architecture and support execution through the platform runtime. Previous instruction for producing these binaries involved numerous manual and error-prone steps.

This process has now been improved through a new Swift package which wraps the runtime handler source files. Adding this dependency into the package manifest file means the downloading, compiling and linking of these source files will be handled by the Swift package manager.

Recent updates to the OpenWhisk provider plugin for The Serverless Framework also added support for pre-compiled Swift binaries. Combined with other plugins, the framework can now completely automate the process of building binaries for the Swift runtime.

Building binaries for Swift OpenWhisk actions has never been easier! ๐Ÿ˜Ž

Serverless Swift With OpenWhisk

Swift is one of the fastest growing programming languages with developers.

Swift has reached a Top 15 ranking faster than any other language we have tracked.

Created for building mobile applications, the language is now popular with backend development.

But for Swift developers beginning to build backend applications, they now find themselves having to manage computing infrastructure to run their applications in the cloud.

Enter serverless cloud platformsโ€ฆ โ˜๏ธโ˜๏ธโ˜๏ธ

These services allow developers to push code, rather than VMs, into the cloud. The platforms allow you to connect external event sources like API requests or message queues to functions in your code. As events occur, your code is instantiated and executed to process each request. Developers are only billed for the milliseconds needed to process each request.

Serverless platforms let you run applications in the cloud without worrying about infrastructure. ๐Ÿ˜Ž

Apache OpenWhisk is currently the only serverless platform to support Swift language functions.

Let’s have a look at how you can use Swift with OpenWhisk before diving into how the platform implements this feature to give us some tips and tricks for Swift on OpenWhiskโ€ฆ

Swift On OpenWhisk

Using the CLI

Create a Swift file with the following source code in.

1
2
3
4
5
6
7
func main(args: [String:Any]) -> [String:Any] {
    if let name = args["name"] as? String {
        return [ "greeting" : "Hello \(name)!" ]
    } else {
        return [ "greeting" : "Hello stranger!" ]
    }
}

Swift actions must consume and return a dictionary. The dictionary passed as the function argument will contain event parameters. Returned dictionary values must support serialisation to JSON.

Create and invoke a new OpenWhisk action using the command-line utility.

1
2
3
4
5
6
7
8
9
10
$ wsk action create swift action.swift
ok: created action swift
$ wsk action invoke swift --result
{
    "greeting": "Hello stranger!"
}
$ wsk action invoke swift --result --param name World
{
    "greeting": "Hello World!"
}

The result flag will only show the action output in the console rather than the full API response.

The source file must have a function called main. Each invocation executes this function. The function name to invoke can be overridden as shown below.

1
2
3
func foo(args: [String:Any]) -> [String:Any] {
    return [ "greeting" : "Hello foo!" ]
}
1
2
3
4
5
6
$ wsk action create foobar action.swift --main foo
ok: created action foobar
$ wsk action invoke foobar --result
{
    "greeting": "Hello foo!"
}

Choosing the runtime for the action can be set using the kind flag. If the source file has the .swift extension this will be automatically set to swift:default.

OpenWhisk uses Swift 3.0.2 that runs on the Linux environment. There are open issues to support Swift 3.1 and Swift 4.

Using the Serverless Framework

The Serverless Framework is a popular open-source framework for building serverless applications. It provides CLI tools and a workflow for managing serverless development.

Developers use a YAML file to define their application functions, events and resources. The framework handles deploying the application to their serverless provider.

Having started as a tool for AWS Lambda, the framework recently added multi-provider support. It now also works with Apache OpenWhisk, Azure Functions and Google Cloud Functions.

Let’s look at an example of using this framework to create a new OpenWhisk Swift application. Using a provider name and runtime, the framework can scaffold a new serverless application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ serverless create -t openwhisk-swift -p swift-action
Serverless: Generating boilerplate...
Serverless: Generating boilerplate in "/home/me/swift-action"
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v1.16.0
 -------'

Serverless: Successfully generated boilerplate for template: "openwhisk-swift"
$ tree swift-action/
swift-action/
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ package.json
โ”œโ”€โ”€ ping.swift
โ””โ”€โ”€ serverless.yml

0 directories, 4 files

The openwhisk-swift directory contains the boilerplate application ready to deploy. It includes a sample action (ping.swift) and the configuration file (serverless.yml).

1
2
3
4
5
6
7
8
9
10
11
func main(args: [String:Any]) -> [String:Any] {
    let formatter = DateFormatter()
    formatter.dateFormat = "yyyy-MM-dd HH:mm:ss"
    let now = formatter.string(from: Date())

    if let name = args["name"] as? String {
      return [ "greeting" : "Hello \(name)! The time is \(now)" ]
    } else {
      return [ "greeting" : "Hello stranger! The time is \(now)" ]
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
service: swift-action

provider:
  name: openwhisk
  runtime: swift

functions:
  hello:
    handler: ping.main

plugins:
  - serverless-openwhisk

Install the provider plugin using npm install and type serverless deploy to deploy this application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ serverless deploy
Serverless: Packaging service...
Serverless: Compiling Functions...
Serverless: Compiling API Gateway definitions...
Serverless: Compiling Rules...
Serverless: Compiling Triggers & Feeds...
Serverless: Deploying Functions...
Serverless: Deployment successful!

Service Information
platform: openwhisk.ng.bluemix.net
namespace:    _
service:  swift-action

actions:
swift-action-dev-hello
...
$ serverless invoke -f hello
{
    "greeting": "Hello stranger! The time is 2017-06-23 10:52:02"
}

For more information on using the Serverless Framework with OpenWhisk, please see this documentation: https://serverless.com/framework/docs/providers/openwhisk/.

How It Works

Swift is a statically typed compiled language. Unlike JavaScript or Python, Swift source code must be compiled into a binary for execution.

Swift actions in OpenWhisk can be created from Swift source files, rather than binaries, meaning the platform must run this compilation step.

Swift on Docker

OpenWhisk uses Docker containers to manage the action runtime environments. This Dockerfile documents the build steps for generating the Swift runtime image used in OpenWhisk.

Images for each of the OpenWhisk runtime environments are available on Docker Hub. Creating containers from these images allows you to explore the Swift runtime environment.

1
2
$ docker pull openwhisk/swift3action
$ docker run -it --rm openwhisk/swift3action bash

For more information on the API exposed by runtime containers to initialise and invoke actions, please see this blog post.

Building Swift actions

Swift runtime environments has a template package available in the /swift3Action/spm-build directory.

All the Swift sources files provided by the user are written into that package’s main.swift file. The following source code is appended to main.swift to support execution within the OpenWhisk runtime. It parses the input parameters from the environment, invokes the registered function name and returns the computation response as a JSON string.

Dependencies for the following packages are included in the existing Package.swift file. These packages can be used from the action source code without further configuration.

1
2
3
4
5
6
7
8
9
10
import PackageDescription

let package = Package(
    name: "Action",
        dependencies: [
          .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.0.1"),
            .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "14.2.0"),
            .Package(url: "https://github.com/IBM-Swift/swift-watson-sdk.git", "0.4.1")
        ]
)

During initialisation, the Swift build process is executed to generate the action binary.

This artifact (/swift3Action/spm-build/.build/release/Action) will be executed for each invocation received by the platform.

Container re-use

Containers used for action runtimes are re-used with subsequent requests. This means any initialisation cost, e.g. compiling Swift source code, will only be incurred once per runtime container.

Runtime containers are evicted from the cache ten minutes after the last activation. Future invocations for that runtime will use a new container and have to run the initialisation step again.

Additionally, runtimes containers cannot process concurrent requests. If a request arrives before the previous one has finished processing, a new environment will need to be initialised.

Improving cold start time

Swift build times are not known for being fast.

Build time is included in the request processing time for each new runtime container provisioned.

In an attempt to reduce this delay, OpenWhisk runs the minimum build steps necessary to compile the source code, rather than a full release build.

During the Docker build for the Swift runtime image, the full release build is executed for the empty action package. This generates object files and other intermediary build outputs which are stored in the build cache.

Logs from the build process are parsed to retrieve the individual compilation and linking commands for the main.swift file. These commands are written into a new shell script (/swift3Action/spm-build/swiftbuildandlink.sh).

When a new Swift runtime container is initialised, the source code for the action is written into the main.swift file. Rather than running a full re-build, the runtime just executes the shell script containing the compilation and linking steps. This re-uses the cached build objects and reduces compilation time.

Modifying package dependencies

Swift packages uses a manifest file (Packages.swift) to list package dependencies. Dependencies are automatically downloaded and compiling during the package build process.

The Swift environment used by OpenWhisk uses the package manifest shown above. This includes dependencies for JSON and HTTP libraries.

Swift actions can be created from Swift source code or zip files. Zip files are expanded into the package directory (/swift3action/spm-build) before initialisation.

If the zip file contains a new package manifest, this will overwrite the default manifest in the environment.

1
2
3
4
5
6
7
8
9
10
11
import PackageDescription

let package = Package(
    name: "Action",
        dependencies: [
          .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.0.1"),
            .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "14.2.0"),
            .Package(url: "https://github.com/IBM-Swift/swift-watson-sdk.git", "0.4.1"),
          .Package(url: "https://github.com/IBM-Swift/swift-html-entities", majorVersion: 3, minor: 0),
        ]
)

Running a full build will download new package dependencies and make them available for use in our action.

OpenWhisk uses a shell script (swiftbuildandlink.sh) to manage the build process during initialisation. This defaults to only running the compiler and linker commands for the main.swift file, rather than a full release build.

Including a replacement swiftbuildandlink.sh file in the zip file will allow us to modify the build command used, e.g. swift build -v -c release.

1
2
3
4
#!/bin/bash
echo "Release build running..."
swift build -v -c release
echo "Release build finished."

Downloading additional packages will add a significant delay to initialising new runtime containers.

If this is an issue, let’s look at skipping the compile step entirelyโ€ฆ

Compiling binaries locally

Swift actions execute a binary that is available at the following path: /swift3action/spm-build/.build/release/Action.

The runtime uses the existence of this binary to control running the build process. If the file does not exist, the build step is executed. It ensures that compilation is only ran once per runtime container.

This also means that developers can include a locally compiled Swift binary inside the action zip file. During initialisation, the existence of this file will stop the build process from running.

If you want to use lots of additional Swift packages, the compile time penalty won’t have to be incurred during action invocations. This will dramatically speed up invocation times for “cold” actions.

Binaries must be compatible with the platform environment they are being executed within. OpenWhisk uses Swift 3.0.2 on Linux.

OpenWhisk publishes the runtime environments as Docker images. Using containers from these images to compile our action binaries will ensure the binary is compatible.

These instructions show you how to compile your source code into a compatible platform binary.

1
2
3
4
5
6
7
8
9
10
11
12
13
# run an interactive Swift action container
docker run -it -v `pwd`:/ow openwhisk/swift3action bash
cd /ow
# now inside the docker shell
# copy the source code and prepare to build it
cat /swift3Action/epilogue.swift >> main.swift
echo '_run_main(mainFunction:main)' >> main.swift
# build and link (the expensive step)
swift build -v -c release
# create the zip archive
zip action.zip .build/release/Action
# exit the docker shell
exit

The action.zip file can then be deployed as a new action using the following command-line.

1
wsk action create static-swift action.zip --kind swift:3

Conclusion

Swift is one of the fastest growing programming languages with developers. People are increasingly using it to develop backend APIs and services. Being able to use Swift on serverless cloud platforms means developers can focus on writing code, rather than managing infrastructure.

Apache OpenWhisk, an open-source serverless platform, supports Swift as a first-class language. Developers can provide Swift source code and have the platform execute these functions in response to external events.

Because OpenWhisk is open-source, we can discover how the platform executes the code using the Swift runtime. Understanding this process allows us to modify the build step to use additional Swift packages within our actions. We can also improve performance by skipping the compilation stage entirely by providing a native binary.