NPM Modules in OpenWhisk

Nov 28, 2016
npm openwhisk serverless
6 min read

OpenWhisk now supports creating Node.js Actions from a zip file. The archive file will be extracted into the runtime environment by the platform. This allows us to split microservice logic across multiple files, use third-party NPM modules or include non-JavaScript assets (configuration files, images, HTML files).

“Hello World” Example

Let’s look at a “Hello World” example of registering a serverless function from a zip file. Our archive will contain two files, the package descriptor and a JavaScript file.

Here is the minimal package.json file required for loading a module from a directory.

{
  "main": "my_file.js"
}

In my_file.js, a function is returned through the main property on the exports object. This function implements the Action interface.

exports.main = function (params) {
  return {result: "Hello World"};
};

Creating a zip file from the current directory, we can deploy this Action through the command-line utility.

$ zip -r action.zip *
$ wsk action create hello_world --kind nodejs:default action.zip

When this Action is invoked, the archive will be unzipped into a temporary directory. OpenWhisk loads the directory as a Node.js module and invokes the function property on the module for each invocation.

$ wsk action invoke hello_world --result
{
    "result": "Hello world"
}

Using NPM Dependencies

Let’s look a more complicated example which uses an external NPM module in our Action.

const leftPad = require("left-pad")
    
function myAction(args) {
    const lines = args.lines || [];
    return { padded: lines.map(l => leftPad(l, 30, ".")) }
}

exports.main = myAction;

This module uses the extremely popular left-pad module to process an array of strings, passed through a request parameter. The resulting output is returned in the response.

Before using this module, we need to install the dependencies listed in package.json.

{
  "name": "my-action",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies" : {
    "left-pad" : "1.1.3"
  }
}

OpenWhisk does not automatically install dependencies listed in package.json in the runtime environment.

The developer has to run npm install locally and include the node_modules directory in the zip file.

Install NPM dependencies locally.

$ npm install

Create a .zip archive containing all files.

$ zip -r action.zip *

Create the action using command-line utility.

$ wsk action create packageAction --kind nodejs:default action.zip

Now we can test out our action to check it works….

$ wsk action invoke --blocking --result packageAction --param lines "[\"and now\", \"for something completely\", \"different\" ]"
{
    "padded": [
        ".......................and now",
        "......for something completely",
        ".....................different"
    ]
}

Native Module Dependencies

Node.js provides a mechanism for JavaScript modules to include native platform code as if they were ordinary modules. This is often used to improve performance by deferring operations to native C/C++ libraries. NPM handles compiling native code during the dependency install process.

Using modules with native dependencies in Actions requires the native code to be compiled for the platform runtime.

Compiling dependencies with Docker

One solution to this problem uses Docker to simulate the same runtime environment.

OpenWhisk uses Docker to manage the runtime environments for Actions. The buildpack-deps:trusty-curl image is used as the base image for all Node.js Actions.

Running a local container from this image will give access to the same runtime environment. Running npm install within this container will produce the node_modules directory with native code compiled for the correct architecture.

Action With Native Modules

Let’s look at an example…

const SHA3 = require('sha3');

function SHA(args) {
  const d = new SHA3.SHA3Hash();
  d.update(args.payload);
  return { sha: d.digest('hex') };
}

exports.main = SHA;

This module returns a function that calculates a SHA3 cryptographic hash for the invocation payload. The hex string for the hash is returned as the function response.

The NPM module (sha3) used to calculate the digest uses a C++ extension for the hashing algorithm.

{
  "name": "hashing-service",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "sha3": "^1.2.0"
  }
}

Action Runtime Environments

OpenWhisk uses a public Docker image as the base image for the Action environments. It then builds a custom image by installing Node.js and NPM for the particular runtime version.

Rather than building this image ourselves, we can use existing images published on Docker Hub.

NodeSource provides public Docker images pre-installed with different Node.js versions. Provided the base image (Ubuntu Trusty) and Node.js version (6.7) matches, the runtime environment will be the same.

Starting a local container from this image, we can use Docker’s host volume support to mount the local directory into the host container.

$ docker run -it -v "/action:/usr/src/app" nodesource/trusty:6.7 /bin/sh

Running npm install in the container, the sha3 dependency is compiled and installed.

# npm install
                                                                                                                      
> sha3@1.2.0 install /usr/src/app/node_modules/sha3                                                                   
> node-gyp rebuild                                                                                                    
                                                                                                                      
make: Entering directory `/usr/src/app/node_modules/sha3/build'                                                       
make: Warning: File `sha3.target.mk' has modification time 0.19 s in the future                                       
  CXX(target) Release/obj.target/sha3/src/addon.o                                                                     
  CXX(target) Release/obj.target/sha3/src/displayIntermediateValues.o                                                 
  CXX(target) Release/obj.target/sha3/src/KeccakF-1600-reference.o                                                    
  CXX(target) Release/obj.target/sha3/src/KeccakNISTInterface.o                                                       
  CXX(target) Release/obj.target/sha3/src/KeccakSponge.o                                                              
  SOLINK_MODULE(target) Release/obj.target/sha3.node                                                                  
  COPY Release/sha3.node                                                                                              
make: warning:  Clock skew detected.  Your build may be incomplete.                                                   
make: Leaving directory `/usr/src/app/node_modules/sha3/build'                                                        
my-action@1.0.0 /usr/src/app                                                                                          
`-- sha3@1.2.0                                                                                                        
  `-- nan@2.4.0

The node_modules directory will be available on the host system after exiting the container. Repeat the steps above to archive the source files and deploy our serverless function.

$ zip -r action.zip *
$ wsk action create packageAction --kind nodejs:6 action.zip  
ok: created action packageAction

Invoking the Action will now use the native code to produce hash values for the invocation parameters.

$ wsk action invoke packageAction -b -p payload "Hello" --result                              
{                                                                                                                     
    "sha": "c33fede18a1ae53ddb8663710f8054866beb714044fce759790459996196f101d94dfc7bd8268577f7ee3d2f8ff0cef4004a963222
7db84df62d2b40682d69e2"                                                                                               
}

Action Package Details

Upon invocation, OpenWhisk extracts the action’s zip file to a temporary directory in the runtime environment. It then loads the directory as a standard Node.js module, using require.

Node.js expects the directory to have a valid package.json file. The main property is used to define which JavaScript file is evaluated when the module is loaded. This file can assign values to the global exports object. These references are then returned when require is called for this module.

OpenWhisk expects the returned module object to have a property called main which references a function. This function will be executed for each invocation request.

Request parameters are passed as object properties on the first function argument. The function must return an object for the invocation response.

Other files included in the archive will be available in the current working directory. These can also be loaded as modules or read directly from the file-system.

Conclusions

OpenWhisk support for Action packages is a huge step forward for the platform. Node.js has an enormous ecosystem of third-party modules. Developers can now easily use any of these modules within their Actions.

This feature can also be used to include non-JS files within the runtime environment. It would be possible to use configuration files in JSON or static assets like HTML or CSS files.

The team are now working on providing support for other runtimes, watch this space…