James Thomas

Notes on JavaScript

Updated IBM Watson Nodes for Node-RED

Earlier this year, I made a major upate to the Node-RED nodes for the IBM Watson services available through IBM Bluemix. Since then, the IBM Watson team has been busy, with lots of changes to APIs. I’ve recently been working through these changes, updating the nodes, to ensure they work against the latest APIs.

Updates to these nodes have now been finished and are available through the boilerplate on IBM Bluemix or by installing the IBM Bluemix Nodes package locally.

If you have an existing Node-RED instance running in IBM Bluemix, please review the documentation for upgrade instructions.

If you encounter any issues, please raise a issue on Github.

For full details on the changes are available in the pull request.

NPM Modules in Node-RED

Before Christmas, my department at IBM had one of our semi-regular Hack Days to get everyone together and work on interesting ideas away from the day job. I spent the time playing with an idea to make exposing NPM packages in Node-RED easier…

Node-RED is a visual tool for wiring the Internet of Things.

It makes it easy to create, combine and control data flowing between hardware devices, web APIs and open protocols. The tool exposes operations through a series of nodes, which can be created through the browser-based editor and connected to other nodes to create message flows.

The tool comes built-in with a huge selection of nodes, from connecting to a Raspberry Pi to handling HTTP requests, for creating flows. Users create their own nodes to expose new functionality, by creating small modules using JavaScript and HTML. People often wrap existing NPM modules into custom Node-RED nodes to use that module functionality in flows.

NPM has over two hundred thousand modules, many of which provide simple “stateless” functions that return a result based upon the input data. These modules are ideally suited for using within Node-RED message flows.

Exposing multiple new NPM modules required creating custom Node-RED nodes for every module you wanted to use. Whilst the Node-RED node boilerplate is small, it becomes a bit laborious and repetitive to keep doing this for extremely simple modules.

Wondering how to make this easier, I started hacking on ideas.

Two days later…

Node-RED node to dynamically expose NPM modules as nodes.

This new node allows you to dynamically expose NPM modules as Node-RED nodes without needing to manually create new nodes for them.

Using the node editor panel, the user can set the name of the NPM package to expose along with the module invocation style.

Incoming flow messages are passed as an argument to the function being executed. Users can set up the node to call the module, a module function or even run custom setup code in response to incoming messages.

Execution results, either returned directly or asynchronously (Promises and callback-style supported), are sent as the outgoing message payload.

When the flow is deployed, the NPM module will be automatically installed and instantiated.

Now we don’t have to write lots of extra boilerplate code every time we want to use a tiny NPM module in a Node-RED flow, hurrah!

Here’s a short demonstration using the node to expose the sentiment package in a flow:

Node.js V4 in Cloud Foundry

Last week, Node.js released the latest version of their project, v4.0.0. This release, representing the convergence of io.js with the original Node.js project, came with lots of exciting features like improved ES6 support.

Cloud Foundry already supports multiple versions of the Node.js runtime. Developers select the desired runtime version using a parameter in their application’s package descriptor.

So, we just update package.json to include “4.0.0” and re-deploy our application?

Not yet.

There is an unresolved technical issue delaying the release of “official” Node.js v4 support for the platform. 😿

Can we add support ourselves?

Yes!

To do this, we need to explore how Cloud Foundry configures the runtime environment for applications.

Buildpacks

Rather than hardcoding supported runtimes and frameworks into the platform, Cloud Foundry borrowed the buildpack model from Heroku. Buildpacks are a set of scripts, run by the platform during deployment, to configure the runtime environment.

Users can set an explicit buildpack for an application, using the manifest, or let the platform decide. Buildpacks for common runtimes are pre-installed with the platform. Buildpacks set through the manifest can point to external URLs, allowing users to create new buildpacks supporting custom runtimes.

Each buildpack must contain the following files as executable scripts.

  • bin/detect - determine whether a buildpack is suitable for an application.
  • bin/compile - install and configure the runtime environment on the DEA.
  • bin/release - provide metadata with information on executing application.

Full details on existing buildpacks for the platform are available here.

Node.js is supported as an “official” buildpack by the platform. This will be the one we will modify to add support for the latest version of the runtime.

Node.js Buildpack

This is the Node.js buildpack for Cloud Foundry. Applications using this buildpack can select the version of Node.js to install using the engine parameter in the package descriptor.

Looking at the bin/compile script will show us how the Node.js runtime is installed during deployment.

This snippet handles accessing the Node.js version configured, using the node.engine parameter from package.json, before calling install_nodejs to install the correct runtime package.

install_bins() {
  local node_engine=$(read_json "$BUILD_DIR/package.json" ".engines.node")
  local npm_engine=$(read_json "$BUILD_DIR/package.json" ".engines.npm")

  echo "engines.node (package.json):  ${node_engine:-unspecified}"
  echo "engines.npm (package.json):   ${npm_engine:-unspecified (use default)}"
  echo ""

  warn_node_engine "$node_engine"
  install_nodejs "$node_engine" "$BUILD_DIR/.heroku/node"
  install_npm "$npm_engine" "$BUILD_DIR/.heroku/node"
  warn_old_npm
}

Searching through the buildpack for this function, it’s in the lib/binaries.sh file. Looking at the function code, it translates the version number into a URL pointing to an archive with the pre-compiled Node.js binary. This archive file is downloaded, extracted and installed into the runtime environment.

Translating Node.js version identifiers into archive URLs uses a special file in the buildpack, manifest.yml. This file maps every supported version to a pre-built binary location.

Looking at previous commits to the Node.js buildpack, adding support for additional versions of Node.js simply requires updating this file with the extra version identifier and archive URL.

Until the Cloud Foundry team updates the buildpack to support Node.js v4, they won’t provide an external archive containing the pre-built runtime environment.

Where can we find a suitable build of the Node.js binary?

Node.js Runtime Binaries

Cloud Foundry borrowed the buildpack concept from Heroku and still maintains backwards compatibility with their platform. Heroku buildpacks will work with Cloud Foundry applications. The Node.js buildpack for Cloud Foundry is actually still a fork of Heroku’s.

Looking back through the original buildpack source, this URL template is used to translate Node.js versions to archive URLs being built by Heroku.

http://s3pository.heroku.com/node/v$version/node-v$version-$os-$cpu.tar.gz

Combining the correct version identifier and platform parameters with this template gave the following location for a potential build of the Node.js v4 runtime.

http://s3pository.heroku.com/node/v4.0.0/node-v4.0.0-linux-x64.tar.gz

Running curl against the location successfully downloaded the Node.js v4 binary archive!

Custom v4 Buildpack

Forking the Cloud Foundry Node.js buildpack on Github, we can update the manifest.yml with the Node.js v4 identifier pointing to the Heroku runtime archive. This external Git repository will be used as the buildpack identifier in the application manfest.

Deploying with v4

Having updated our application manifest with the custom buildpack location and set the updated node version flag, re-deploying our application will start it running on Node.js v4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[20:02:29 ~]$ cf app sample-demo-app
Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.mybluemix.net
last uploaded: Fri Sep 18 18:33:56 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-18 07:35:01 PM   0.0%   65.3M of 256M   59M of 1G
[20:03:13 ~]$

Looking at the logs from the deployment we can see the latest Node.js runtime has been downloaded and installed within our runtime environment.

Conclusion

Buildpacks are a brilliant feature of Cloud Foundry.

Understanding how buildpacks are structured and used by the platform means we can start customising existing buildpacks and even start creating our own.

If you want to run Node.js applications using v4 on Cloud Foundry today, you can use the following buildpack created using the instructions above.

Cloud Foundry is currently adding support for the version to the official buildpack, follow their progress here.

Location-Based Cloud Foundry Applications Using Nginx and Docker

Routing application traffic based upon the geographic location of incoming requests can be used for a number of scenarios…

  • Restricting access to your application outside defined geographic regions.
  • Load-balancing traffic to the closest region for improved performance.
  • Providing custom applications for different countries.

IBM Bluemix allows deploying applications to different geographic regions through hosting instances of the Cloud Foundry platform in multiple locations.

Cloud Foundry supports simple HTTP routing rules for deployed applications. Organisations can register domains and routes for applications. Routes can be bound to one or more deployed applications. Incoming HTTP traffic is load-balanced, using the Round-Robin policy, between the application instances bound to a route.

However, the platform does not currently support traffic routing based upon the geographic location of incoming requests or sharing domains and routes between regions.

So, say we want to deploy custom versions of an application to different regions and automatically forward users to the correct version based upon their location. How can we achieve this?

Let’s find out…

Deploying Application To Different Regions

IBM Bluemix currently provides Cloud Foundry in two regions for deploying applications.

  • US South (api.ng.bluemix.net)
  • Europe (api.eu-gb.bluemix.net)

Moving between regions is as simple as providing the different region endpoint during the authentication command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[16:25:47 ~]$ cf login -a api.ng.bluemix.net -u james.thomas@uk.ibm.com -s dev
API endpoint: api.ng.bluemix.net

Password>
Authenticating...
OK

Targeted org james.thomas@uk.ibm.com

Targeted space dev

API endpoint:   https://api.ng.bluemix.net (API version: 2.27.0)
User:           james.thomas@uk.ibm.com
Org:            james.thomas@uk.ibm.com
Space:          dev
[16:26:44 ~]$

We’re now authenticated against the US South region.

Let’s start by deploying our sample application, which displays a web page showing the application URL, to this region.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[16:44:24 ~/code/sample]$ cf api
API endpoint: https://api.ng.bluemix.net (API version: 2.27.0)
[16:44:32 ~/code/sample]$ cf push sample-demo-app
Using manifest file /Users/james/code/sample/manifest.yml

Updating app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

...

Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.mybluemix.net
last uploaded: Fri Sep 11 15:45:04 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-11 04:46:00 PM   0.0%   67.1M of 256M   59M of 1G
[16:45:14 ~/code/sample]$ 

Once that has finished, we can move over to the European region and deploy our application there.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[16:52:33 ~/code/sample]$ cf login -a api.eu-gb.bluemix.net -u james.thomas@uk.ibm.com -s dev
[16:52:58 ~/code/sample]$ cf push sample-demo-app
Using manifest file /Users/james/code/sample/manifest.yml

Updating app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

...

Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.eu-gb.mybluemix.net
last uploaded: Fri Sep 11 15:53:31 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-11 04:54:17 PM   0.0%   67.4M of 256M   59M of 1G
[16:54:25 ~/code/bluemix/sample]$

With the second deployment completed, there are now instances of the same application running in separate regions.

Each instance is available through a separate URL.

Now we need to set up traffic forwarding from the relevant locations to the correct region.

Reverse Proxy with Region Traffic Forwarding

Due to the platform not supporting multi-region traffic routing, we need to set up a custom reverse proxy. This server will receive requests from our external application domain and transparently forward them onto the correct region application.

We’re going to use Nginx.

Nginx (pronounced engine-x) is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server

Nginx comes with a module for looking up locations associated with IP address using the MaxMind GeoIP library. The module can resolve incoming request addresses into continents, countries and even cities. Using the variables defined by the module, we can write traffic forwarding rules to send requests to the correct region.

Nginx Configuration

Nginx defines two configuration directives, geoip_country and geoip_city, to specify locations for the MaxMind GeoIP database files.

http { 
    ...
    geoip_country /usr/share/GeoIP/GeoIP.dat;
    geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
    ...
}

When configured, Nginx will expose a series of variables for each request with geographical information.

  • $geoip_country_code - two-letter country code, for example, “RU”, “US”.
  • $geoip_country_name - country name, for example, “Russian Federation”, “United States”.
  • $geoip_city_continent_code - two-letter continent code, for example, “EU”, “NA”.
  • $geoip_city - city name, for example, “Moscow”, “Washington”.

Starting with the default nginx configuration, there are only a few modifications needed to set up a reverse proxy based upon location.

For each request, we check the $geoip_city_continent_code against our list of regions. If the request is valid, setting the proxy_pass directive forwards the request onto the correct region. We also overwrite the Host: HTTP header with the region URL. IBM Bluemix uses this header to internally route incoming requests to the correct application host.

Requests coming from outside these locations will be sent to a custom error page.

Due to a known issue with IBM Containers, we must use IP addresses rather than the host names with the proxy_pass directive.

Here is the full configuration for the enabled-site/default file.

server {
  listen 80 default_server;
  listen [::]:80 default_server ipv6only=on;

  root /usr/share/nginx/html;
  index index.html index.htm;
  error_page 404 /404.html;

# Make site accessible from http://localhost/
  server_name localhost;

  location = /404.html {
    internal;
  }

  location / {
    set $host_header "unknown";

    if ($geoip_city_continent_code = "EU") { 
      proxy_pass http://5.10.124.141;
      set $host_header "sample-demo-app.eu-gb.mybluemix.net";
    }

    if ($geoip_city_continent_code = "NA") { 
      proxy_pass http://75.126.81.66;
      set $host_header "sample-demo-app.mybluemix.net";
    }

    if ($host_header = "unknown") {
      return 404;
    }

    proxy_set_header Host $host_header;
  }
}

With the reverse proxy server configured, we need to provision a new production server, install Linux and Nginx, configure networking, security updates and backup services…

…or we can use Docker.

Running Nginx using Docker

There are thousands of repositories on Docker Hub providing Nginx, including the official image. Unfortunately, the official image provides a version of Nginx that is not built with the geo_ip module.

Ubuntu’s default package repository for Nginx does provide a build including the geo_ip module. By modifying the Dockerfile for the official image, we can build a new image from Ubuntu with the required version of Nginx and include our custom configuration files.

FROM ubuntu
RUN apt-get -y install nginx

# copy custom configuration
COPY nginx.conf /etc/nginx/nginx.conf
COPY default /etc/nginx/sites-available/
COPY geoip /etc/nginx/geoip
COPY 404.html /usr/share/nginx/html/

# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log

# expose HTTP and HTTP ports
EXPOSE 80 443

CMD ["nginx", "-g", "daemon off;"]

Building and running this container locally, we can test that Nginx is configured correctly. The repository containing the Dockerfile and build artificats is located here.

1
2
3
4
5
6
7
8
9
10
[16:58:40 ~/code/final]$ docker build -t geo_ip .
Sending build context to Docker daemon 15.88 MB
Step 0 : FROM ubuntu
 ---> 91e54dfb1179
...
Step 9 : CMD nginx -g daemon off;
 ---> Using cache
 ---> 7bb6dbaafe3e
Successfully built 7bb6dbaafe3e
[16:58:50 ~/code/final]$ docker run -Pti geo_ip

With the custom image ready, we just need to deploy it somewhere…

Running Nginx on IBM Containers

IBM Bluemix supports deploying Docker containers alongside Cloud Foundry applications, allowing us to use the same cloud platform for running our custom region applications as providing the reverse proxy

Pushing pre-built images to the IBM Containers service is really as simple as creating a new tag and typing docker push.

Please read and follow the documentation about installing the command-line container management tools and authenticating with the remote service before attempting the commands below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[14:10:52 ~]$ docker tag geo_ip registry.ng.bluemix.net/jthomas/geo_ip
[14:10:59 ~]$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
geo_ip                                   latest              7bb6dbaafe3e        3 days ago          222.3 MB
registry.ng.bluemix.net/jthomas/geo_ip   latest              7bb6dbaafe3e        3 days ago          222.3 MB
[14:11:07 ~]$ cf ic login
** Retrieving client certificates from IBM Containers
** Storing client certificates in /Users/james/.ice/certs
Successfully retrieved client certificates
** Authenticating with registry at registry.eu-gb.bluemix.net
Successfully authenticated with registry
[14:24:25 ~]$ docker push registry.ng.bluemix.net/jthomas/geo_ip
The push refers to a repository [registry.ng.bluemix.net/jthomas/geo_ip] (len: 1)
Sending image list
Pushing repository registry.ng.bluemix.net/jthomas/geo_ip (1 tags)
...
Pushing tag for rev [7bb6dbaafe3e] on {https://registry.ng.bluemix.net/v1/repositories/jthomas/geo_ip/tags/latest}
[14:25:39 ~]$ cf ic images
REPOSITORY                                        TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
registry.ng.bluemix.net/jthomas/geo_ip            latest              7b1865be-778        About a minute ago   0 B
registry.ng.bluemix.net/ibmliberty                latest              2209a9732f35        3 weeks ago          263.6 MB
registry.ng.bluemix.net/ibmnode                   latest              8f962f6afc9a        3 weeks ago          178.9 MB
registry.ng.bluemix.net/ibm-mobilefirst-starter   latest              97513e56aaa7        3 weeks ago          464.9 MB
[14:26:43 ~]$ 

We can now use the IBM Bluemix dashboard to start a new container from our custom image, binding a public IP address and exposing ports.

Once the container starts, accessing the bound IP address shows the web page coming back with the region-specific application route.

Using DNS A records, we can now map our external URL to the IP address of the container. Users visiting this URL will be sent to the reverse proxy server which will then forward the request onto the correct region application.

Testing it all out…

Testing out the forwarding rules requires us to send HTTP requests from multiple regions. GeoWebView will run web browsers located in different geographies and show you the rendered page output.

Running the tool with our application’s web address, shows the following rendered page images.

We can see the browsers from the United States and Europe are sent to the correct region. The browser from South Africa is shown the custom error page.

Using Nginx we’ve configured a reverse proxy to route users, based upon their location, to applications running in different IBM Bluemix regions. We’re hosting the service on the same platform as our applications, using Docker. Most importantly, the whole process is transparent to the user, they aren’t forced to visit country-specific URLs.

Success!

Running One-off Tasks in Cloud Foundry

Whether making changes to a database schema, bulk importing data to initialise a database or setting up a connected service, there are often administrative tasks that needed to be carried out for an application to run correctly.

These tasks usually need finishing before starting the application and should not be executed more than once.

Previously, the CF CLI provided commands, tunnel and console, to help running one-off tasks manually. These commands were deprecated with the upgrade from v5 to v6 to discourage snowflake environments.

It is still possible, with a bit of hacking, to run one-off tasks manually from the application container.

A better way is to describe tasks as code and run them automatically during normal deployments. This results in applications that can be recreated without manual intervention.

We’ll look at both options before introducing a new library, oneoff, that automates running administration tasks for Node.js applications.

Running Tasks Manually

Local Environment

Rather than running administrative tasks from the application console, we can run them from a local development environment by remotely connecting to the bound services.

This will be dependent on the provisioned services allowing remote access. Many “built-in” platform services, e.g. MySQL, Redis, do not allow this.

Third-party services generally do.

Using the cf env command we can list service credentials for an application. These authentication details can often be used locally by connecting through a client library running in a local development environment.

For example, to access a provisioned Cloudant instance locally, we can grab the credentials and use with a Node.js client library.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[15:48:22 ~/code/sample]$ cf env sample-demo-app
Getting env variables for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

System-Provided:
{
 "VCAP_SERVICES": {
  "cloudantNoSQLDB": [
   {
    "credentials": {
     "host": "1234-bluemix.cloudant.com",
     "password": "sample_password",
     "port": 443,
     "url": "https://1234-bluemix:sample_password@1234-bluemix.cloudant.com",
     "username": "1234-bluemix"
    }
....

[15:48:22 ~/code/sample]$ cat connect.js
var Cloudant = require('cloudant');

var me = '1234-bluemix';
var password = 'sample_password';

// Initialize the library with my account.
var cloudant = Cloudant({account:me, password:password});

cloudant.db.list(function(err, allDbs) {
  console.log('All my databases: %s', allDbs.join(', '))
  // Run administrative tasks
});
[15:48:22 ~/code/sample]$ node connect.js
All my databases: example_db, jasons_stuff, scores

Remote Environment

When provisioned services don’t allow external access, the cf-ssh project creates SSH access to application containers running within Cloud Foundry.

How does this work?!

cf-ssh deploys a new Cloud Foundry application, containing the same bits as your target application, with the same bound services.
This new application’s container does not start your web application as per normal. Instead, it starts an outbound reverse SSH tunnel to a public proxy.
The local cf-ssh client then launches an interactive ssh connect to the public proxy, which tunnels through to the application container.

See the explanation here for full details.

This approach will let you connect to services from within the Cloud Foundry platform environment.

This video from Stark & Wayne’s Dr. Nic shows the command in action…

IBM Bluemix Console (Java and Node.js)

This technique is only for the IBM Bluemix platform.

If you are deploying Node.js and Java applications on IBM Bluemix, the platform provides the following tools to assist with application management.

  • proxy: Minimal application management that serves as a proxy between your application and Bluemix.
  • devconsole: Enables the development console utility.
  • shell: Enables a web-based shell.
  • trace: (Node.js only) Dynamically set trace levels if your application is using log4js, ibmbluemix, or bunyan logging modules.
  • inspector: (Node.js only) Enables node inspector debugger.
  • debug: (Liberty only) Enables clients to establish a remote debugging session with the application.
  • jmx: (Liberty only) Enables the JMX REST Connector to allow connections from remote JMX clients

The tools are enabled by setting the environment variable (BLUEMIX_APP_MGMT_ENABLE) with the desired utilities.

1
$ cf set-env myApp BLUEMIX_APP_MGMT_ENABLE devconsole+shell+trace

Applications must be restarted for the changes to take effect.

If we enable the shell utility, the following web-based console will be available at https://your-app-name.mybluemix.net/bluemix-debug/shell.

Cloud Foundry Diego Runtime

Diego is the next-generation runtime that will power upcoming versions of Cloud Foundry. Diego will provide many benefits over the existing runtime, e.g. Docker support, including enabling SSH access to containers without the workarounds needed above.

Yay!

Follow the instructions here for details on SSH access to applications running on the new runtime.

Access to this feature will be dependent on your Cloud Foundry provider migrating to the new runtime.

Running Tasks Automatically

Manually running one-off administrative tasks for Cloud Foundry applications is a bad idea.

It affects your ability to do continuous delivery and encourages snowflake environments.

Alternatively, defining tasks as code means they can run automatically during normal deployments. No more manual steps are required to deploy applications.

There are many different libraries for every language to help you programmatically define, manage and run tasks.

With tasks defined as code, you need to configure your application manifest to run these automatically during deployments.

Cloud Foundry uses the command parameter, set in the manifest or through the command-line, to allow applications to specify a custom start command. We can use this parameter to execute the task library command during deployment.

The Cloud Foundry documentation also details these approaches, with slightly different implementations here and specifically for Ruby developers here.

Temporary Task Deploy

For applications which only need occasional administrative tasks, it’s often easier to push a temporary deploy with a custom start command. This deploy runs your tasks without then starting your application. Once the tasks have completed, redeploy your application normally, destroying the task instance.

The following command will deploy a temporary instance for this purpose:

1
$ cf push -c 'YOUR_TASK_LIB_COMMAND && sleep infinity' -i 1 --no-route

We’re overriding the default start command, setting it to run the command for our task library, e.g. rake db:migrate.

The sleep infinity command stops the application exiting once the task runner has finished. If this happens, the platform will assume that application has crashed and restart it.

Also, the task runner will not be binding to a port so we need to use the –no-route argument to stop the platform assuming the deploy has timed out.

Setting the deploy to a single instance stops the command being executed more than once.

Checking the logs to verify the task runner has finished correctly, we can now redeploy our application. Using the null start command will force the platform to use the buildpack default rather than our previous option.

1
$ cf push -c 'null'

Running Tasks Before Startup

If we’re regularly running administrative tasks, we should incorporate the task execution into our normal application startup. Once the task command has finished successfully, we start the application as normal.

Applications may have multiple instances running, we need to ensure the tasks are only executed by one instance.

The following custom start command will execute tasks during startup, using the CF_INSTANCE_ID environment variable to enforce execution at most-once.

[ $CF_INSTANCE_INDEX -eq 0 ]] && node lib/tasks/runner.js; node app.js

With this approach, tasks will be automatically executed during regular deployments without any manual intervention.

Hurrah!

Managing tasks for Node.js applications

If you’re running Node.js applications on Cloud Foundry, oneoff is a task library that helps you define tasks as code and integrates with the Cloud Foundry runtime. The module handles all the complexities with automating tasks during deployments across multi-instance applications.

oneoff provides the following features…

* ensure tasks are completed before application startup
* coordinating app instances to ensure at-most once task execution
* automagically discovering tasks from the task directory
* dependency ordering, ensure task a completes before task b starts
* parallel task execution
* ignore completed tasks in future deployments

Check it out to help make writing tasks as code for Node.js applications much easier!

Full details on usage are available in the README.

Conclusion

Running one-off tasks for application configuration is a normal part of any development project.

Carrying out these tasks manually used to be the norm, but with the devops movement we now prefer automated configuration rather manual intervention. Relying on manual configuration steps to deploy applications restricts our ability to implement continuous delivery.

Cloud Foundry is an opinionated platform, actively discouraging the creation of snowflake environments.

Whilst it is still possible to manually run administrative tasks, either by connecting to bound services locally or using a remote console, it’s preferable to describe our tasks as code and let the platform handle it.

Using custom start commands, we can deploy applications which run tasks automatically during their normal startup procedure.

GeoPix Live Photos

Andrew Trice wrote a great sample application for IBM Bluemix called GeoPix.

GeoPix uses the IBM MobileFirst services to provide a native iOS application which allows users to capture images from their mobile phones, storing them on the local device with automatic syncing to the cloud when online.

Using a web application, the user can view their images over a map based upon their location when the photo was taken.

I’ve been using the demonstration to highlight the mobile capabilities of IBM Bluemix and had an idea for an enhancement…

Could the web page update with new pictures without having to refresh the page?

Looking at the source code, the web application is a Node.js application using the Leaflet JavaScript library to create interactive maps. Images captured from mobile devices are synchronised to a remote CouchDB database. When the user visits the GeoPix site, the application queries this database for all mobile images and renders the HTML using the Jade templating language.

Adding support for live photos will require two new features…

  • Triggering backend events when new photos are available
  • Sending these photos in real-time to the web page

Change Notifications Using CouchDB

CouchDB comes with built-in support for listening to changes in a database, change notifications. The _changes feed for a database is an activity stream publishing all document modifications.

GeoPix uses the following CouchDB client library, to interact with our database from NodeJS. This library provides an API to start following database changes and register callbacks for updates.

Modifying our application code, upon connecting to the CouchDB database, we register a change notification handler. We follow all changes that occur in the future (since: “now”) and include the full document contents in the change event (include_docs: true).

1
2
3
4
5
6
7
8
9
10
Cloudant({account:credentials.username, password:credentials.password}, function(err, cloudant) {
    var geopix = cloudant.use(database);
    var feed = geopix.follow({include_docs: true, since: "now"});

    feed.on('change', function (change) {
      // ....we can now send this data to the web pages
    });

    feed.follow();
})

Now, every time a user sync their local photos to the cloud, the registered callback will be executed.

How do we send new photos to the web page over a real-time stream?

Real-time Web with Socket.IO

Introducing Socket.IO

Socket.IO enables real-time bidirectional event-based communication.
It works on every platform, browser or device, focusing equally on reliability and speed.

Sounds great!

By embedding this library into our application, we can open a real-time event stream between the server and client. This channel will be used by the client to listen for new images and then update the page.

The library has great documentation and provides both server and client modules. It also integrates with ExpressJS, the web framework used in GeoPix. Socket.IO can use either WebSocket or long-polling transport protocols.

Socket.IO supports running under ExpressJS with minimal configuration, here are the changes needed to start our real-time stream in GeoPix:

1
2
3
4
5
6
7
8
9
10
11
12
13
var express = require('express');
var app = express();
var server = require('http').Server(app);
var io = require('socket.io')(server);

// ...snipped out the app routes for express

io.on('connection', function (socket) {
    console.log('New Client WSS Connection.')
});

var port = (process.env.VCAP_APP_PORT || 3000);
server.listen(port);

When a document change event is fired, executing the handle we registered above, we want to send this data to all connected clients.

Using the emit call from the server-side API will do this for us.

1
2
3
feed.on('change', function (change) {
    io.sockets.emit('image', change);
});

Now we’re sending changes to the clients, we need to modify the client-side to listen for events and update the page.

Socket.IO provides a JavaScript client library that exposes a simple API for listening to events from the server-side stream. Once we’ve included the script tag pointing to the client library, we can register a callback for image events and update the DOM with the new elements.

We’re sending the full database document associated with each photo to the client. The raw image bytes are stored as an attachment.

1
2
3
4
5
6
7
8
9
10
11
12
13
var socket = io(); // TIP: io() with no args does auto-discovery
socket.on('connect', function () {
    console.log('WSS Connected');

    socket.on('image', function (image) { // TIP: you can avoid listening on `connect` and listen on events directly too!
        var attachment = Object.keys(image.doc._attachments)[0]
        var url = "/image/" + image.doc._id + "/" + attachment;
        add_new_image(url, image.doc.clientDate, 'latitude: '
            + image.doc.latitude + ', longitude: '
            + image.doc.longitude + ', altitude: '
            + image.doc.altitude);
    });
});

…and that’s it! Now our web pages will automatically update with new photos whenever the mobile application syncs with the cloud.

CouchDB + Socket.IO = Real-time Awesome!

Adding real-time photos to our application was amazingly simple by combining CouchDB with Socket.IO.

CouchDB’s _changes API provided an easy way to follow all modifications to database documents in real-time. Socket.IO made the configuration and management of real-time event streams between our server and client straightforward.

With minimal code changes, we simply connected these two technologies to create a real-time photo stream for our GeoPix application. Awesome.

AlchemyAPI & Updated Watson Nodes for Node-RED

I’ve recently been working on a number of updates to the Node-RED nodes for the IBM Bluemix platform…

Highlights below:

New AlchemyAPI Nodes

There are two new nodes (Feature Extract and Image Extract) in the package, allowing users to call services from the AlchemyAPI platform.

  • Feature Extract. This node will analyse external URLs, HTML or text content with features for text-based analysis from the AlchemyAPI service, e.g. keywords, sentiment, relationships, etc.

  • Image Analysis. This node will analyse images, passed in as external URLs or raw image bytes, to extract faces, content and URLs.

Configuration for each node is available through the node editor panel.

For full details on all the capabilities of the AlchemyAPI platform, please see their documentation.

Updated IBM Watson Nodes

With the recent changes to the IBM Watson services, there were a number of changes needed to support the API changes. All the IBM Watson nodes now work with the GA versions of the services.

Users must ensure they are using GA versions of the service with the nodes. Details on migration steps are available on the IBM Watson blog post about the updates.

Running Locally

When running Node-RED on IBM Bluemix, credentials for the services bound to the application are automatically registered. Previously, running the nodes outside of IBM Bluemix required complex configuration to register service credentials. With this release, users will be prompted to input the service credentials in the node editor panel if the application isn’t running on IBM Bluemix. Much easier!

If you have questions or encounter issues, please ask over on Stackoverflow or raise issues in Github

Debugging Cloud Foundry Stack Issues

Recent changes to the Cloud Foundry stacks supported by IBM Bluemix have led to a number of issues for users. I’ve helped users diagnose and fix issues that have occurred due to a mistmatches between the platform stack, applications and the buildpack. Learning a number of techniques for helping to discover and resolve these issues and I wanted to share them with everyone else.

Running on Cloud Foundry’s Platform-as-a-Service solution, we take for granted that low-level concepts like operating systems are abstracted away from the developer.

However, when we run into issues it can be necessary to jump into the weeds and find out what’s going on under the hood…

What are Cloud Foundry “stacks”?

According to the documentation

A stack is a prebuilt root filesystem (rootfs) which works in tandem with a buildpack and is used to support running applications.

Think of the stack as the underlying operating-system running your application. This will be combined with the buildpack to instantiate the runtime environment.

Most users don’t have to care which stack they are running on.

However, if your application needs a specific version of a system library or you want to verify a specific command line application is installed, you may need to dig deeper…

What “stacks” does my platform support?

Using the Cloud Foundry CLI, issue the following command to see what stacks are available on the platform.

1
2
3
4
5
6
7
8
[16:27:30 ~]$ cf stacks
Getting stacks in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

name         description
lucid64      Ubuntu 10.04
seDEA        private
cflinuxfs2   Ubuntu 14.04.2 trusty

Stack information contains the unique name for each stack and the underlying operating system version.

Which “stack” is my application running on?

Since v6.11.0, the stack for an application has been shown in the CLI application info output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[16:34:39 ~]$ cf app debug-testing
Showing health and status for app debug-testing in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: debug-testing.mybluemix.net
last uploaded: Tue Jun 16 15:47:21 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM)

     state     since                    cpu    memory           disk           details
#0   running   2015-06-30 08:53:57 PM   0.0%   242.5M of 512M   196.8M of 1G

How can I choose the “stack” my application runs on?

Users can set the stack for an application using the -s command-line parameter during deployment. The stack identifier should match one of the names shown in the output from the cf stacks command.

1
$ cf push -s stack_identifier

How are the “stacks” defined?

This Github repository contains the source files for building the stacks. There’s a Dockerfile for the current cflinuxfs2 stack to build the image used in Cloud Foundry.

How can I poke around inside a “stack” locally?

Using Docker, we can easily pull down the same “base” operating system used for a specifc “stack” and run locally.

For the cflinuxfs2 stack, we can pull down the Ubuntu Trusty image and run a terminal inside it.

1
2
$ docker pull ubuntu:trusty
$ docker run -i -t ubuntu:trusty /bin/bash

How can I easily migrate existing applications to a new stack?

Rather than having to re-deploy each application separately, there’s a great CF CLI plugin to automatically migrate all your applications from lucid64 to cflinuxfs2.

Making Logs Awesome - Elasticsearch in the Cloud Using Docker

Logs are boring.

It used to be the only time you’d be looking at your application logs was when something went wrong.

Logs filled up disk space until they rotated out of existence.

…but now businesses are increasingly focused on using data to drive decisions.

Which advert leads to the highest click-through rates?

How did that last website change affect user retention?

What customer devices should our website support?

Guess where the answers lie?

Logs.

Storing, processing and querying logs effectively is helping businesses succeed.

Introducing the ELK (Elasticsearch, Logstash, Kibana) stack…

Five years ago, Elasticsearch, an open-source full-text search engine, was released. It’s now the second most popular enterprise search engine. Complementing this project were Logstash and Kibana. Logstash was a log processing pipeline that could normalize streaming logs into a centralised Elasticsearch cluster. Kibana was an analytics and visualisation platform for turning those logs into actionable insights.

These tools were commonly used together, now known as the ELK stack, to deliver…

“an end-to-end stack that delivers actionable insights in real time from almost any type of structured and unstructured data source.”

ELK, making logs awesome!

Manually installing and configuring Elasticsearch, Logstash and Kibana is not a trivial task.

Luckily, there is a better way…

Docker

“Docker allows you to pack, ship and run any application as a lightweight container”.

Docker images define pre-configured environments that containers are started from. Docker Hub is the public image registry, where anyone can publish, search and retrieve new images.

Rather than having to install and configure individual software packages, we can pull down one of the many existing Docker images for the ELK stack.

With one command, we can spin up an entire ELK instance on any platform with no extra configuration needed.

Magic.

IBM Containers

IBM recently announced Docker support for their Platform-as-a-Service cloud service, IBM Bluemix. Developers can now deploy and manage Docker containers on a scalable cloud platform.

IBM Containers provides the following services:

  • Private image registry
  • Elastic scaling and auto-recovery
  • Persistent storage and advanced networking configuration
  • Automated security scans
  • Integration with the IBM Bluemix cloud services.

Using this service, we can build and test a custom ELK container in our local development environment and “web-scale” it by pushing to the IBM Bluemix cloud platform.

Manging Application Logs

Once our ELK instance is running, we can then start to push application logs from other applications running on IBM Bluemix into the service. We’ll look at automatically setting up a log drain to forward all applications logs into a centralised Elasticsearch service. We can then start to drive business decisions using data rather than intuition using Kibana, the visualisation dashboard.

This blog post will explain the technical details of using Docker to create a customised ELK service that can be hosted on a scalable cloud platform.

Running ELK instances Using Docker

Docker Hub has over forty five thousands public images available. There are multiple public images we can pull down with a pre-configured ELK stack. Looking at the options, we’re going to use the sebp/elk repository because it’s popular and easily modifiable with a custom configuration.

We’re going to start by pulling the image into our local machine and running a container to check it’s working…

1
2
$ docker pull sebp/elk
$ docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk sebp/elk

That last command will start a new container from the sebp/elk image, exposing the ports for Kibana (5601), Elasticsearch (9200) and Logstash (5000) for external access. The container has been started with the -i flag, interactive mode, allowing us to monitor the container logs in the console. When the instance has started, we can view the status output from command line.

1
2
3
$ docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                                                                              NAMES
42d40d1fb59c        sebp/elk:latest     "/usr/local/bin/star   27 seconds ago      Up 26 seconds       0.0.0.0:5000->5000/tcp, 0.0.0.0:5601->5601/tcp, 0.0.0.0:9200->9200/tcp, 9300/tcp   elk

Using Mac OS X for local development, we’re using the Boot2Docker project to host a Linux VM for deploying Docker containers locally. With the following command, we can discover the virtual IP address for the ELK container.

1
2
$ boot2docker ip
192.168.59.103

Opening a web browser, we can now visit http://192.168.59.103:5601 to show the Kibana application. For now, this isn’t very useful because Elasticsearch has no logs!

Let’s fix that…

Draining Logs from Cloud Foundry

Cloud Foundry, the open-source project powering IBM Bluemix, supports setting up a syslog drain to forward all applications logs to a third-party logging service. Full details on configuring this will be shown later.

Scott Frederick has already written an amazing blog post about configuring Logstash to support the log format used by the Cloud Foundry. Logstash expects the older RFC3164 syslog formatting by default, whilst Cloud Foundry emits log lines that follow the newer RFC5424 standard.

Scott provides the following configuration file that sets up the syslog input channels, running on port 5000, along with a custom filter that converts the incoming RFC5424 logs into an acceptable format.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}" }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
  }
}

output {
  elasticsearch { }
}

Using this configuration, Logstash will accept and index our application logs into Elasticsearch.

Note: There is also a custom plugin to enable RFC5424 support.

Building Custom Docker Images

Using the custom Logstash configuration relies on building a new Docker image with this configuration baked in. We could download the Git repository containing the image source files, modify those and rebuild from scratch. However, an easier way uses the existing image as a base, applies our modifications on top and then generates a brand new image.

So, how do we build our own Docker images? Using a Dockerfile.

A Dockerfile is a text document that contains all the commands you would
normally execute manually in order to build a Docker image.

Reviewing the Dockerfile for the sebp/elk image, configuration for logstash is stored in the /etc/logstash/conf.d/ directory. All we need to do is replace these files with our custom configuration.

Creating the custom configuration locally, we define a Dockerfile with instructions for building our image.

1
2
3
4
5
6
7
$ ls
01-syslog-input.conf 10-syslog.conf       Dockerfile
$ cat Dockerfile
FROM sebp/elk
RUN rm /etc/logstash/conf.d/01-lumberjack-input.conf
ADD ./01-syslog-input.conf /etc/logstash/conf.d/01-syslog-input.conf
ADD ./10-syslog.conf /etc/logstash/conf.d/10-syslog.conf

The Dockerfile starts with the “sebp/elk” image as a base layer. Using the RUN command, we execute a command to remove existing input configuration. After this the ADD command copies files from our local directory into the image.

We can now run the Docker build system to generate our new image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ docker build -t jthomas/elk .
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM sebp/elk
 ---> 2b71e915297f
Step 1 : RUN rm /etc/logstash/conf.d/01-lumberjack-input.conf
 ---> Using cache
 ---> f196b6833121
Step 2 : ADD ./01-syslog-input.conf /etc/logstash/conf.d/01-syslog-input.conf
 ---> Using cache
 ---> 522ba2c76b00
Step 3 : ADD ./10-syslog.conf /etc/logstash/conf.d/10-syslog.conf
 ---> Using cache
 ---> 79256ffaac3b
Successfully built 79256ffaac3b
$ docker images jthomas/elk
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jthomas/elk         latest              79256ffaac3b        26 hours ago        1.027 GB

…and that’s it! We have a customised Docker image with our configuration changes ready for running.

Testing Our Custom Image

Before pushing this image to the cloud, we need to check it’s working correctly. Let’s start by starting a new container from our custom image locally.

1
$ docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk jthomas/elk

Now, use the CF CLI to access recent logs for a sample application and paste the output into a telnet connection to port 5000 on our container.

1
2
3
4
5
6
7
8
9
10
11
$ cf logs APP_NAME --recent
Connected, dumping recent logs for app debug-testing in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...

2015-07-02T17:14:47.58+0100 [RTR/1]      OUT nodered-app.mybluemix.net - [02/07/2015:16:14:47 +0000] "GET / HTTP/1.1" 200 0 7720 "-" "Java/1.7.0" 75.126.70.42:56147 x_forwarded_for:"-" vcap_request_id:1280fe18-e53a-4bd4-40a9-2aaf7c53cc54 response_time:0.003247100 app_id:f18c2dea-7649-4567-9532-473797b0818d
2015-07-02T17:15:44.56+0100 [RTR/2]      OUT nodered-app.mybluemix.net - [02/07/2015:16:15:44 +0000] "GET / HTTP/1.1" 200 0 7720 "-" "Java/1.7.0" 75.126.70.43:38807 x_forwarded_for:"-" vcap_request_id:4dd96d84-c61d-45ec-772a-289ab2f37c67 response_time:0.003848360 app_id:f18c2dea-7649-4567-9532-473797b0818d
2015-07-02T17:16:29.61+0100 [RTR/2]      OUT nodered-app.mybluemix.net - [02/07/2015:16:14:29 +0000] "GET /red/comms HTTP/1.1" 101 0 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36" 75.126.70.42:54826 x_forwarded_for:"75.126.70.42" vcap_request_id:15c2d4f8-e6ba-4a20-77b7-345aafd32e95 response_time:MissingFinishedAt app_id:f18c2dea-7649-4567-9532-473797b0818d
$ telnet 192.168.59.103 5000
Trying 192.168.59.103...
Connected to 192.168.59.103.
Escape character is '^]'.
// PASTE LOG LINES....

Starting a web browser and opening the Kibana page, port 5601, the log lines are now available in the dashboard. Success!

Pushing Docker Images To The Cloud

Having successfully built and tested our custom Docker image locally, we want to push this image to our cloud platform to allow us to start new containers based on this image.

Docker supports pushing local images to the public registry using the docker push command. We can choose to use a private registry by creating a new image tag which prefixes the repository location in the name.

IBM Containers’ private registry is available at the following address, registry.ng.bluemix.net.

Let’s push our custom image to the IBM Containers private registry…

1
2
3
4
5
6
7
8
9
10
11
12
13
$ docker tag jthomas/elk registry.ng.bluemix.net/jthomas/elk
$ docker images
REPOSITORY                                     TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jthomas/elk                                   latest              79256ffaac3b        43 hours ago        1.027 GB
registry.ng.bluemix.net/jthomas/elk           latest              79256ffaac3b        43 hours ago        1.027 GB
$ docker push registry.ng.bluemix.net/jthomas/elk
The push refers to a repository [registry.ng.bluemix.net/jthomas/elk] (len: 1)
Sending image list
Pushing repository registry.ng.bluemix.net/jthomas/elk (1 tags)
511136ea3c5a Image successfully pushed
...
79256ffaac3b: Image successfully pushed
Pushing tag for rev [79256ffaac3b] on {https://registry.ng.bluemix.net/v1/repositories/jthomas/elk/tags/latest}

Pushing custom images from a local environment can be a slow process. For the elk image, this means transferring over one gigabyte of data to the external registry.

We can speed this up by using IBM Containers to create our image from the Dockerfile, rather than uploading the built image.

Doing this from the command line requires the use of the IBM Containers command-line application.

Managing IBM Containers

IBM Containers enables you to manage your containers from the command-line with two options

Both approaches handle the interactions between the local and remote Docker hosts, while providing extra functionality not supported natively by Docker.

Full details on the differences and installation procedures for the two applications are available here.

Building Images Using IBM Containers

Building our image using the IBM Containers service uses the same syntax as Docker build. Local files from the current directory will be sent with the Dockerfile to the remote service. Once the image has been built, we can verify it’s available in the remote repository.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ ice build -t registry.ng.bluemix.net/jthomas/elk .
zipped tar size: 706
Posting 706 bytes... It may take a while...
Step 0 : FROM sebp/elk
 ---> 2b71e915297f
Step 1 : RUN rm /etc/logstash/conf.d/01-lumberjack-input.conf
 ---> Using cache
 ---> ed13d91e0197
Step 2 : ADD ./01-syslog-input.conf /etc/logstash/conf.d/01-syslog-input.conf
 ---> Using cache
 ---> 808a4c7410c7
Step 3 : ADD ./10-syslog.conf /etc/logstash/conf.d/10-syslog.conf
 ---> Using cache
 ---> 117e4454b015
Successfully built 117e4454b015
The push refers to a repository [registry.ng.bluemix.net/jthomas/elk] (len: 1)
Sending image list
Pushing repository registry.ng.bluemix.net/jthomas/elk (1 tags)
Image 117e4454b015 already pushed, skipping
Pushing tag for rev [117e4454b015] on {https://registry.ng.bluemix.net/v1/repositories/jthomas/elk/tags/latest}
$ ice images
REPOSITORY                                TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
registry.ng.bluemix.net/jthomas/elk       latest              5454d3ec-0f3        44 hours ago        0 B
registry.ng.bluemix.net/ibmliberty        latest              3724d2e0-06d        9 days ago          0 B
registry.ng.bluemix.net/ibmnode           latest              9435349e-8b4        9 days ago          0 B

All private repositories on IBM Bluemix have two official images for supported versions of NodeJS and Websphere Liberty.

We can now see the third image is the custom ELK stack that was built.

Starting ELK Containers

Starting containers from images in the IBM Containers registry can be done using the command-line applications or through the IBM Bluemix UI. In this example, we’ll be using the IBM Bluemix UI to start and configure a new ELK container from our pre-configured image.

Logging into the IBM Bluemix, the Catalogue page shows the list of available images used to create new containers. We have both the official images from IBM Containers and our custom ELK service.

Selecting the ELK image, we can configure and run a new container from this image. Setting up a new container with a public IP address, memory limit to 1GB and expose the same ports as running locally (5000, 5601 and 9200).

Clicking the Create button, IBM Bluemix will provision and start our new container.

Once the container has started, we can view the Dashboard page for this instance. Here we can view details about the container instance, modify the running state and access monitoring and logs tools.

…and that’s it! We now have our ELK service running using IBM Containers ready to start processing logs from our applications.

Visiting the external IP address assigned to the container on the Kibana application port (5601) shows the Kibana web interface demonstrating our container has started correctly.

Draining Cloud Foundry Logs

Cloud Foundry supports draining applications logs to a third-party syslog service. The ELK container has a syslog drain configured on port 5000 of the public IP address bound to the instance.

Binding this custom syslog drain to Cloud Foundry applications uses a custom user-provided service. Creating user-provided services using the CF CLI, there is a special flag, -l, that notifies the platform this service is a syslog drain. Binding this special syslog drain service to an application will automatically set up log forwarding. Once the application has been restarted, logs will start to flow into the external service.

1
2
3
$ cf cups logstash-drain -l syslog://[CONTAINER_IP]:5000
$ cf bind-service [app-name] logstash-drain
$ cf restart [app-name]

Cloud Foundry supports multiple syslog drains for the same application.

Testing this out is as simple as visiting our application to generate sample logs and then looking at the Kibana page to see they are showing up. Here is a screenshot of the expected output when our ELK container is successfully processing logs from a Cloud Foundry application.

Conclusion

Elastic Search, Kibana and Logstash is the modern log processing framework. Using Docker, we’ve been able to create a custom ELK service without manually installing and configuring a multitude of different software packages. Pushing this image to the IBM Containers platform means we can spin up new ELK containers on-demand within minutes!

Elasticsearch, Docker and IBM Containers… Making Logs Awesome.

Continuous Delivery for Phonebot

Since creating Phonebot last month, I’ve been working on setting up a fully-automated build and deploy for the project. Using IBM DevOps Services, Phonebot now has ”Continuous Delivery” enabled.

When new code is commited to the external Github repository, the build service will perform the following tasks.

  • Run Unit Tests and Code Lint Tools
  • Deploy To Test Server
  • Run Integration Tests Against Test Server
  • Deploy To Production

Each stage will only be executed if the following stage passes.

In the following post, I’ll explain how to set up each stage and share tips making it easy to replicate this setup for your projects…

Writing Tests for Phonebot

Phonebot comes with a comprehensive test suite. I’ve used the Mocha test framework for creating unit and integration tests. Test assertions use NodeJS’ built-in library. The mockery library is used to replace module dependencies with mock objects.

Setting up the scripts field in package.json allows us to use NPM to run our tests.

NPM will look into the ”node_modules/.bin” directory for binaries when running scripts. This means we don’t need Mocha installed on the deployment host to run tests. The ”devDependencies” field includes modules we rely on during development but not production.

1
2
3
4
5
6
7
8
9
10
"devDependencies": {
    "mocha": "^2.2.5",
    "mocha-eslint": "^0.1.7",
    "mockery": "^1.4.0"
},
"scripts": {
    "test": "mocha test/unit",
    "integration-test": "mocha test/integration",
    "start": "node app.js"
  },

Running the following commands will run the unit and integration tests.

1
2
$ npm test  // defaults to 'run test'
$ npm run integration-test

Running Code Linters

Along with unit tests, we want to run ‘code linters’ to catch any errors in our JavaScript code. We’re using the eslint tool with the following configuration. Using this module, we’re setting up the eslint tool as a test case.

This test will be automatically run in the unit test phase and errors incorporated into the test report.

Mocking Services In Integration Tests

When the unit tests have passed, we’re going to deploy a test instance of the application. Integration tests will make HTTP requests to simulate user activity, capture the responses and then verify the application is behaving as expected.

Phonebot uses external services, provisioned through IBM Bluemix, to make phone calls, translate speech to text and communicate with Slack channels. Services configuration parameters, e.g. username, password, host, are passed into the application using environment variables.

During integration tests, we want to capture all requests to external services and provide hardcoded HTTP responses to be returned. With service parameters coming from environment properties, rather than hardcoded in the application, we can simply replace the bound services configuration with our own values. This application will pick up these new values, pointing to our stub server, at runtime without any changes needed to the code.

This stub server has been created to capture all incoming HTTP requests and make them available at a custom HTTP endpoint. We’re also configured HTTP routes to simulate each of the external services and return hardcoded responses.

Deploying our test server in a different space to production means we can have custom credentials set up without having to modify the service configuration in the production environment.

The following commands will show the existing configuration values that we can replicate in the test environment.

1
2
3
4
5
6
$ cf env phonebot
$ cf create-space test
$ cf target -s test
$ cf cups twilio -p "accountSID, authToken, url"
$ cf cups speech_to_text -p "username, password, url"
$ cf cups slack_webhooks -p "slackbot-testing"

With the test credentials created, we can deploy the application to the “test” space without modifications.

Setting up Build and Deploy Pipeline

We’re going to use IBM DevOps Services to build and manage the “Continuous Delivery” pipeline. From the home page, click the “Create Project” button to import our existing Github project into the workspace.

The “Create Project” page allows us to link an existing project from Github to the new project. Changes to the external repository will be automatically pushed through to our project.

Selecting the “Make a Bluemix project” option will automatically configure deploying to the Bluemix platform.

When the project has finished importing, we can access the “Build and Deploy” pipeline…

… which will currently be empty. Clicking the “Add Stage” button will allow us to start configuring the build, test and deploy jobs for our pipeline.

Running Unit Tests and Code Lint Tools

The first stage in our pipeline will run the unit tests when a new commit is published to the Phonebot repository on Github.

Using the “Input” tab, we’re configuring the stage to pick up all changes in the “master” branch of the https://github.com/jthomas/phonebot.git repository. The input for a stage can also be the build artifacts from a previous stage.

On the “Jobs” tab, we can configure multiple tasks to be executed when triggered by the stage input. For the unit tests, we’re using a simple shell script to install the project dependencies and run the NPM task.

Deploy Test Server

Adding a second stage to the pipeline after the unit tests, we will use it to deploy our test server. This stage will only be executed if the first stage completes successfully.

Using the “Deploy” rather than “Test” job, presents us with a configuration panel to set up the deployment parameters. The “test” space which contains our test configuration for our mock services. Choosing a different application name means our test server won’t clash with the existing production host already deployed.

Running Integration Tests Against Test Server

Once the test server has been deployed, we can trigger the pipeline stage to run integration tests against this host.

Using Mocha to run out integration tests means we can follow the setup as the unit test stage. Defining a “test” job, we install the project dependencies and then run the test harness.

Phonebot’s integration tests use environment variables to define the test and stub server locations. We can define these through the stage setup page, as shown below.

Deploy To Production

Finally, provided all the previous stages were successfully, the last stage will deploy our application into production.

Configuring a “Deploy” task, this time we use the production space “dev” and use the proper application name.

…and that’s it!

With our “Continuous Delivery” pipeline now configured, new versions of Phonebot will be automatically deployed to production without any manual work.

For testing, each stage can be triggered manually. Logs are available in to diagnose any issues that may occur.

Using IBM DevOps Services, we rapidly created a build and deploy pipeline linked to a project on Github without having to manually configure build systems, test servers or anything else you would expect.

Our example was relatively simple, the service can be configured for far more complicated build and deploy tasks. The documentation gives full details on the capabilities of that platform. If you have any issues, please use the IBM Answers support forum to post questions and get answers from the development team.