James Thomas

Notes on JavaScript

Debugging Live Containers on IBM Bluemix

For the last few months, I’ve been using the ELK stack to collect logs from my Cloud Foundry applications. This service has been deployed on IBM Bluemix using a Docker container, previously detailed in this blog post, and running happily until it ran into issues this week.

Trying to load the Kibana web application, the server was returning connection refused errors. Looking at the container in the IBM Bluemix dashboard showed no obvious signs of issues. Reviewing the container log output uncovered nothing indicating what had failed.

Hmmm…

Fixing this issue would require me to start debugging from within the live container, but how?

This container image had not included an SSH daemon that would allow remote access over SSH.

Looking over the documentation for the IBM Containers plugin for the Cloud Foundry CLI, I noticed the exec command.

Docker exec allows a user to spawn a process inside their Docker container via
the Docker API and CLI.

Since Docker 1.3, released in October 2014, the exec command has allowed users to run new commands within existing containers.

The IBM Containers implementation now supports this Docker command.

Using the IBM Containers plugin for the Cloud Foundry CLI, I can find the container id for the instance I want to debug and then start a bash shell to start resolving my issue.

1
2
$ cf ic ps
$ cf ic exec -it <container_id> /bin/bash

Having a live shell to my container allowed me to resolve the issue within a few minutes, without having to affect the running state of the container. This command also removes the need to keep an SSH daemon running on containers for remote access.

For more information on the subset of Docker commands supported by IBM Containers, see the following documentation.

Cloud Foundry Application Monitoring Bot for Slack

Cloud Foundry makes it so easy to build, deploy and manage applications that it can be a struggle just to keep up with development progress…

“Who is restarting this application?”
“What is this new service instance?”
“When did this application instance run out of memory?”

Development teams are increasingly using Slack to collaborate on projects and using custom bots to manage and monitor applications, triggered through the channel messages. This approach, popularised by Github, has now become known as “ChatOps”. Using group chat for development projects gives greater operational visibility to everyone in the team.

Slack has exploded in use over the past two years, recently signing up more than a million active users. The platform publishes an API for writing bots that respond automatically to messages, allowing users to write custom integrations for external services.

Users can register webhooks to receive channel messages, based upon keyword triggers, and allow bots to reply with new channel messages. The platform also provides a websocket channel with registered bots for real-time communication.

Could we write a custom bot for monitoring applications on the Cloud Foundry platform?

The bot would publish notifications about applications and services into group channels, helping keep teams updated with platform events in real-time.

Cloud Foundry Monitoring APIs

Cloud Foundry provides access to the platform through a series of RESTful APIs, exposed by the Cloud Controller component. User commands from the CF CLI tool are translated into calls to these APIs.

Tip: Setting the CF_TRACE environment parameter to true will show the API calls generated by the CLI commands.

Platform user account credentials are used to obtain OAuth2 tokens for authenticating service calls.

Looking at the documentation, there’s an endpoint for retrieving all platform events. This API is used to retrieve events for an application when using the CF CLI events command. Events can be filtered by the application, event type and timestamps. Responses include events about changes to applications, services and service instances.

Polling this API, with timestamp filtering to ignore old events, we can retrieve a continuous stream of new platform events.

Slack Integration

Setting up a new bot integration for a Slack group provides you with a token you can use to authenticate with the Real-Time Messaging API. Rather than having to implement the Websocket-based API handler ourselves, we can use one of the many existing community libraries.

Using the Node.js client library, passing in the authentication token, we just need to implement callback handlers for the API events.

Slack Client
1
2
3
4
5
6
7
8
9
10
11
12
var Slack = require('slack-client')

var slackToken = 'xoxb-YOUR-TOKEN-HERE' # Add a bot at https://my.slack.com/services/new/bot and copy the token here.
var autoReconnect = true # Automatically reconnect after an error response from Slack.
var autoMark = true # Automatically mark each message as read after it is processed.

var slack = new Slack(slackToken, autoReconnect, autoMark)
slack.on('message', function (message) {...})
slack.on('error', function (err) {...})
slack.on('open', function () {})

slack.login()

When platform events occur, we forward these to any channels the bot is registered in.

Plugging together the Cloud Foundry event monitoring code with the Slack bot integration, cfbot was born…

cfbot

This Cloud Foundry monitoring bot can be deployed to… Cloud Foundry!

You will need to register the bot with your Slack group to receive an authentication token. This token, along with login details for a platform account, need to be created as user-provided service credentials. The bot will read these service credentials on deployment and start monitoring for events.

Full installation instructions available in the project README.

usage

cfbot will monitor events from applications in all spaces and organisations that the user account has access to.

Users can filter the applications and events being reported using the apps and events commands. Both commands take application or event identifiers that are used to match incoming events. The wildcard ‘*’ identifier can be used to revert to matching all events.

@cf apps // show the currently application filter
@cf apps app_name // add the 'app_name' to the filter list
@cf apps * // reset to the filter to wildcard matching

@cf events // show the currently event filter
@cf events event_type // add the 'event_type' to the filter list
@cf events * // reset to the filter to wildcard matching

@cf status // show the current bot status message

@cf polling_frequency // show the cf events api polling time in seconds
@cf polling_frequency 10 // set the cf events api polling time in seconds

The following events are currently registered:

  • App Creation and Deletion Events.
  • App Lifecycle Events (start, stop, restart, restage)
  • Instance Crash Events.
  • Service Creation, Deleting and Binding.
  • Scaling (memory, CPU, disk)
  • Routes Changes (map, unmap)

Other bots

Other people have written Cloud Foundry bots before cfbot. Here are the other projects I discovered that might be useful…

Updated IBM Watson Nodes for Node-RED

Earlier this year, I made a major upate to the Node-RED nodes for the IBM Watson services available through IBM Bluemix. Since then, the IBM Watson team has been busy, with lots of changes to APIs. I’ve recently been working through these changes, updating the nodes, to ensure they work against the latest APIs.

Updates to these nodes have now been finished and are available through the boilerplate on IBM Bluemix or by installing the IBM Bluemix Nodes package locally.

If you have an existing Node-RED instance running in IBM Bluemix, please review the documentation for upgrade instructions.

If you encounter any issues, please raise a issue on Github.

For full details on the changes are available in the pull request.

NPM Modules in Node-RED

Before Christmas, my department at IBM had one of our semi-regular Hack Days to get everyone together and work on interesting ideas away from the day job. I spent the time playing with an idea to make exposing NPM packages in Node-RED easier…

Node-RED is a visual tool for wiring the Internet of Things.

It makes it easy to create, combine and control data flowing between hardware devices, web APIs and open protocols. The tool exposes operations through a series of nodes, which can be created through the browser-based editor and connected to other nodes to create message flows.

The tool comes built-in with a huge selection of nodes, from connecting to a Raspberry Pi to handling HTTP requests, for creating flows. Users create their own nodes to expose new functionality, by creating small modules using JavaScript and HTML. People often wrap existing NPM modules into custom Node-RED nodes to use that module functionality in flows.

NPM has over two hundred thousand modules, many of which provide simple “stateless” functions that return a result based upon the input data. These modules are ideally suited for using within Node-RED message flows.

Exposing multiple new NPM modules required creating custom Node-RED nodes for every module you wanted to use. Whilst the Node-RED node boilerplate is small, it becomes a bit laborious and repetitive to keep doing this for extremely simple modules.

Wondering how to make this easier, I started hacking on ideas.

Two days later…

Node-RED node to dynamically expose NPM modules as nodes.

This new node allows you to dynamically expose NPM modules as Node-RED nodes without needing to manually create new nodes for them.

Using the node editor panel, the user can set the name of the NPM package to expose along with the module invocation style.

Incoming flow messages are passed as an argument to the function being executed. Users can set up the node to call the module, a module function or even run custom setup code in response to incoming messages.

Execution results, either returned directly or asynchronously (Promises and callback-style supported), are sent as the outgoing message payload.

When the flow is deployed, the NPM module will be automatically installed and instantiated.

Now we don’t have to write lots of extra boilerplate code every time we want to use a tiny NPM module in a Node-RED flow, hurrah!

Here’s a short demonstration using the node to expose the sentiment package in a flow:

Node.js V4 in Cloud Foundry

Last week, Node.js released the latest version of their project, v4.0.0. This release, representing the convergence of io.js with the original Node.js project, came with lots of exciting features like improved ES6 support.

Cloud Foundry already supports multiple versions of the Node.js runtime. Developers select the desired runtime version using a parameter in their application’s package descriptor.

So, we just update package.json to include “4.0.0” and re-deploy our application?

Not yet.

There is an unresolved technical issue delaying the release of “official” Node.js v4 support for the platform. 😿

Can we add support ourselves?

Yes!

To do this, we need to explore how Cloud Foundry configures the runtime environment for applications.

Buildpacks

Rather than hardcoding supported runtimes and frameworks into the platform, Cloud Foundry borrowed the buildpack model from Heroku. Buildpacks are a set of scripts, run by the platform during deployment, to configure the runtime environment.

Users can set an explicit buildpack for an application, using the manifest, or let the platform decide. Buildpacks for common runtimes are pre-installed with the platform. Buildpacks set through the manifest can point to external URLs, allowing users to create new buildpacks supporting custom runtimes.

Each buildpack must contain the following files as executable scripts.

  • bin/detect - determine whether a buildpack is suitable for an application.
  • bin/compile - install and configure the runtime environment on the DEA.
  • bin/release - provide metadata with information on executing application.

Full details on existing buildpacks for the platform are available here.

Node.js is supported as an “official” buildpack by the platform. This will be the one we will modify to add support for the latest version of the runtime.

Node.js Buildpack

This is the Node.js buildpack for Cloud Foundry. Applications using this buildpack can select the version of Node.js to install using the engine parameter in the package descriptor.

Looking at the bin/compile script will show us how the Node.js runtime is installed during deployment.

This snippet handles accessing the Node.js version configured, using the node.engine parameter from package.json, before calling install_nodejs to install the correct runtime package.

install_bins() {
  local node_engine=$(read_json "$BUILD_DIR/package.json" ".engines.node")
  local npm_engine=$(read_json "$BUILD_DIR/package.json" ".engines.npm")

  echo "engines.node (package.json):  ${node_engine:-unspecified}"
  echo "engines.npm (package.json):   ${npm_engine:-unspecified (use default)}"
  echo ""

  warn_node_engine "$node_engine"
  install_nodejs "$node_engine" "$BUILD_DIR/.heroku/node"
  install_npm "$npm_engine" "$BUILD_DIR/.heroku/node"
  warn_old_npm
}

Searching through the buildpack for this function, it’s in the lib/binaries.sh file. Looking at the function code, it translates the version number into a URL pointing to an archive with the pre-compiled Node.js binary. This archive file is downloaded, extracted and installed into the runtime environment.

Translating Node.js version identifiers into archive URLs uses a special file in the buildpack, manifest.yml. This file maps every supported version to a pre-built binary location.

Looking at previous commits to the Node.js buildpack, adding support for additional versions of Node.js simply requires updating this file with the extra version identifier and archive URL.

Until the Cloud Foundry team updates the buildpack to support Node.js v4, they won’t provide an external archive containing the pre-built runtime environment.

Where can we find a suitable build of the Node.js binary?

Node.js Runtime Binaries

Cloud Foundry borrowed the buildpack concept from Heroku and still maintains backwards compatibility with their platform. Heroku buildpacks will work with Cloud Foundry applications. The Node.js buildpack for Cloud Foundry is actually still a fork of Heroku’s.

Looking back through the original buildpack source, this URL template is used to translate Node.js versions to archive URLs being built by Heroku.

http://s3pository.heroku.com/node/v$version/node-v$version-$os-$cpu.tar.gz

Combining the correct version identifier and platform parameters with this template gave the following location for a potential build of the Node.js v4 runtime.

http://s3pository.heroku.com/node/v4.0.0/node-v4.0.0-linux-x64.tar.gz

Running curl against the location successfully downloaded the Node.js v4 binary archive!

Custom v4 Buildpack

Forking the Cloud Foundry Node.js buildpack on Github, we can update the manifest.yml with the Node.js v4 identifier pointing to the Heroku runtime archive. This external Git repository will be used as the buildpack identifier in the application manfest.

Deploying with v4

Having updated our application manifest with the custom buildpack location and set the updated node version flag, re-deploying our application will start it running on Node.js v4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[20:02:29 ~]$ cf app sample-demo-app
Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.mybluemix.net
last uploaded: Fri Sep 18 18:33:56 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-18 07:35:01 PM   0.0%   65.3M of 256M   59M of 1G
[20:03:13 ~]$

Looking at the logs from the deployment we can see the latest Node.js runtime has been downloaded and installed within our runtime environment.

Conclusion

Buildpacks are a brilliant feature of Cloud Foundry.

Understanding how buildpacks are structured and used by the platform means we can start customising existing buildpacks and even start creating our own.

If you want to run Node.js applications using v4 on Cloud Foundry today, you can use the following buildpack created using the instructions above.

Cloud Foundry is currently adding support for the version to the official buildpack, follow their progress here.

Location-Based Cloud Foundry Applications Using Nginx and Docker

Routing application traffic based upon the geographic location of incoming requests can be used for a number of scenarios…

  • Restricting access to your application outside defined geographic regions.
  • Load-balancing traffic to the closest region for improved performance.
  • Providing custom applications for different countries.

IBM Bluemix allows deploying applications to different geographic regions through hosting instances of the Cloud Foundry platform in multiple locations.

Cloud Foundry supports simple HTTP routing rules for deployed applications. Organisations can register domains and routes for applications. Routes can be bound to one or more deployed applications. Incoming HTTP traffic is load-balanced, using the Round-Robin policy, between the application instances bound to a route.

However, the platform does not currently support traffic routing based upon the geographic location of incoming requests or sharing domains and routes between regions.

So, say we want to deploy custom versions of an application to different regions and automatically forward users to the correct version based upon their location. How can we achieve this?

Let’s find out…

Deploying Application To Different Regions

IBM Bluemix currently provides Cloud Foundry in two regions for deploying applications.

  • US South (api.ng.bluemix.net)
  • Europe (api.eu-gb.bluemix.net)

Moving between regions is as simple as providing the different region endpoint during the authentication command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[16:25:47 ~]$ cf login -a api.ng.bluemix.net -u james.thomas@uk.ibm.com -s dev
API endpoint: api.ng.bluemix.net

Password>
Authenticating...
OK

Targeted org james.thomas@uk.ibm.com

Targeted space dev

API endpoint:   https://api.ng.bluemix.net (API version: 2.27.0)
User:           james.thomas@uk.ibm.com
Org:            james.thomas@uk.ibm.com
Space:          dev
[16:26:44 ~]$

We’re now authenticated against the US South region.

Let’s start by deploying our sample application, which displays a web page showing the application URL, to this region.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[16:44:24 ~/code/sample]$ cf api
API endpoint: https://api.ng.bluemix.net (API version: 2.27.0)
[16:44:32 ~/code/sample]$ cf push sample-demo-app
Using manifest file /Users/james/code/sample/manifest.yml

Updating app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

...

Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.mybluemix.net
last uploaded: Fri Sep 11 15:45:04 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-11 04:46:00 PM   0.0%   67.1M of 256M   59M of 1G
[16:45:14 ~/code/sample]$ 

Once that has finished, we can move over to the European region and deploy our application there.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[16:52:33 ~/code/sample]$ cf login -a api.eu-gb.bluemix.net -u james.thomas@uk.ibm.com -s dev
[16:52:58 ~/code/sample]$ cf push sample-demo-app
Using manifest file /Users/james/code/sample/manifest.yml

Updating app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

...

Showing health and status for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sample-demo-app.eu-gb.mybluemix.net
last uploaded: Fri Sep 11 15:53:31 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM) (node.js-4.0.0)

     state     since                    cpu    memory          disk        details
#0   running   2015-09-11 04:54:17 PM   0.0%   67.4M of 256M   59M of 1G
[16:54:25 ~/code/bluemix/sample]$

With the second deployment completed, there are now instances of the same application running in separate regions.

Each instance is available through a separate URL.

Now we need to set up traffic forwarding from the relevant locations to the correct region.

Reverse Proxy with Region Traffic Forwarding

Due to the platform not supporting multi-region traffic routing, we need to set up a custom reverse proxy. This server will receive requests from our external application domain and transparently forward them onto the correct region application.

We’re going to use Nginx.

Nginx (pronounced engine-x) is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server

Nginx comes with a module for looking up locations associated with IP address using the MaxMind GeoIP library. The module can resolve incoming request addresses into continents, countries and even cities. Using the variables defined by the module, we can write traffic forwarding rules to send requests to the correct region.

Nginx Configuration

Nginx defines two configuration directives, geoip_country and geoip_city, to specify locations for the MaxMind GeoIP database files.

http { 
    ...
    geoip_country /usr/share/GeoIP/GeoIP.dat;
    geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
    ...
}

When configured, Nginx will expose a series of variables for each request with geographical information.

  • $geoip_country_code - two-letter country code, for example, “RU”, “US”.
  • $geoip_country_name - country name, for example, “Russian Federation”, “United States”.
  • $geoip_city_continent_code - two-letter continent code, for example, “EU”, “NA”.
  • $geoip_city - city name, for example, “Moscow”, “Washington”.

Starting with the default nginx configuration, there are only a few modifications needed to set up a reverse proxy based upon location.

For each request, we check the $geoip_city_continent_code against our list of regions. If the request is valid, setting the proxy_pass directive forwards the request onto the correct region. We also overwrite the Host: HTTP header with the region URL. IBM Bluemix uses this header to internally route incoming requests to the correct application host.

Requests coming from outside these locations will be sent to a custom error page.

Due to a known issue with IBM Containers, we must use IP addresses rather than the host names with the proxy_pass directive.

Here is the full configuration for the enabled-site/default file.

server {
  listen 80 default_server;
  listen [::]:80 default_server ipv6only=on;

  root /usr/share/nginx/html;
  index index.html index.htm;
  error_page 404 /404.html;

# Make site accessible from http://localhost/
  server_name localhost;

  location = /404.html {
    internal;
  }

  location / {
    set $host_header "unknown";

    if ($geoip_city_continent_code = "EU") { 
      proxy_pass http://5.10.124.141;
      set $host_header "sample-demo-app.eu-gb.mybluemix.net";
    }

    if ($geoip_city_continent_code = "NA") { 
      proxy_pass http://75.126.81.66;
      set $host_header "sample-demo-app.mybluemix.net";
    }

    if ($host_header = "unknown") {
      return 404;
    }

    proxy_set_header Host $host_header;
  }
}

With the reverse proxy server configured, we need to provision a new production server, install Linux and Nginx, configure networking, security updates and backup services…

…or we can use Docker.

Running Nginx using Docker

There are thousands of repositories on Docker Hub providing Nginx, including the official image. Unfortunately, the official image provides a version of Nginx that is not built with the geo_ip module.

Ubuntu’s default package repository for Nginx does provide a build including the geo_ip module. By modifying the Dockerfile for the official image, we can build a new image from Ubuntu with the required version of Nginx and include our custom configuration files.

FROM ubuntu
RUN apt-get -y install nginx

# copy custom configuration
COPY nginx.conf /etc/nginx/nginx.conf
COPY default /etc/nginx/sites-available/
COPY geoip /etc/nginx/geoip
COPY 404.html /usr/share/nginx/html/

# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log

# expose HTTP and HTTP ports
EXPOSE 80 443

CMD ["nginx", "-g", "daemon off;"]

Building and running this container locally, we can test that Nginx is configured correctly. The repository containing the Dockerfile and build artificats is located here.

1
2
3
4
5
6
7
8
9
10
[16:58:40 ~/code/final]$ docker build -t geo_ip .
Sending build context to Docker daemon 15.88 MB
Step 0 : FROM ubuntu
 ---> 91e54dfb1179
...
Step 9 : CMD nginx -g daemon off;
 ---> Using cache
 ---> 7bb6dbaafe3e
Successfully built 7bb6dbaafe3e
[16:58:50 ~/code/final]$ docker run -Pti geo_ip

With the custom image ready, we just need to deploy it somewhere…

Running Nginx on IBM Containers

IBM Bluemix supports deploying Docker containers alongside Cloud Foundry applications, allowing us to use the same cloud platform for running our custom region applications as providing the reverse proxy

Pushing pre-built images to the IBM Containers service is really as simple as creating a new tag and typing docker push.

Please read and follow the documentation about installing the command-line container management tools and authenticating with the remote service before attempting the commands below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[14:10:52 ~]$ docker tag geo_ip registry.ng.bluemix.net/jthomas/geo_ip
[14:10:59 ~]$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
geo_ip                                   latest              7bb6dbaafe3e        3 days ago          222.3 MB
registry.ng.bluemix.net/jthomas/geo_ip   latest              7bb6dbaafe3e        3 days ago          222.3 MB
[14:11:07 ~]$ cf ic login
** Retrieving client certificates from IBM Containers
** Storing client certificates in /Users/james/.ice/certs
Successfully retrieved client certificates
** Authenticating with registry at registry.eu-gb.bluemix.net
Successfully authenticated with registry
[14:24:25 ~]$ docker push registry.ng.bluemix.net/jthomas/geo_ip
The push refers to a repository [registry.ng.bluemix.net/jthomas/geo_ip] (len: 1)
Sending image list
Pushing repository registry.ng.bluemix.net/jthomas/geo_ip (1 tags)
...
Pushing tag for rev [7bb6dbaafe3e] on {https://registry.ng.bluemix.net/v1/repositories/jthomas/geo_ip/tags/latest}
[14:25:39 ~]$ cf ic images
REPOSITORY                                        TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
registry.ng.bluemix.net/jthomas/geo_ip            latest              7b1865be-778        About a minute ago   0 B
registry.ng.bluemix.net/ibmliberty                latest              2209a9732f35        3 weeks ago          263.6 MB
registry.ng.bluemix.net/ibmnode                   latest              8f962f6afc9a        3 weeks ago          178.9 MB
registry.ng.bluemix.net/ibm-mobilefirst-starter   latest              97513e56aaa7        3 weeks ago          464.9 MB
[14:26:43 ~]$ 

We can now use the IBM Bluemix dashboard to start a new container from our custom image, binding a public IP address and exposing ports.

Once the container starts, accessing the bound IP address shows the web page coming back with the region-specific application route.

Using DNS A records, we can now map our external URL to the IP address of the container. Users visiting this URL will be sent to the reverse proxy server which will then forward the request onto the correct region application.

Testing it all out…

Testing out the forwarding rules requires us to send HTTP requests from multiple regions. GeoWebView will run web browsers located in different geographies and show you the rendered page output.

Running the tool with our application’s web address, shows the following rendered page images.

We can see the browsers from the United States and Europe are sent to the correct region. The browser from South Africa is shown the custom error page.

Using Nginx we’ve configured a reverse proxy to route users, based upon their location, to applications running in different IBM Bluemix regions. We’re hosting the service on the same platform as our applications, using Docker. Most importantly, the whole process is transparent to the user, they aren’t forced to visit country-specific URLs.

Success!

Running One-off Tasks in Cloud Foundry

Whether making changes to a database schema, bulk importing data to initialise a database or setting up a connected service, there are often administrative tasks that needed to be carried out for an application to run correctly.

These tasks usually need finishing before starting the application and should not be executed more than once.

Previously, the CF CLI provided commands, tunnel and console, to help running one-off tasks manually. These commands were deprecated with the upgrade from v5 to v6 to discourage snowflake environments.

It is still possible, with a bit of hacking, to run one-off tasks manually from the application container.

A better way is to describe tasks as code and run them automatically during normal deployments. This results in applications that can be recreated without manual intervention.

We’ll look at both options before introducing a new library, oneoff, that automates running administration tasks for Node.js applications.

Running Tasks Manually

Local Environment

Rather than running administrative tasks from the application console, we can run them from a local development environment by remotely connecting to the bound services.

This will be dependent on the provisioned services allowing remote access. Many “built-in” platform services, e.g. MySQL, Redis, do not allow this.

Third-party services generally do.

Using the cf env command we can list service credentials for an application. These authentication details can often be used locally by connecting through a client library running in a local development environment.

For example, to access a provisioned Cloudant instance locally, we can grab the credentials and use with a Node.js client library.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[15:48:22 ~/code/sample]$ cf env sample-demo-app
Getting env variables for app sample-demo-app in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

System-Provided:
{
 "VCAP_SERVICES": {
  "cloudantNoSQLDB": [
   {
    "credentials": {
     "host": "1234-bluemix.cloudant.com",
     "password": "sample_password",
     "port": 443,
     "url": "https://1234-bluemix:sample_password@1234-bluemix.cloudant.com",
     "username": "1234-bluemix"
    }
....

[15:48:22 ~/code/sample]$ cat connect.js
var Cloudant = require('cloudant');

var me = '1234-bluemix';
var password = 'sample_password';

// Initialize the library with my account.
var cloudant = Cloudant({account:me, password:password});

cloudant.db.list(function(err, allDbs) {
  console.log('All my databases: %s', allDbs.join(', '))
  // Run administrative tasks
});
[15:48:22 ~/code/sample]$ node connect.js
All my databases: example_db, jasons_stuff, scores

Remote Environment

When provisioned services don’t allow external access, the cf-ssh project creates SSH access to application containers running within Cloud Foundry.

How does this work?!

cf-ssh deploys a new Cloud Foundry application, containing the same bits as your target application, with the same bound services.
This new application’s container does not start your web application as per normal. Instead, it starts an outbound reverse SSH tunnel to a public proxy.
The local cf-ssh client then launches an interactive ssh connect to the public proxy, which tunnels through to the application container.

See the explanation here for full details.

This approach will let you connect to services from within the Cloud Foundry platform environment.

This video from Stark & Wayne’s Dr. Nic shows the command in action…

IBM Bluemix Console (Java and Node.js)

This technique is only for the IBM Bluemix platform.

If you are deploying Node.js and Java applications on IBM Bluemix, the platform provides the following tools to assist with application management.

  • proxy: Minimal application management that serves as a proxy between your application and Bluemix.
  • devconsole: Enables the development console utility.
  • shell: Enables a web-based shell.
  • trace: (Node.js only) Dynamically set trace levels if your application is using log4js, ibmbluemix, or bunyan logging modules.
  • inspector: (Node.js only) Enables node inspector debugger.
  • debug: (Liberty only) Enables clients to establish a remote debugging session with the application.
  • jmx: (Liberty only) Enables the JMX REST Connector to allow connections from remote JMX clients

The tools are enabled by setting the environment variable (BLUEMIX_APP_MGMT_ENABLE) with the desired utilities.

1
$ cf set-env myApp BLUEMIX_APP_MGMT_ENABLE devconsole+shell+trace

Applications must be restarted for the changes to take effect.

If we enable the shell utility, the following web-based console will be available at https://your-app-name.mybluemix.net/bluemix-debug/shell.

Cloud Foundry Diego Runtime

Diego is the next-generation runtime that will power upcoming versions of Cloud Foundry. Diego will provide many benefits over the existing runtime, e.g. Docker support, including enabling SSH access to containers without the workarounds needed above.

Yay!

Follow the instructions here for details on SSH access to applications running on the new runtime.

Access to this feature will be dependent on your Cloud Foundry provider migrating to the new runtime.

Running Tasks Automatically

Manually running one-off administrative tasks for Cloud Foundry applications is a bad idea.

It affects your ability to do continuous delivery and encourages snowflake environments.

Alternatively, defining tasks as code means they can run automatically during normal deployments. No more manual steps are required to deploy applications.

There are many different libraries for every language to help you programmatically define, manage and run tasks.

With tasks defined as code, you need to configure your application manifest to run these automatically during deployments.

Cloud Foundry uses the command parameter, set in the manifest or through the command-line, to allow applications to specify a custom start command. We can use this parameter to execute the task library command during deployment.

The Cloud Foundry documentation also details these approaches, with slightly different implementations here and specifically for Ruby developers here.

Temporary Task Deploy

For applications which only need occasional administrative tasks, it’s often easier to push a temporary deploy with a custom start command. This deploy runs your tasks without then starting your application. Once the tasks have completed, redeploy your application normally, destroying the task instance.

The following command will deploy a temporary instance for this purpose:

1
$ cf push -c 'YOUR_TASK_LIB_COMMAND && sleep infinity' -i 1 --no-route

We’re overriding the default start command, setting it to run the command for our task library, e.g. rake db:migrate.

The sleep infinity command stops the application exiting once the task runner has finished. If this happens, the platform will assume that application has crashed and restart it.

Also, the task runner will not be binding to a port so we need to use the –no-route argument to stop the platform assuming the deploy has timed out.

Setting the deploy to a single instance stops the command being executed more than once.

Checking the logs to verify the task runner has finished correctly, we can now redeploy our application. Using the null start command will force the platform to use the buildpack default rather than our previous option.

1
$ cf push -c 'null'

Running Tasks Before Startup

If we’re regularly running administrative tasks, we should incorporate the task execution into our normal application startup. Once the task command has finished successfully, we start the application as normal.

Applications may have multiple instances running, we need to ensure the tasks are only executed by one instance.

The following custom start command will execute tasks during startup, using the CF_INSTANCE_ID environment variable to enforce execution at most-once.

[ $CF_INSTANCE_INDEX -eq 0 ]] && node lib/tasks/runner.js; node app.js

With this approach, tasks will be automatically executed during regular deployments without any manual intervention.

Hurrah!

Managing tasks for Node.js applications

If you’re running Node.js applications on Cloud Foundry, oneoff is a task library that helps you define tasks as code and integrates with the Cloud Foundry runtime. The module handles all the complexities with automating tasks during deployments across multi-instance applications.

oneoff provides the following features…

* ensure tasks are completed before application startup
* coordinating app instances to ensure at-most once task execution
* automagically discovering tasks from the task directory
* dependency ordering, ensure task a completes before task b starts
* parallel task execution
* ignore completed tasks in future deployments

Check it out to help make writing tasks as code for Node.js applications much easier!

Full details on usage are available in the README.

Conclusion

Running one-off tasks for application configuration is a normal part of any development project.

Carrying out these tasks manually used to be the norm, but with the devops movement we now prefer automated configuration rather manual intervention. Relying on manual configuration steps to deploy applications restricts our ability to implement continuous delivery.

Cloud Foundry is an opinionated platform, actively discouraging the creation of snowflake environments.

Whilst it is still possible to manually run administrative tasks, either by connecting to bound services locally or using a remote console, it’s preferable to describe our tasks as code and let the platform handle it.

Using custom start commands, we can deploy applications which run tasks automatically during their normal startup procedure.

GeoPix Live Photos

Andrew Trice wrote a great sample application for IBM Bluemix called GeoPix.

GeoPix uses the IBM MobileFirst services to provide a native iOS application which allows users to capture images from their mobile phones, storing them on the local device with automatic syncing to the cloud when online.

Using a web application, the user can view their images over a map based upon their location when the photo was taken.

I’ve been using the demonstration to highlight the mobile capabilities of IBM Bluemix and had an idea for an enhancement…

Could the web page update with new pictures without having to refresh the page?

Looking at the source code, the web application is a Node.js application using the Leaflet JavaScript library to create interactive maps. Images captured from mobile devices are synchronised to a remote CouchDB database. When the user visits the GeoPix site, the application queries this database for all mobile images and renders the HTML using the Jade templating language.

Adding support for live photos will require two new features…

  • Triggering backend events when new photos are available
  • Sending these photos in real-time to the web page

Change Notifications Using CouchDB

CouchDB comes with built-in support for listening to changes in a database, change notifications. The _changes feed for a database is an activity stream publishing all document modifications.

GeoPix uses the following CouchDB client library, to interact with our database from NodeJS. This library provides an API to start following database changes and register callbacks for updates.

Modifying our application code, upon connecting to the CouchDB database, we register a change notification handler. We follow all changes that occur in the future (since: “now”) and include the full document contents in the change event (include_docs: true).

1
2
3
4
5
6
7
8
9
10
Cloudant({account:credentials.username, password:credentials.password}, function(err, cloudant) {
    var geopix = cloudant.use(database);
    var feed = geopix.follow({include_docs: true, since: "now"});

    feed.on('change', function (change) {
      // ....we can now send this data to the web pages
    });

    feed.follow();
})

Now, every time a user sync their local photos to the cloud, the registered callback will be executed.

How do we send new photos to the web page over a real-time stream?

Real-time Web with Socket.IO

Introducing Socket.IO

Socket.IO enables real-time bidirectional event-based communication.
It works on every platform, browser or device, focusing equally on reliability and speed.

Sounds great!

By embedding this library into our application, we can open a real-time event stream between the server and client. This channel will be used by the client to listen for new images and then update the page.

The library has great documentation and provides both server and client modules. It also integrates with ExpressJS, the web framework used in GeoPix. Socket.IO can use either WebSocket or long-polling transport protocols.

Socket.IO supports running under ExpressJS with minimal configuration, here are the changes needed to start our real-time stream in GeoPix:

1
2
3
4
5
6
7
8
9
10
11
12
13
var express = require('express');
var app = express();
var server = require('http').Server(app);
var io = require('socket.io')(server);

// ...snipped out the app routes for express

io.on('connection', function (socket) {
    console.log('New Client WSS Connection.')
});

var port = (process.env.VCAP_APP_PORT || 3000);
server.listen(port);

When a document change event is fired, executing the handle we registered above, we want to send this data to all connected clients.

Using the emit call from the server-side API will do this for us.

1
2
3
feed.on('change', function (change) {
    io.sockets.emit('image', change);
});

Now we’re sending changes to the clients, we need to modify the client-side to listen for events and update the page.

Socket.IO provides a JavaScript client library that exposes a simple API for listening to events from the server-side stream. Once we’ve included the script tag pointing to the client library, we can register a callback for image events and update the DOM with the new elements.

We’re sending the full database document associated with each photo to the client. The raw image bytes are stored as an attachment.

1
2
3
4
5
6
7
8
9
10
11
12
13
var socket = io(); // TIP: io() with no args does auto-discovery
socket.on('connect', function () {
    console.log('WSS Connected');

    socket.on('image', function (image) { // TIP: you can avoid listening on `connect` and listen on events directly too!
        var attachment = Object.keys(image.doc._attachments)[0]
        var url = "/image/" + image.doc._id + "/" + attachment;
        add_new_image(url, image.doc.clientDate, 'latitude: '
            + image.doc.latitude + ', longitude: '
            + image.doc.longitude + ', altitude: '
            + image.doc.altitude);
    });
});

…and that’s it! Now our web pages will automatically update with new photos whenever the mobile application syncs with the cloud.

CouchDB + Socket.IO = Real-time Awesome!

Adding real-time photos to our application was amazingly simple by combining CouchDB with Socket.IO.

CouchDB’s _changes API provided an easy way to follow all modifications to database documents in real-time. Socket.IO made the configuration and management of real-time event streams between our server and client straightforward.

With minimal code changes, we simply connected these two technologies to create a real-time photo stream for our GeoPix application. Awesome.

AlchemyAPI & Updated Watson Nodes for Node-RED

I’ve recently been working on a number of updates to the Node-RED nodes for the IBM Bluemix platform…

Highlights below:

New AlchemyAPI Nodes

There are two new nodes (Feature Extract and Image Extract) in the package, allowing users to call services from the AlchemyAPI platform.

  • Feature Extract. This node will analyse external URLs, HTML or text content with features for text-based analysis from the AlchemyAPI service, e.g. keywords, sentiment, relationships, etc.

  • Image Analysis. This node will analyse images, passed in as external URLs or raw image bytes, to extract faces, content and URLs.

Configuration for each node is available through the node editor panel.

For full details on all the capabilities of the AlchemyAPI platform, please see their documentation.

Updated IBM Watson Nodes

With the recent changes to the IBM Watson services, there were a number of changes needed to support the API changes. All the IBM Watson nodes now work with the GA versions of the services.

Users must ensure they are using GA versions of the service with the nodes. Details on migration steps are available on the IBM Watson blog post about the updates.

Running Locally

When running Node-RED on IBM Bluemix, credentials for the services bound to the application are automatically registered. Previously, running the nodes outside of IBM Bluemix required complex configuration to register service credentials. With this release, users will be prompted to input the service credentials in the node editor panel if the application isn’t running on IBM Bluemix. Much easier!

If you have questions or encounter issues, please ask over on Stackoverflow or raise issues in Github

Debugging Cloud Foundry Stack Issues

Recent changes to the Cloud Foundry stacks supported by IBM Bluemix have led to a number of issues for users. I’ve helped users diagnose and fix issues that have occurred due to a mistmatches between the platform stack, applications and the buildpack. Learning a number of techniques for helping to discover and resolve these issues and I wanted to share them with everyone else.

Running on Cloud Foundry’s Platform-as-a-Service solution, we take for granted that low-level concepts like operating systems are abstracted away from the developer.

However, when we run into issues it can be necessary to jump into the weeds and find out what’s going on under the hood…

What are Cloud Foundry “stacks”?

According to the documentation

A stack is a prebuilt root filesystem (rootfs) which works in tandem with a buildpack and is used to support running applications.

Think of the stack as the underlying operating-system running your application. This will be combined with the buildpack to instantiate the runtime environment.

Most users don’t have to care which stack they are running on.

However, if your application needs a specific version of a system library or you want to verify a specific command line application is installed, you may need to dig deeper…

What “stacks” does my platform support?

Using the Cloud Foundry CLI, issue the following command to see what stacks are available on the platform.

1
2
3
4
5
6
7
8
[16:27:30 ~]$ cf stacks
Getting stacks in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

name         description
lucid64      Ubuntu 10.04
seDEA        private
cflinuxfs2   Ubuntu 14.04.2 trusty

Stack information contains the unique name for each stack and the underlying operating system version.

Which “stack” is my application running on?

Since v6.11.0, the stack for an application has been shown in the CLI application info output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[16:34:39 ~]$ cf app debug-testing
Showing health and status for app debug-testing in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: debug-testing.mybluemix.net
last uploaded: Tue Jun 16 15:47:21 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM)

     state     since                    cpu    memory           disk           details
#0   running   2015-06-30 08:53:57 PM   0.0%   242.5M of 512M   196.8M of 1G

How can I choose the “stack” my application runs on?

Users can set the stack for an application using the -s command-line parameter during deployment. The stack identifier should match one of the names shown in the output from the cf stacks command.

1
$ cf push -s stack_identifier

How are the “stacks” defined?

This Github repository contains the source files for building the stacks. There’s a Dockerfile for the current cflinuxfs2 stack to build the image used in Cloud Foundry.

How can I poke around inside a “stack” locally?

Using Docker, we can easily pull down the same “base” operating system used for a specifc “stack” and run locally.

For the cflinuxfs2 stack, we can pull down the Ubuntu Trusty image and run a terminal inside it.

1
2
$ docker pull ubuntu:trusty
$ docker run -i -t ubuntu:trusty /bin/bash

How can I easily migrate existing applications to a new stack?

Rather than having to re-deploy each application separately, there’s a great CF CLI plugin to automatically migrate all your applications from lucid64 to cflinuxfs2.