Debugging Cloud Foundry Stack Issues

Recent changes to the Cloud Foundry stacks supported by IBM Bluemix have led to a number of issues for users. I’ve helped users diagnose and fix issues that have occurred due to a mistmatches between the platform stack, applications and the buildpack. Learning a number of techniques for helping to discover and resolve these issues and I wanted to share them with everyone else.

Running on Cloud Foundry’s Platform-as-a-Service solution, we take for granted that low-level concepts like operating systems are abstracted away from the developer.

However, when we run into issues it can be necessary to jump into the weeds and find out what’s going on under the hood…

What are Cloud Foundry “stacks”?

According to the documentation

A stack is a prebuilt root filesystem (rootfs) which works in tandem with a buildpack and is used to support running applications. Cloud Foundry Concepts https://docs.cloudfoundry.org/concepts/stacks.html

Think of the stack as the underlying operating-system running your application. This will be combined with the buildpack to instantiate the runtime environment.

Most users don’t have to care which stack they are running on.

However, if your application needs a specific version of a system library or you want to verify a specific command line application is installed, you may need to dig deeper…

What “stacks” does my platform support?

Using the Cloud Foundry CLI, issue the following command to see what stacks are available on the platform.

[16:27:30 ~]$ cf stacks
Getting stacks in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

name         description
lucid64      Ubuntu 10.04
seDEA        private
cflinuxfs2   Ubuntu 14.04.2 trusty

Stack information contains the unique name for each stack and the underlying operating system version.

Which “stack” is my application running on?

Since v6.11.0, the stack for an application has been shown in the CLI application info output.

[16:34:39 ~]$ cf app debug-testing
Showing health and status for app debug-testing in org james.thomas@uk.ibm.com / space dev as james.thomas@uk.ibm.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: debug-testing.mybluemix.net
last uploaded: Tue Jun 16 15:47:21 UTC 2015
stack: lucid64
buildpack: SDK for Node.js(TM)

     state     since                    cpu    memory           disk           details
#0   running   2015-06-30 08:53:57 PM   0.0%   242.5M of 512M   196.8M of 1G

How can I choose the “stack” my application runs on?

Users can set the stack for an application using the -s command-line parameter during deployment. The stack identifier should match one of the names shown in the output from the cf stacks command.

$ cf push -s stack_identifier 

How are the “stacks” defined?

This Github repository contains the source files for building the stacks. There’s a Dockerfile for the current cflinuxfs2 stack to build the image used in Cloud Foundry.

How can I poke around inside a “stack” locally?

Using Docker, we can easily pull down the same “base” operating system used for a specifc “stack” and run locally.

For the cflinuxfs2 stack, we can pull down the Ubuntu Trusty image and run a terminal inside it.

$ docker pull ubuntu:trusty
$ docker run -i -t ubuntu:trusty /bin/bash

How can I easily migrate existing applications to a new stack?

Rather than having to re-deploy each application separately, there’s a great CF CLI plugin to automatically migrate all your applications from lucid64 to cflinuxfs2.