For the last few months, I’ve been using the ELK stack to collect logs from my Cloud Foundry applications. This service has been deployed on IBM Bluemix using a Docker container, previously detailed in this blog post, and running happily until it ran into issues this week.
Trying to load the Kibana web application, the server was returning connection refused errors. Looking at the container in the IBM Bluemix dashboard showed no obvious signs of issues. Reviewing the container log output uncovered nothing indicating what had failed.
Hmmm…
Fixing this issue would require me to start debugging from within the live container, but how?
This container image had not included an SSH daemon that would allow remote access over SSH.
Looking over the documentation for the IBM Containers plugin for the Cloud Foundry CLI, I noticed the exec command.
Docker exec allows a user to spawn a process inside their Docker container via the Docker API and CLI.
Since Docker 1.3, released in October 2014, the exec command has allowed users to run new commands within existing containers.
The IBM Containers implementation now supports this Docker command.
Using the IBM Containers plugin for the Cloud Foundry CLI, I can find the container id for the instance I want to debug and then start a bash shell to start resolving my issue.
$ cf ic ps
$ cf ic exec -it <container_id> /bin/bash
Having a live shell to my container allowed me to resolve the issue within a few minutes, without having to affect the running state of the container. This command also removes the need to keep an SSH daemon running on containers for remote access.
For more information on the subset of Docker commands supported by IBM Containers, see the following documentation.