Debugging serverless applications in production is often reliant on application logs, due to having no access to the runtime environment. No SSHing into the machine and attaching a debugger to a process or using strace to dump system calls.
Storing, searching and analysing serverless application logs is crucial to diagnosing and fixing bugs on serverless platforms.
The “ELK Stack” has become a popular solution for managing applications logs. Combining three open-source projects (ElasticSearch, Logstash and Kibana), this solution provides a scalable platform for importing, storing and searching application logs.
How can we use the ELK stack to manage logs for serverless applications running on Apache OpenWhisk?
ELK and OpenWhisk
In traditional application runtimes, like a VM or a Docker container, a background agent is used to automatically forward application and system logs to the ingestion service for the ELK stack.
However, serverless applications run in an ephemeral environment. Runtimes are instantiated on-demand per request and destroyed after the function returns. These runtimes do not support the use of background agents.
One solution for this is the custom OpenWhisk plugin for Logstash. This plugin polls the platform for new logs and automatically ingests them into ElasticSearch.
But what if you are using a hosted ELK service that does not support installing custom plugins?
OpenWhisk Logstash Forwarder
“OpenWhisk Logstash Forwarder” is designed for this scenario. It can ingest logs into ElasticSearch using standard Logstash input plugins.
https://github.com/jthomas/openwhisk-logstash-forwarder
This project contains an OpenWhisk action which acts as a “serverless” version of the logstash-forwarder agent. When the action executes, it retrieves all new logs from a user-provided list of actions to monitor. Log messages are pushed into Logstash using the Lumberjack protocol.
The action is connected to an alarm trigger feed with a customisable schedule. This event source will ensure all logs are forwarded on a regular schedule.
Demo
In this example, the developer has the serverless logstash forwarder agent deployed in their workspace. The agent is configured to monitor logs from the forecast action. The alarm trigger feed is connected to the monitoring action and runs once per minute.
Invoking the forecast action generates log messages to be ingested.
When the alarm trigger feed next fires, the monitoring action is executed. It retrieves log messages generated by new forecast activations and pushes those logs into the configured ELK instance.
Opening Kibana and refreshing the monitoring dashboard, new log messages are shown as individual documents. Selecting the individual documents shows the log message contents with activation record details.
Source Code
The source code for this project is now available on Github:
https://github.com/jthomas/openwhisk-logstash-forwarder
See the installation instructions for how to deploy this project on an OpenWhisk platform.
This project needs an instance of OpenWhisk platform and an ELK-stack service accessible on a public IP address.
This project can be deployed using The Serverless Framework or the OpenWhisk CLI.