OpenWhisk’s Python runtime includes popular third-party libraries like requests, scrapy and simplejson. Developers don’t have to manually install packages to use those libraries.
Great, but what about using other libraries that aren’t pre-installed?
In a previous blog post, we showed how to deploy Node.js actions from zip files containing third-party modules. These modules are then made available in the Node.js runtime.
Recent updates to OpenWhisk allow us to use the same approach with the Python runtime!
Python packages can be installed using the pip tool. This can be used to install individual packages or a series of dependencies from an external file.
pip defaults to installing packages in a global location (site-packages) which is shared between all users. This can cause issues when different projects require different versions of the same package.
virtualenv is a tool that solves this issue by creating virtual python environments for projects. The virtual environment includes a custom
site-packages folder to install packages into.
1 2 3 4 5
OpenWhisk recently added support for using virtualenv in the Python runtime.
custom packages on openwhisk
OpenWhisk actions can be created from a zip file containing source files and other resources.
If the archive includes a virtual Python environment folder, the platform runs the
./bin/activate_this.py script before executing Python actions. This script modifies the module search path to include the local
This will only happen during “cold” activations.
This feature comes with the following restrictions.
- Virtual Python environment must be in a folder called
virtualenvunder the top-level directory.
- Packages must be available for the Python runtime being used in OpenWhisk (2.7 or 3.6).
Let’s look at an example of building an OpenWhisk Python action which uses an external Python package.
Python Package Example
The pyjokes package provides a library for generating (terrible) jokes for programmers. Let’s turn this package into an API (Jokes-as-a-Service!) using the Python runtime on OpenWhisk.
Start by creating a new directory for your project and set up the virtual Python environment.
1 2 3 4 5 6 7 8 9 10 11 12 13
In the project directory, create a new file (
__main__.py) and paste the following code.
1 2 3 4
Check the script works with the Python intepreter.
1 2 3 4
virtualenv folder and Python script to a new zip file.
1 2 3 4 5 6 7 8
Create a new OpenWhisk action for the Python runtime using the
Invoking our new action will return (bad) jokes on-demand using the third-party Python package.
1 2 3 4
Installing Packages With Docker
In the example above, the Python runtime used in development (v3.6) matched the OpenWhisk runtime environment. Packages installed using
virtualenv must be for the same major and minor versions of the Python runtime used by OpenWhisk.
OpenWhisk publishes the runtime environments as Docker images on Docker Hub.
Running containers from those runtime images provides a way to download packages for the correct environment.
1 2 3 4 5 6 7 8 9 10 11
This will leave you a
virtualenv folder in the current directory with packages for the correct Python runtime.
Speeding Up Deployments
Peeking inside the
virtualenv folder reveals a huge number of files to set up the virtual Python environment. If we just want to use a third-party package from the local
site-packages folder, most of those files are unnecessary.
Adding this entire folder to the zip archive will unnecessarily inflate the file size. This will slow down deployments and increase execution time for cold activations. OpenWhisk also has a maximum size for action source code of 48MB.
Manually including individual
site-packages folders, rather than the entire
virtualenv directory, will ensure the archive file only contains packages being used. We must also add the Python script (
virtualenv/bin/activate_this.py) executed by OpenWhisk to modify the module search path.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
The archive file is now less than ten kilobytes! 🏃
With The Serverless Framework
The Serverless Framework is a popular open-source framework for building serverless applications. This framework handles the configuration, packaging and deployment of your serverless application.
Using the application configuration file for the framework, users can add
exclude parameters to control the contents of the archive file before deployment.
Here’s an example of the configuration needed to only include the necessary files for the application above.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Python has a huge community of third-party packages for everything from parsing JSON, making HTTP requests and even generating jokes. OpenWhisk already provided a number of the most popular packages within the Python runtime.
Users can install additional packages locally using the
virtualenv tools. Bundling those files within the deployment archive means they are extracted into the OpenWhisk Python runtime environment.
Recent changes to the Python runtime allows the platform to automatically add local package folders to the module search path.
This means Python functions running on OpenWhisk can now use any third-party library as if it was installed globally.