Automating this process has the following benefits…
- Removes the chance of human errors compared to the previously manual validation process.
- Allows me to validate new releases without access to my dev machine.
- Usable by all committers by hosting as an external serverless web app.
Automating release candidate validation makes it easier for project committers to participate in release voting. This should make it faster to get necessary release votes, allowing us to ship new versions sooner!
apache software foundation
An Apache release is a set of valid & signed artifacts, voted on by the appropriate PMC and distributed on the ASF’s official release infrastructure.
Releasing a new software version requires the release manager to create a release candidate from the project source files. Source archives must be cryptographically signed by the release manager. All source archives for the release must be comply with strict criteria to be considered valid release candidates. This includes (but is not limited to) the following requirements:
- Checksums and PGP signatures for source archives are valid.
- LICENSE, NOTICE and DISCLAIMER files included and correct.
- All source files have license headers.
- No compiled archives bundled in source archives.
Release candidates can then be proposed on the project mailing list for review by members of the Project Management Committee (PMC). PMC members are eligible to vote on all release candidates. Before casting their votes, PMC members are required to check release candidate meets the requirements above.
If a minimum of three positive votes is cast (with more positive than negative votes), the release passes! The release manager can then move the release candidate archives to the release directory.
apache openwhisk releases
As a committer and PMC member on the Apache OpenWhisk project, I’m eligible to vote on new releases.
Apache OpenWhisk (currently) has 52 separate source repositories under the project on GitHub. With a fast-moving open-source project, new releases candidate are constantly being proposed, which all require the necessary number of binding PMC votes to pass.
Manually validating release candidates can be a time-consuming process. This can make it challenging to get a quorum of binding votes from PMC members for the release to pass. I started thinking how I could improve my productivity around the validation process, enabling me to participate in more votes.
Would it be possible to automate some (or all) of the steps in release candidate verification? Could we even use a serverless application to do this?
apache openwhisk release verifier
Spoiler Alert: YES! I ended up building a serverless application to do this for me.
It is available at https://apache.jamesthom.as/
Source code for this project is available here.
IBM Cloud Functions is used to run the serverless backend for the web application. This means Apache OpenWhisk is being used to validate future releases of itself… which is awesome.
HTML, JS and CSS files are served by Github Pages from the project repository.
Backend APIs are Apache OpenWhisk actions running on IBM Cloud Functions.
Both the front-page and API are served from a custom sub-domains of my personal domain.
available release candidates
When the user loads the page, the drop-down list needs to contain the current list of release candidates from the ASF development distribution site.
This information is available to the web page via the https://apache-api.jamesthom.as/api/versions endpoint. The serverless function powering this API parses that live HTML page (extracting the current list of release candidates) each time it is invoked.
1 2 3 4 5 6 7 8 9 10 11 12 13
release candidate version info
Release candidates may have multiple source archives being distributed in that release. Validation steps need to be executed for each of those archives within the release candidate.
Once a user has selected a release candidate version, source archives to validate are shown in the table. This data is available from the https://apache-api.jamesthom.as/api/versions/VERSION endpoint. This information is parsed from the HTML page on the ASF site.
1 2 3 4 5 6 7 8 9 10 11
Having selected a release candidate version, clicking the ”Validate” button will start validation process. Triggering the https://apache-api.jamesthom.as/api/versions/VERSION/validate endpoint will run the serverless function used to execute the validation steps.
This serverless function will carry out the following verification steps…
checking download links
All the source archives for a release candidate are downloaded to temporary storage in the runtime environment. The function also downloads the associated SHA512 and PGP signature files for comparison. Multiple readable streams can be created from the same file path to allow the verification steps to happen in parallel, rather than having to re-download the archive for each task.
checking SHA512 hash values
SHA512 sums are distributed in a text file containing hex strings with the hash value.
1 2 3
The serverless function needs to dynamically compute the hash for the source archive and compare the hex bytes against the text file contents. Node.js comes with a built-in crypto library making it easy to create hash values from input streams.
This is the function used to compute and compare the hash values.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
validating PGP signatures
Node.js’ crypto library does not support validating PGP signatures.
- Message contents to check.
- PGP signature for the message.
- Public key for the private key used to sign the release.
The “message” to check is the source archive. PGP signatures come from the
.asc files located in the release candidate directory.
1 2 3 4 5 6
Public keys used to sign releases are stored in the root folder of the release directory for that project.
This function is used to implement the signature checking process.
1 2 3 4 5 6 7 8 9 10 11 12 13
scanning archive files
Using the node-tar library, downloaded source archives are extracted into the local runtime to allow scanning of individual files.
LICENSE.txt, DISCLAIMER.txt and NOTICE.txt files are checked to ensure correctness. An external NPM library is used to check all files in the archive for binary contents. The code also scans for directory names that might contain third party libraries (
capturing validation logs
It is important to provide PMC members with verifiable logs on the validation steps performed. This allows them to sanity check the steps performed (including manual validation). This verification text can also be provided in the voting emails as evidence of release candidate validity.
Once all the validation tasks have been executed - the results are returned to the front-end as a JSON response. The client-side JS parses these results and updates the validation table. Validation logs are shown in a collapsible window.
Using visual emojis for pass and failure indicators for each step - the user can easily verify whether a release passes the validation checks. If any of the steps have failed, the validation logs provide an opportunity to understand why.
This is not the only tool that can automate checks needed to validate Apache Software Foundation releases.
Another community member has also built a bash script (rcverify.sh) that can verify releases on your local machine. This script will automatically download the release candidate files and run many of the same validation tasks as the remote tool locally.
There is also an existing tool (Apache Rat) from another project that provides a Java-based application for auditing license headers in source files.
Getting new product releases published for an open-source project under the ASF is not a simple task for developers used to pushing a button on Github! The ASF has a series of strict guidelines on what constitutes a release and the ratification process from PMC members. PMC members need to run a series of manual verification tasks before casting binding votes on proposed release candidates.
This can be a time-consuming task for PMC members on a project like Apache OpenWhisk, with 52 different project repositories all being released at different intervals. In an effort to improve my own productivity around this process, I started looking for ways to automate the verification tasks. This would enable me to participate in more votes and be a “better” PMC member.
This led to building a serverless web application to run all the verification tasks remotely, which is now hosted at https://apache.jamesthom.as. This tool uses Apache OpenWhisk (provided by IBM Cloud Functions), which means the project is being used to verify future releases of itself! I’ve also open-sourced the code to provide an example of how to use the platform for automating tasks like this.
With this tool and others listed above, verifying new Apache OpenWhisk releases has never been easier!