Reading about the library, I wanted to test it out with a simple task… 🧐
Unfortunately, most of the documentation and example code provided uses the library in a browser. Project utilities provided to simplify loading and using pre-trained models have not yet been extended with Node.js support. Getting this working did end up with me spending a lot of time reading the Typescript source files for the library. 👎
However, after a few days’ hacking, I managed to get this completed! Hurrah! 🤩
Before we dive into the code, let’s start with an overview of the different TensorFlow libraries.
TensorFlow is an open-source software library for machine learning applications. TensorFlow can be used to implement neural networks and other deep learning algorithms.
Released by Google in November 2015, TensorFlow was originally a Python library. It used either CPU or GPU-based computation for training and evaluating machine learning models. The library was initially designed to run on high-performance servers with expensive GPUs.
Recent updates have extended the software to run in resource-constrained environments like mobile devices and web browsers.
Tensorflow Lite, a lightweight version of the library for mobile and embedded devices, was released in May 2017. This was accompanied by a new series of pre-trained deep learning models for vision recognition tasks, called MobileNet. MobileNet models were designed to work efficiently in resource-constrained environments like mobile devices.
TensorFlow.js was recently extended to run on Node.js, using an extension library called
The Node.js extension is an alpha release and still under active development.
Importing Existing Models Into TensorFlow.js
Existing TensorFlow and Keras models can be executed using the TensorFlow.js library. Models need converting to a new format using this tool before execution. Pre-trained and converted models for image classification, pose detection and k-nearest neighbours are available on Github.
Using TensorFlow.js in Node.js
Installing TensorFlow Libraries
TensorFlow.js can be installed from the NPM registry.
@tensorflow/tfjs- Core TensorFlow.js library
@tensorflow/tfjs-node- TensorFlow.js Node.js extension
@tensorflow/tfjs-node-gpu- TensorFlow.js Node.js extension with GPU support
1 2 3
Both Node.js extensions use native dependencies which will be compiled on demand.
Loading TensorFlow Libraries
1 2 3 4 5
Loading TensorFlow Models
In the project’s README, the following example code is used to load the model.
1 2 3 4
One of the first challenges I encountered was that this does not work on Node.js.
Looking at the source code, the
mobilenet library is a wrapper around the underlying
tf.Model class. When the
load() method is called, it automatically downloads the correct model files from an external HTTP address and instantiates the TensorFlow model.
The Node.js extension does not yet support HTTP requests to dynamically retrieve models. Instead, models must be manually loaded from the filesystem.
After reading the source code for the library, I managed to create a work-around…
Loading Models From a Filesystem
Rather than calling the module’s
load method, if the
MobileNet class is created manually, the auto-generated
path variable which contains the HTTP address of the model can be overwritten with a local filesystem path. Having done this, calling the
load method on the class instance will trigger the filesystem loader class, rather than trying to use the browser-based HTTP loader.
1 2 3 4
Awesome, it works!
But how where do the models files come from?
Models for TensorFlow.js consist of two file types, a model configuration file stored in JSON and model weights in a binary format. Model weights are often sharded into multiple files for better caching by browsers.
Looking at the automatic loading code for MobileNet models, models configuration and weight shards are retrieved from a public storage bucket at this address.
The template parameters in the URL refer to the model versions listed here. Classification accuracy results for each version are also shown on that page.
According to the source code, only MobileNet v1 models can be loaded using the
The HTTP retrieval code loads the
model.json file from this location and then recursively fetches all referenced model weights shards. These files are in the format
Downloading Models Manually
Saving all model files to a filesystem can be achieved by retrieving the model configuration file, parsing out the referenced weight files and downloading each weight file manually.
I want to use the MobileNet V1 Module with 1.0 alpha value and image size of 224 pixels. This gives me the following URL for the model configuration file.
Once this file has been downloaded locally, I can use the
jq tool to parse all the weight file names.
1 2 3 4 5
sed tool, I can prefix these names with the HTTP URL to generate URLs for each weight file.
1 2 3 4 5
curl commands, I can then download all of these files to my local directory.
This example code is provided by TensorFlow.js to demonstrate returning classifications for an image.
1 2 3 4
This does not work on Node.js due to the lack of a DOM.
classify method accepts numerous DOM elements (
image) and will automatically retrieve and convert image bytes from these elements into a
tf.Tensor3D class which is used as the input to the model. Alternatively, the
tf.Tensor3D input can be passed directly.
Rather than trying to use an external package to simulate a DOM element in Node.js, I found it easier to construct the
Generating Tensor3D from an Image
Reading the source code for the method used to turn DOM elements into Tensor3D classes, the following input parameters are used to generate the Tensor3D class.
1 2 3 4
pixels is a 2D array of type (Int32Array) which contains a sequential list of channel values for each pixel.
numChannels is the number of channel values per pixel.
Creating Input Values For JPEGs
This will return a
Uint8Array with four channel values (
RGBA) for each pixel (
width * height). The MobileNet model only uses the three colour channels (
RGB) for classification, ignoring the alpha channel. This code converts the four channel array into the correct three channel version.
1 2 3 4 5 6 7 8 9
MobileNet Models Input Requirements
The MobileNet model being used classifies images of width and height 224 pixels. Input tensors must contain float values, between -1 and 1, for each of the three channels pixel values.
Input values for images of different dimensions needs to be re-sized before classification. Additionally, pixels values from the JPEG decoder are in the range 0 - 255, rather than -1 to 1. These values also need converting prior to classification.
TensorFlow.js has library methods to make this process easier but, fortunately for us, the
tfjs-models/mobilenet library automatically handles this issue! 👍
Developers can pass in Tensor3D inputs of type
int32 and different dimensions to the
classify method and it converts the input to the correct format prior to classification. Which means there’s nothing to do… Super 🕺🕺🕺.
MobileNet models in Tensorflow are trained to recognise entities from the top 1000 classes in the ImageNet dataset. The models output the probabilities that each of those entities is in the image being classified.
The full list of trained classes for the model being used can be found in this file.
tfjs-models/mobilenet library exposes a
classify method on the
MobileNet class to return the top X classes with highest probabilities from an image input.
predictions is an array of X classes and probabilities in the following format.
1 2 3 4
Having worked how to use the TensorFlow.js library and MobileNet models on Node.js, this script will classify an image given as a command-line argument.
- Save this script file and package descriptor to local files.
testing it out
Download the model files to a
mobilenetdirectory using the instructions above.
Install the project dependencies using NPM
- Download a sample JPEG file to classify
- Run the script with the model file and input image as arguments.
If everything worked, the following output should be printed to the console.
1 2 3 4
The image is correctly classified as containing a Panda with 99.93% probability! 🐼🐼🐼
Having been released as a browser-based library, TensorFlow.js has now been extended to work on Node.js, although not all of the tools and utilities support the new runtime. With a few days’ hacking, I was able to use the library with the MobileNet models for visual recognition on images from a local file.
Getting this working in the Node.js runtime means I now move on to my next idea… making this run inside a serverless function! Come back soon to read about my next adventure with TensorFlow.js. 👋