Join our Discord server for community support and discussions

Add ML-Assisted Handwritten Digit Recognizer with C# to your Mobile Apps

18 Jan 2023

This article will explore loading a pre-trained ONNX model, trained on the popular MNIST dataset, into an application built with the Uno Platform. By loading a pre-trained ONNX model, we can leverage the power of machine learning to add advanced functionality to our mobile, .NET, and Uno Platform-built applications, such as image classification or handwriting recognition. We will provide a step-by-step guide on how to load an ONNX model into an Uno Platform application and explain how to use the model to make predictions on new data. By the end of this article, you will have a solid understanding of how to integrate machine learning into your Uno Platform applications.

To get a better understanding of training, evaluating and converting an ML model trained with the MNIST dataset to an ONNX-compatible format or how to load an ONNX Model, perform processing tasks on both the input and output and connect the prediction pipeline to the frontend UI, check out Part 1 and Part 2 of this series.

Getting Started

The objective of the MNIST classifier model is to recognise handwritten numerical digits from 0-9. The model accepts as input an image of size 14 * 14 and outputs an integer between 0-9 as an output. The codebase uses on sample image.

NuGet Packages Used

The following NuGet packages were installed in the Uno platform application project for this code base.

INFERENCING WITH TensorFlowMNISTClassifier

The TensorFlowMNISTClassifier C# file houses the class–based code that will take care of loading, processing, and prediction tasks.

STEP 1: LOAD EMBEDDED RESOURCES

In the C# file mentioned above, a private function called InitTask handles the fetching, loading from memory stream and storing in appropriate C# variables of the following embedded resources

ONNX MNIST Model
Handwriting Sample.

The code snippet above loads the model and sample image saved as embedded resources and stores each as byte arrays. With the byte array obtained from loading the model, an ONNX Runtime inference session is created and that is what will be used to make the prediction in subsequent code snippets.

STEP 2: Pre Processing the Input

With the byte array gotten from the sample image, a bitmap is created using the byte array. This bitmap will undergo several transformations to make the result compatible with the Input format expected by the ONNX model.

The bitmap needed to be rescaled to the expected 14 * 14 input size, flattened and normalized with the range 0-1 as shown in the code snippet above. The result is a float array. Using the Netron App, I confirmed the shape of both the expected input and output along with the names of the input and output as this will be needed for later.

The float array is used to create a Dense Tensor of generic type float which will be fed to the prediction session as input.

				
					var input = new DenseTensor<float>(channelData, new int[] { 1, 28*28 });

STEP 3: Prediction

After the creation of the Dense Tensor as input, the input is fed to the session the output is the results which will need to be filtered to check for value to which a key string equals the model’s output name as shown in the code snippet below:

				
					using var results = _session.Run(new List<NamedOnnxValue> { Named OnnxValue.CreateFromTensor (ModelInputName, input) }); 

var output = results.FirstOrDefault(i => i.Name.StartsWith(“dense”));

STEP 4: Post Processing

The interpretation of results is achieved in this step. Once the output required is filtered out as described in the previous step, the Human readable output which is passed to the frontend is obtained by first converting the output to a list of floats, getting the Max value in that list and getting the index of the max value which corresponds to the prediction from the ML model. The code snippet below illustrates the post-processing code.

				
					var output = results.FirstOrDefault(I => I.Name.StartsWith(“dense”)); 

if (output == null) 
    return “Unknown”; 

var scores = output.AsTensor<float>().ToList(); 
var highestScore = scores.Max(); 
var highestScoreIndex = scores.IndexOf(highestScore); 

return highestScoreIndex.ToString();

STEP 5: Connecting to the Frontend

Building up from the previous article, I used the 3rd tab bar items to connect the inference session for this article with the frontend. The UX for the MNIST Classifier is a three-rowed Grid consisting of a text block for displaying the page description, the sample image and finally, the button meant to trigger the prediction session in the code behind. Below are the relevant screenshots.

You will notice the result prediction is wrong. This is because the article aimed to demonstrate a model explicitly trained for the task at hand rather than focusing on achieving the highest level of accuracy.

About Uno Platform

For those new to the Uno Platform, it allows for creating pixel-perfect, single-source C# and XAML apps that run natively on Windows, iOS, Android, macOS, Linux and Web via WebAssembly. In addition, it offers Figma integration for design-development handoff and a set of extensions to bootstrap your projects. Uno Platform is free, open-source (Apache 2.0), and available on GitHub.

Next Steps

To upgrade to the latest release of Uno Platform, please update your packages to 4.6 via your Visual Studio NuGet package manager! If you are new to Uno Platform, following our official getting started guide is the best way to get started. (5 min to complete)