IoT: Detecting bad data at the Edge

A while back I wrote an article on methods for connecting Industrial Control Systems (ICS) with Azure and why you would want to do that. One of those ways is by using the Azure IoT Edge gateway open source project available on GitHub.

We recently announced some new features in Azure IoT Edge gateway to support "intelligent edge" scenarios, including the ability to deploy modules via docker containers from Azure to the gateway. This article provides a nice walk-through.

One of the other new features is the ability to run Azure Functions as a module on the IoT Edge gateway. Colby Tresness (@ColbyTresness) recently wrote an article on running Azure Functions on IoT Edge.

When I read this article one of the use cases that immediately came to mind was to use an Azure Function to detect bad data or malformed payloads coming from sensors and causing issues with downstream solutions (management, analytics, etc.) running in Azure. Of course, you could also accomplish this by building a custom IoT Edge module.

In the past I've seen IoT messages with bad payload data cause issues that result in:

• Azure Stream Analytics jobs failing due to malformed JSON (try "accidentally" sending an XML payload to Stream Analytics);
• Azure Stream Analytics Output errors. For example, if the Output is an Azure SQL Database and the table is setup to not "Allow Nulls", and a sensor decides to send "null" in the JSON payload, Stream Analytics gets grumpy; and
• Wasted time performing data cleansing work using tools such as Azure Data Lake Analytics or Spark.

But by detecting and rerouting bad messages at the edge, you can prevent these issues.

Regardless whether you build an Azure Function module or custom module, your code will need to detect:

• Invalid payload formats (e.g. XML, invalid JSON, etc.); and
• Invalid data within valid JSON (e.g. "Temp" : null)

The easiest way to do this is to:

1. Attempt to deserialize the JSON payload into some model classes. If you develop in C# you could use Newtonsoft DeserializeObject wrapped inside a try/catch block. If you throw an exception or error, you consider the message bad.; and
2. Check if the values are actually valid. For example:

if (sensor.temperature == null)
{

  //Message contains bad data. Temperature should never be null.
  
}

The instructions and code sample here provide a good starting point and include some sample model classes that you will need to customize to match the shape of your incoming sensor data. This site helps quickly generate C# model classes from JSON.

 class MessageBody
 {
     public Machine machine {get;set;}
     public Ambient ambient {get; set;}
     public string timeCreated {get; set;}
 }
 class Machine
 {
    public double temperature {get; set;}
    public double pressure {get; set;}         
 }
 class Ambient
 {
    public double temperature {get; set;}
    public int humidity {get; set;}         
 }

Next, you will also want to customize the code sample to handle any bad messages. If you detect bad message you have a couple options:

1. Send the bad message as-is to another Azure IoT Hub (aka Exception Hub) setup to receive bad IoT messages and is separate from the IoT Hub receiving good messages. This IoT Hub could write all the bad messages straight to blob storage using a custom endpoint and message routing.
2. Prepend a JSON "name" : "value" pair to the message header of the bad message and send it to the same IoT Hub that is accepting all messages (both good and bad). Within this IoT Hub, configure a custom endpoint and message routing that searches for the name : value in the message header (since the message body is bad you cannot use message routing on the body) and writes the bad message data to blob storage.

Once the bad message is saved to Azure Blob Storage, it could trigger an Azure Function followed by a Logic App for logging and alerting. Alternatively, using custom endpoints, IoT Hub could send the bad message to an Azure Service Bus Queue, Topic or Event Hub for further processing.

I hope this article gives you some ideas for detecting and handling bad IoT data.

Kevin Hilscher @khilscher

Azure Cloud Solution Architect @ Microsoft