Going serverless with Internet-of-Things and Azure Functions

The internet-of-things concept prevails in near every article on future IT prospects. And such a fascinating topic it is,too, the scenarios are limitless and the use-cases palpable, feels as if we should have been able to fulfill them ages ago. But let’s put the philosophical notions aside; I’d like to demonstrate my choice of technology for an IoT-solution based on the Azure serverless-offerings. So without further ado.

The internet-of-things - IoT from hence forward - may be realized in a myriad of fashions; I’ll cover today the use of Microsoft’s Azure cloud functionality, specifically the use of Azure Function Apps, Azure Service Bus and Azure Storage - with a very little bit of Azure Logic Apps thrown in for good measure. So it’s a whole lot of Azure, and the title speaks of ‘serverless’, too. In case you’re unfamiliar with the term, I’ll try my best at my own definition of the term: ‘Serverless’ refers to resources you do not need to carefully provision on your own. Of course we’re all well aware of how no request for a specific resource is ever truly ‘serverless’; until further, future notice there’s, however deep down in the many-layered stack, some physical hardware-element that is destined to provide the response to the request, there’s no getting around that yet. So the ‘serverless’ term should not be taken literally, as opposed to providing a moniker for getting resources up and running, and responding to requests, without having to care much about physical hardware restraints. Such that you do not need to, in honor of the phrasing of the term, set up an actual server to response to requests, rather you can simply deploy the response-mechanism itself to the Azure cloud, and instead of tending to hardware-specific metrics - how many CPU’s, whether to deploy load balancing, such considerations - you can focus on how you would like the response-mechanism to behave, for example how to scale. 

But I digress; you’re likely fully aware of the term and its implications. For the sake of this article, however, serverless also implies a consideration about the setup: my IoT devices will send data to, and receive data from, the aforementioned cloud services. This as opposed to letting requests be handled by a central broker such as an MQTT implementation, for example. This is a wholly deliberate choice; this particular implementation of an IoT architecture is suited for smaller projects, where the requirements can be the more readily fixed and thus we can make these assumptions that we would otherwise abstract away. Also, further limitations of this particular implementation is, to be later amended, a lack of device security focus and provisioning, entirely complicated topics in their own right. Make no mistake, this IoT implementation will work fine and have extremely limited financial impact. If you’re dealing with tens of thousands of distinct sensors and devices, across a multitude of locations, you will likely find better use of the suite of dedicated Microsoft IoT portfolio, and I recommend you then investigate that route further.

Tedious limitations aside, let’s dive into the fun stuff! Please keep the below architecture-diagram in mind as we go through a simple use-case, along the way deliberating on the technological choices.

The example will be of a temperature-sensor that sends a temperature-reading to our cloud-based back-end, optionally in turn retrieving a device-specific command to execute.

The device distributes its reading to an Azure Function (1) by way of a HTTPS REST-call. The function is responsible for storing raw data into an Azure Storage queue (2). Azure Functions are tremendously cheap to execute - you can have millions of executions for very low cost, thus ideal for a network of sensors that frequently sends data. Similar financial argument applies for the Azure Storage account that we will use to hold our sensor data. The queue storage is ideal for our purpose; it is designed to hold work-items to be processed at some later stage, at which point the data will simply be removed from the queue - and we can also specify a dedicated ‘time-to-live’ for the data, if needed. But, most importantly, we can make use of a dedicated Azure Function App trigger (3) that activates on new items in the queue. In this case specific case it retrieves the data from the queue and, from the raw sensor data, creates a more specific, and enriched, data model. We could do this in the first Azure Function, certainly, but the abstraction point is important in as it enables us to inject business logic here, if this is later needed. At present the only logic is in retrieving sensor-device information from an Azure storage table (4), and adding a bit of this information to the device-message that then goes into an Azure storage table that holds sensor data (5) - but later on we might add sensor-authentication in there, and at least now we have an abstraction point in which to implement this further down the line. The Azure Functions scale well; if your queue becomes crowded and you’re running the functions on a so-called consumption plan, for example, the functions will simply spin up more instances to handle the load. That really speaks to the core of the serverless term.

So, the second Azure function creates a more meaningful piece of data (6) - I add information about the specific type of sensor, for example - and sends this to an Azure Service Bus topic (7). The Azure Service Bus is a data ingestion and distribution mechanism, quite capable of receiving and handling millions of messages within an ambitious time-frame - just what we might need for a sensor-rich IoT solution. It is not the only tool-choice in regards to mass-message ingestion; Microsoft offers the dedicated IoT event hub, for example, and other vendors will have their own offerings. The reasons I chose it for are as follows, it’s cheap, fast, simple, and it plays extremely well with Azure Functions, as we’ll get around to shortly. The Azure Service Bus receives messages in two various ways: directly into a queue, not unlike the table storage queue albeit with some significantly enhanced features. For our purpose, however, we’ll utilize the Service Bus Topic feature, where we send messages into a so-called ‘topic’, which we may then subscribe to. This is the general publish-subscribe mechanism most often associated with various service bus implementations, and it works well with an IoT scenario such as this. In my specific implementation, I create a generic ‘message received’-topic, and into this I then send every piece of sensor-data that is received and enriched. This enrichment of data is mainly what facilitates a meaningful filtering of the message into dedicated subscriptions (8). A simple example might be in how a temperature sensor sends a reading to the receiving Azure function. The raw data is enriched with device-type information, so that we can infer the sensor-type - a temperature sensor - from the device-id. This enriched message is then sent to the service bus. A subscription to this topic will have been created and will pick up, for example, any messages from temperature sensors with a temperature exceeding an x degrees threshold.

The advantages of this, in conjunction with the use of the Azure Function App, becomes quite clear as we react to messages being picked up by our various topic-subscriptions, such as the ‘TemperatureHigh’-subscription. The subscriptions act as nothing more than message-filters and -routers. In order to consume the messages we have, as is almost always the case with the Azure platform, multiple ways of going about it. For our implementation we’ll implement another Azure Function, specific to messages being sent to the 'TemperatureHigh'-subscription. It’s that simple - we specify a subscription-name as we create the function, then deploy it, and the Azure infrastructure sees to it that the function is triggered appropriately (9). We do not have to poll continuously, we’re always hooked up, so to speak. This is a major advantage of integrating these two technologies, i.e. the possibility of quickly building an infrastructure that’s equally capable and reliable. The downside remains, of course, that it’s a very efficient yet hard-coupled architecture - there’s no replacing the service bus component with another cloud provider’s similar product. There’s always that trade-off that we need make; for my purposes, this coupling works extremely well: as messages arrive at their dedicated subscription, an equally dedicated Azure Function is triggered and the message is thus consumed and acted upon. The act, in this scenario, is in issuing an appropriate command (10) for the device itself, or possibly another device. For example, given a higher than usual temperature, we might issue a command to the device itself to sound an alert-buzzer. The command goes on an Azure Storage table (11), where we keep the history of issued commands, for audit trail and visualization purposes. It’s from this table the final Azure Function retrieves this command, upon a periodic request (12) from the sensor device that then executes it.

 So that’s an example of an IoT architecture based on Azure cloud technologies, without a central broker. Once again, it’s not to be considered best practice for all scenarios, please don’t implement it without regard for the circumstances pertaining to your own demands.

Now, for some technical aspects, I’d like to present some bits of the code of the Azure Functions, and thus go further into the details behind the implementation.

The Azure function into which is sent the raw data from the sensor is an HTTP-triggered one such. Here’s the code, annotated further below:

[FunctionName("QueueRawValue")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "QueueRawValue")]HttpRequestMessage req, TraceWriter log)
{
    try
    {
        log.Info("C# QueueRawValue http trigger function processed a request.");

        string deviceId = req.GetQueryNameValuePairs().FirstOrDefault(q => string.Compare(q.Key, "deviceId", true) == 0).Value; 
        string value = req.GetQueryNameValuePairs().FirstOrDefault(q => string.Compare(q.Key, "value", true) == 0).Value;
        DateTime timestamp = DateTime.Now;

        CloudStorageAccount cloudStorageAccount = CloudConfigurationFactory.GetCloudStorageAccount();
        var queueClient = cloudStorageAccount.CreateCloudQueueClient();
        var queueReference = queueClient.GetQueueReference("iotl15queue");
        // Create the queue if it doesn't already exist
        await queueReference.CreateIfNotExistsAsync();

        RawIngestDataModel data = new RawIngestDataModel
        {
            DeviceId = deviceId,
            RawValue = value
        };
        string queueMessage = JsonConvert.SerializeObject(data, Formatting.None);
        var message = new CloudQueueMessage(queueMessage);
        await queueReference.AddMessageAsync(message);

        return deviceId == null
            ? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a deviceId on the query string or in the request body")
            : req.CreateResponse(HttpStatusCode.OK);
    }
    catch (Exception e)
    {
        // todo: do some logging
        throw e;
    }
}

Please disregard the deliberately blatant lack of security considerations and other such concerns, and focus on the basic functionality. The raw json-data from the sensor is put onto a storage queue for later processing. That’s all the function does, grabbing raw input data from the request parameters and storing this into a 'RawIngestDataModel' object, representing just that, raw input data in any shape or form. So we have a very basic way of capturing information and storing this into a queue, for eventual later - hopefully swift and efficient - processing. We could process the raw data at this stage, but this design provides us with an extension point we might need good use of, later on: if the number of request were to suddenly sky-rocket, the queue would easily scale to fit this requirement, by virtue of its built-in cloud capabilities thus regarding.

The next function, in turn, is then triggered by this queue-adding:

[FunctionName("ProcessRawQueueMessage")]
public static void Run([QueueTrigger("iotl15queue", Connection = "AzureStorageConnectionString")]string myQueueItem, TraceWriter log)
{
    try
    {
        RawIngestDataModel rawIngestData = JsonConvert.DeserializeObject<RawIngestDataModel>(myQueueItem);
        
        CloudStorageAccount cloudStorageAccount = CloudConfigurationFactory.GetCloudStorageAccount();
        var cloudService = new AzureTableStorageService(cloudStorageAccount);
        RegisteredValueModel registeredValueModel = CreateRegisteredDatamodelFromRawInput(rawIngestData);
        cloudService.SendRegisteredDataToTableStorage(registeredValueModel);
  
        // send to servicesbus
        string ServiceBusConnectionString = @"Endpoint=sb://myservicesbus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=ssharedkeyD/2zayhog=";
        string TopicName = @"devicemessagetopic";
        ITopicClient topicClient = new TopicClient(ServiceBusConnectionString, TopicName);

        // Create a new message to send to the topic.
        string messageBody = JsonConvert.SerializeObject(registeredValueModel);
        var message = new Message(Encoding.UTF8.GetBytes(messageBody));

        message.UserProperties.Add("DeviceId", registeredValueModel.DeviceId);
        message.UserProperties.Add("TextValue", registeredValueModel.TextValue);
        message.UserProperties.Add("NumericalValue", registeredValueModel.NumericalValue);
        DeviceModel deviceInformation = GetDataAboutDevice(registeredValueModel.DeviceId);
        message.UserProperties.Add("DeviceType", deviceInformation.DeviceType);

        // TODO: enrich with device-type, etc.

        // Send the message to the topic.
        topicClient.SendAsync(message);

        log.Info($"C# Queue trigger function processed: {myQueueItem}");
    }
    catch (Exception e)
    {
        // todo: do some logging
        throw e;
    }
}

private static RegisteredValueModel CreateRegisteredDatamodelFromRawInput(RawIngestDataModel rawIngestData)
{
    RegisteredValueModel registeredValueModel = new RegisteredValueModel()
    {
        DeviceId = rawIngestData.DeviceId,
        TextValue = rawIngestData.RawValue,
    };

    float attemptToParseValueAsNumerical;
    if (float.TryParse(rawIngestData.RawValue, out attemptToParseValueAsNumerical))
        registeredValueModel.NumericalValue = attemptToParseValueAsNumerical;

    return registeredValueModel;
}

/// <summary>
/// Get device-data from table storage
/// </summary>
/// <remarks>
/// Return dummy data for now.
/// </remarks>
private static DeviceModel GetDataAboutDevice(string deviceId)
{
    // TODO: implement this. Consider memory caching.

    DeviceModel temporaryDeviceModel = new DeviceModel()
    {
        DeviceId = deviceId,
        DeviceType = "TemperatureMeasurementDevice"
    };
    return temporaryDeviceModel;
}

Above function dequeues data from the queue, for further processing. The bulk of work is already done for us, in terms of how to connect to the queue and react on new entries into said queue, all this rather crucial functionality is already wired up for us and ready to be made good use of. Almost seems to good to be true, does it not. It’s well worth remembering the old adage, ‘if it seems to good to be true…’: we do get a tremendous amount of proven functionality ‘for free’, so to speak, but of course we also give up the possibility of having a say in how all this is achieved; we’re tied into the Azure platform. This is an acceptable choice for my particular IoT implementation, but may not be for you - it’s pros and cons and something you should take into serious consideration, as per your particular scenario. The above code will retrieve the first available raw data from the queue, and transform it into a RegisteredValueModel-object. Note how this inherits from the TableEntity-object; so we can store it within an Azure Table Storage table. For my purposes I'm using the device-id as partition-key on the table, as this seems a natural fit. That's behind the scenes and not shown here, for brevity's sake. From this table we'll later be able to do visualizations and historic compilations on the device data, though that's for a later blog-entry. The most important bit, for now, is in noting how the registered device data is sent to the Azure Service Bus topic, with the 'devicemessagetopic' name that indicates how, indeed, this topic receives all messages from all devices. Here stops, then, the responsibility of the Azure function. Now we can go and create subscriptions to this topic, as pertains to our specific use-cases. For example creating aforementioned subscription to dangerously high temperatures from my temperature-sensors. "temperatureHighSubscription" is my name for it, and given this name and a valid connection into the service bus, we can easily crate an Azure Function that triggers when the Azure service bus filters messages to this subscription:

[FunctionName("GeneralHighTempTriggerFunction")]
public static async Task Run([ServiceBusTrigger("devicemessagetopic", "temperatureHighSubscription", Connection = "AzureServiceBusConnectionString")]string mySbMsg, TraceWriter log)
{
    log.Info($"C# ServiceBus topic trigger function processed message: {mySbMsg}");

    RegisteredValueModel dataFromServiceBusSubscription = JsonConvert.DeserializeObject<RegisteredValueModel>(mySbMsg);

    // Add to commandModel history data table
    DeviceCommandModel deviceCommand = new DeviceCommandModel()
    {
        DeviceId = dataFromServiceBusSubscription.DeviceId,
        CommandText = "SoundAlarm",
        SentToDevice = false
    };

    CloudStorageAccount cloudStorageAccount = CloudConfigurationFactory.GetCloudStorageAccount();
    var cloudService = new AzureTableStorageService(cloudStorageAccount);
    cloudService.SendDeviceCommandToTableStorage(deviceCommand);

    // Send notification of high temperature to azure logic app:
    INotificationService azureLogicAppNotificationService = new AzureLogicAppSendPushOverNotificationService();
    NotificationModel notification = new NotificationModel()
    {
        Title = "Temperature-alarm",
        Message = $"Temperature-device {dataFromServiceBusSubscription.DeviceId} at {dataFromServiceBusSubscription.NumericalValue:F} degrees",
        From = "IOT",
        To = "pushOverUserId" 
    };
    await azureLogicAppNotificationService.SendNotification(notification);
}

Couldn't be much easier, it's already wired up by design and the functionality to act on the trigger is all that remains for us to implement. In my case, the subscription is a call to action, namely triggering a device-specific command to act on the high temperature, and "SoundAlarm". All commands are stored into an Azure storage table, for both audit trail and command repository: all devices may, if so configured, continuously poll this table for any command that needs be executed by them - identified by their device-id. A quick Azure Function, http-triggered, delivers the goods:

/// <summary>
/// Retrieves the latest non-yet-retrieved command for a device, if any such command exists.
/// </summary>
[FunctionName("GetCommandFromServicesBus")]
public static HttpResponseMessage Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "GetCommandFromServicesBus")]HttpRequestMessage req, TraceWriter log)
{
    string deviceId = req.GetQueryNameValuePairs().FirstOrDefault(q => string.Compare(q.Key, "deviceId", true) == 0).Value; //req.Content["deviceId"];

    CloudStorageAccount cloudStorageAccount = CloudConfigurationFactory.GetCloudStorageAccount();
    var cloudService = new AzureTableStorageService(cloudStorageAccount);
    DeviceCommandModel commandForDevice = cloudService.GetCommandFromTableStorage(deviceId);

    return commandForDevice == null ?
        req.CreateResponse(HttpStatusCode.OK) // no commands found, just OK status
        :
        req.CreateResponse(HttpStatusCode.OK, JsonConvert.SerializeObject(commandForDevice)); // command found, return as json.
}

And so round and round it goes; devices ships data, the data is enriched, sent to a services bus and maybe/maybe not picked up by a subscription, which in turns triggers a command, so on and so forth.

I haven't touched upon the use of Azure Logic Apps, and I shan't go into them safe but note that I do implement a couple, for notification purposes - for example in the above 'GeneralHighTempTriggerFunction' code. Azure Logic Apps gives us the ability/possibility of gluing many Azure offerings together, but that's not my use-case as yet. You can have an Azure Logic App listen for subscription-hits on your services bus, for example, and compile multiple messages into a single command to a device, or vice versa. The graphical interface with which you create the Logic Apps is intuitive, yet offers great levels of complexity in the execution. You could also make use of it as an elaborate extension point, and out-source business logic to others while you take care of data yourself, for example.

So that's a bit of inspiration, I hope, on going serverless with Azure and getting those IoT-message flowing. I won't lie, getting just a dozen devices up and running and sending data and commands back and forth is fun to watch - and those Azure offerings make it simple and mostly intuitive to get started. Of course there's tons of stuff I haven't covered in detail in the above, and I'll leave you to second-guess the missing functions that'll enable to code to compile. It's meant as an appetizer, and I'll look forward to learning about your particular 'main course', so please by all means drop me a note about what you're doing with Azure and IoT. 

IoT projects are fun to be part of, I wish I could do more of it but, to my chagrin, my career-path never led me down that road besides trying it out for fun at home. I hope the above will inspire you in your endeavors. Thanks for checking it out, and if there's anything I can do to help out get in touch me and I'll try and do that.



Hope it helps!

Buy me a coffee