Recently I had a requirement to load streaming JSON data to provide a data feed for near real-time reporting. The solution streamed data into an “Ingress” Azure Event Hub, shred the JSON via Azure Stream Analytics and then push subsections of data as micro-batches (1 sec) into a “Egress” Azure Event Hub (for loading into a stage table in Azure SQL DW).
In Event Hubs and Stream Analytics there are only a few performance levers to help tune a solution like this, or said another way, doing nothing with these levers can affect your ongoing performance!
So this blog is to show the performance differences when using different Azure Event Hub partition configurations and the Azure Stream Analytics PARTITION BY clause.
Recently I did a presentation at our local SQL Server User Group (SSUG) on Managing Streaming Data Pipelines Using Azure Data Services and as such wanted to build a compelling Azure demo that worked with simple streaming data which under certain event conditions would trigger an outbound phone call.
If interested the presentation deck is here – SSUG Melbourne – Building Streaming Data Pipelines Using Azure Cloud Services
The solution had several key components and stages outlined in the architecture below.
- A mobile phone app which generates JSON events with the X, Y, Z location of the device and G (g-force) detected in the device during movement.
- An Azure IoT Hub (AIH) which accepts the JSON events posted from the mobile device
- An Azure Stream Analytics (ASA) job that queries the Event Hub and routes the event data to several outputs, including…
- Azure Blob Storage (for archive of all event data)
- Power BI (for a live dashboard of all event data)
- Azure SQL Database (ASDB) (for tabular storage of all event data)
- Azure Event Hub + Azure Function (AF) (for queuing events which have a G Force reading greater than 3 and then triggering a phone call back to the original device from which the event originated)
The entire demo solution is actually really interesting (tsk, of course!) – and I will blog about other separate parts of this presentation at some point later. However the part of the demo that received the most interest was the external phone call integration with Azure Functions.
To be clear up front – Azure itself does not have native phone capability – so to make outbound phone calls I leverage an external “Twilio” API from within an Azure Function, and “Twilio” connects the outbound call.
And so, lets see the Twilio phone setup and c# Function code in action!