Streaming Reporting: SQL Change Data Capture (CDC) to Power BI

Extending on my previous post about redirecting SQL CDC changes to Azure Event Hub, I have had a few people ask for details/options to stream SQL data into the Power BI API.

Specifically – they were looking for an easy method to leverage the ADD ROWS functionality of the Power BI API so they could push real-time data into a Power BI service dataset.

This method provides the ability to update the Power BI Dataset with new rows every few seconds, instead of a Power BI report having to either use Direct Connect or Scheduled data refresh capability which can be very limiting.

If interested in how the SQL CDC and Event Hubs work together, then read here from my previous post – https://mrfoxsql.wordpress.com/2017/07/12/streaming-etl-send-sql-change-data-capture-cdc-to-azure-event-hub/

The purpose of this post is to quickly show how to extend and explore pushing new SQL data rows via Azure Stream Analytics into Power BI.

And so, lets get into some CDC to Power BI streaming action!

Continue reading

Streaming ETL: SQL Change Data Capture (CDC) to Azure Event Hub

I had a recent requirement to capture and stream real-time data changes on several SQL database tables from an on-prem SQL Server to Azure for downstream processing.

Specifically we needed to create a streaming ETL solution that …

  1. Captured intermediate DML operations on tables in an on-prem SQL database
  2. Transmit data securely and real-time into Azure
  3. Store the delta changes as TXT files in Azure Data Lake Store (ADLS)
  4. Visualise the real-time change telemetry on a Power BI dashboard (specifically the number of Inserts, Updates, Deletes over time).

The first part was easy; SQL has a feature called Change Data Capture (CDC) which does an amazing job of tracking DML changes to seperate system tables.  If you dont know about CDC then see here – https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server

The second part wasn’t easy, and after some searching I came across this blog post by Spyros Sakellariadis which gave me inspiration and starter code for my streaming ETL solution.  Excellent post.  See here – https://azure.microsoft.com/en-us/resources/samples/event-hubs-dotnet-import-from-sql/

And so, the final architecture looks something like this…

The solution picks up the SQL data changes from the CDC Change Tracking system tables, creates JSON messages from the change rows, and then posts the message to an Azure Event Hub.  Once landed in the Event Hub an Azure Stream Analytics (ASA) Job distributes the changes into the multiple outputs.

What I found pretty cool was that I could transmit SQL delta changes from source to target in as little as 5 seconds end to end!

And so, lets get into some CDC to Event Hub data streaming action!

Continue reading

Making Phone Calls from Azure Event Hub Messages

Recently I did a presentation at our local SQL Server User Group (SSUG) on Managing Streaming Data Pipelines Using Azure Data Services and as such wanted to build a compelling Azure demo that worked with simple streaming data which under certain event conditions would trigger an outbound phone call.

If interested the presentation deck is here – SSUG Melbourne – Building Streaming Data Pipelines Using Azure Cloud Services

The solution had several key components and stages outlined in the architecture below.

 

  1. A mobile phone app which generates JSON events with the X, Y, Z location of the device and G (g-force) detected in the device during movement.
  2. An Azure IoT Hub (AIH) which accepts the JSON events posted from the mobile device
  3. An Azure Stream Analytics (ASA) job that queries the Event Hub and routes the event data to several outputs, including…
    • Azure Blob Storage (for archive of all event data)
    • Power BI (for a live dashboard of all event data)
    • Azure SQL Database (ASDB) (for tabular storage of all event data)
    • Azure Event Hub + Azure Function (AF) (for queuing events which have a G Force reading greater than 3 and then triggering a phone call back to the original device from which the event originated)

The entire demo solution is actually really interesting (tsk, of course!) – and I will blog about other separate parts of this presentation at some point later.  However the part of the demo that received the most interest was the external phone call integration with Azure Functions.  

To be clear up front – Azure itself does not have native phone capability – so to make outbound phone calls I leverage an external “Twilio” API from within an Azure Function, and “Twilio” connects the outbound call.

And so, lets see the Twilio phone setup and c# Function code in action!

Continue reading

What Exactly is the “Data Platform” Nowadays?

A couple of months ago I was presenting at SQL Saturday Melbourne (582) on Azure Cognitive Services and got chatting with some of the other presenters about our sessions.

I co-presented with Krissy Rumpff from Microsoft Data Platform Team (https://www.linkedin.com/in/krumpff/) – and for those interested our session is here – http://www.sqlsaturday.com/582/Sessions/Details.aspx?sid=56483    …or…    you can look at the recording here – https://channel9.msdn.com/Events/Ignite/Australia-2017/DA321

Anyway, whats interesting is that some of the other presenters were asking why we were presenting on Cognitive Services, when in fact this was SQL Saturday?  And, you know, Cognitive is not the Data Platform?

This is actually an interesting point – and since then I have had a pretty good think about what this means – and so this is the purpose of this blog post!

Continue reading

Azure Cognitive Services Text Analytics – An API Calling Application

Just last week we had the fantastic opportunity to present at Microsoft Ignite 2017 in the Gold Coast on Azure Cognitive Services – and we had an absolute blast of a time!

I co-presented with Kristina Rumpff who works at Microsoft in the Data Platform team as a Solution Architect.  I focused on an overview of the suite of Azure Cognitive Services along with a deep dive into the Text Analytics service, and Krissy focused on the LUIS service coupled together with Azure Bots.

 

Fast Start to Azure Text Analytics Cognitive API’s

Leading up to, and since, the session I had a few people ask if there is anything pre-canned application wise which can call the Text Analytics API’s which they can just use.

The answer to that is kind of yes

However apart from that I didn’t find anything else out there which people can quickly leverage to do this for them… so I wrote one!

UPDATE:  I have since found this app handy myself when I need to quickly process a stack of once-off random text that someone sent to me for some other downstream reporting.  (…AND this was a good chance to learn more c# coding!)

And so, lets see the application code in action!

Continue reading

SQL Saturday 582 Melbourne (11 Feb 2017) and Ignite 2017 (14-17 Feb 2017)

For those not aware there are 2 cool local and national events coming to both Melbourne and Gold Coast

  • SQL Saturday 582 (Sat 11 Feb 2017).  Melbourne.  For those looking for some great free local SQL (and related!) learning, you simply cannot go past a SQL Saturday anywhere in the world!  And this one right here in Melbourne will again be no exception.  There is a lineup of fantastic local speakers including Microsoft and MVP’s as well as international speakers too.  http://www.sqlsaturday.com/582/EventHome.aspx
  • Ignite 2017 (14-17 Feb 2017).  Gold Coast.  For those wanting more than a day and across a wider variety of topics, then Ignite will also be another you simply cannot miss.  As above there is a fantastic lineup of both local Microsoft and local MVP/experts as well as an impressive list of international speakers.  This is a 4 day back-to-back set of technical sessions. https://msftignite.com.au/

UPDATE: Ignite 2017 has come and gone – and so here’s some links!

 

I want to focus this short blog post on SQL Saturday 582 as there are some great things happening right here locally in Melbourne… 

SQL Saturday 582 – Melbourne

SQL Saturday is an excellent free learning resource for all things SQL Server – all costs are covered by donations and sponsorships.  Some of the excellent sponsors this year are Microsoft, Wardy IT, Idera and PASS.

Some of the session focus areas include SQL 2016 (many deep dives across almost all facets!), SQL DB/DW in Azure, Azure Cognitive Services and Machine Learning, R, Agile Methods, Power BI, Powershell, BIML …and more!

Furthermore – Some of the top Microsoft product group (PG) speakers will also be attending:

  • Lindsey Allen
  • Sunil Agarwal
  • Matt Masson
  • Julie Koesmarno
  • Ajay Jagannathan

Sunil will also be helping to run a SQL Clinic on the day where you can talk directly to Microsoft about your biggest pain points or suggestions for the next versions of SQL Server.

For those wanting to come along here are the links you need to know.  Please go to the website and register to attend.

The event is being held at Monash University (Caulfield Campus, 888 Dandenong Road, Caulfield East, Victoria)

 

The Rise of the Machines…

For those attending – I am co-presenting a pretty cool session with the awesome Krissy Rumpff (from local Microsoft SQL Team) on Azure Cognitive Services.  

Session Details here – http://www.sqlsaturday.com/582/Sessions/Details.aspx?sid=56483

We’ll talk though what they are, why you should care and how to interact with them (via API’s).  We hope to show some pretty fun demos and draw some relevant use cases – and if time permits will look for some unwilling audience participation!

Feel free to pop in and introduce yourself!

 

I hope to see you all in Melbourne at SQL Saturday!


Disclaimer: all content on Mr. Fox SQL blog is subject to the disclaimer found here

Azure Cognitive Services API’s with SQL Server 2016 CLR

Azure Cognitive Services is relatively new functionality within Azure that exposes some truly amazing APIs that have the ability to do some truly amazing things.

Before I dive into SQL and DLL code to make use of Cognitive Services, lets take a second to understand what I am talking about – imagine this;

  • A customer walks to an electronic kiosk in a shopping centre and says “Hi Cortana, I want to book a holiday, I really need a break from this bad weather.  Any ideas on where I should go?
  • The kiosk is run by a “bot” capable of conversing in 30 languages and which has been trained on understanding context and intent.
  • It recognises you as a 35 yo male who looks and sounds unhappy, and because it recognises your face and voice it knows it has talked to you before in another shopping centre last week when you asked for directions to a Surf Shop clothing store.
  • It also recognises a beach image on your T-Shirt, making note of the link between your previously asked directions and your clothing.
  • As you spoke in English, it replies in English – “Good to see you again.  Now, would you consider a beach holiday to Bali or Thailand?” – Why beach? Well the recommendations engine has determined that is where 35yo male surfer types go when the local weather is bad!
  • You negotiate a package using natural language, and close out the conversation.
  • The “bot” visualises your increased sentiment from the initial baseline and says “I’m glad I could make your day better! Enjoy your flight next week!

It may sound futuristic – but this is exactly what Cognitive Services (API’s) can do right now – and in my example I have only used 6 out of the 21 Azure Cognitive Services!  Microsoft Research has built these powerful Azure ML Models and wrapped them up into a single, simple, consumable publicly available API.

Some other amazing deployments for Cognitive Services

  • At a trade show, or even a window display at a shopping centre, a company could use emotion detection to see how people are reacting to their products.
  • Facial recognition could be used to find missing children quickly at an amusement park.
  • The APIs can determine the male:female ratio and ages of patrons at a nightclub, and identify VIPs or banned guests.
  • The object recognition capabilities can enable a blind person to read a menu in a restaurant or have their surroundings described to them

For those not familiar with Azure Cognitve Services APIs, check out this link which has online demos you can try – https://www.microsoft.com/cognitive-services/en-us/apis

For those not familiar with the Azure Bot Framework, check out this link – https://dev.botframework.com/

Anyway – despite all this, for this post today we’ll just focus on something pretty simple – making usage of the Text Analytics API right within SQL Server 2016.

And so, lets get to scoring some sentiment!

Continue reading

Azure Cortana Intelligence Suite – with Azure Data Catalog

I presented at our local Melbourne SQL Server User Group (SSUG) this week on Azure Cortana Intelligence Suite – and the integration with the Azure Data Catalog.  If you are interested to see the presentation then have a look here – https://mrfoxsql.wordpress.com/presentations/

One of the things which stood out to me from attendees is that the number of new services and new cloud terms which are being introduced across the industry is sometimes difficult to keep track of – especially when you have a day job to do!  Right now I cannot think of another time in IT where the rate of change is so dramatic.

More-so it can also sometimes be hard to know which services to use together to get the best architecture outcome.

So this blog post is aimed to explain at a high level the Azure Cortana Intelligence Suite of services and one example of how they could be deployed together in a cohesive architecture.

I will talk to each of these services in the context of a business solution to manage a farm of wind turbines!

And so, lets get into some discussion on Azure IoT Solutions!

WindFarms

 

Continue reading

Azure DocumentDB – Preparing, Loading and Querying Data

 

Recently I have been using Azure DocumentDB to validate several business use cases for a variety of application purposes.

For those SQL DBA’s and others who are new to Azure DocumentDB, its a recent entrant to the NoSQL document database world, and as its a PaaS document database cloud service it has the agility, scalability and availability of the Azure Cloud.

Being a schema-less Azure PaaS “document database” for my use case I wanted to verify…

  • basic costing and performance levels
  • methods to create valid JSON documents from SQL Server
  • methods to load JSON documents into Azure DocumentDB
  • performing basic like-for-like document query comparisons with SQL Server

Some homework reading for those interested…

 

22 May 2017 [EDIT]

Microsoft have just recently added significant new functionality and also formally renamed Azure DocumentDB as Azure CosmosDBa major evolution of the NoSQL database engine.  The details on the renamed service is here – https://azure.microsoft.com/en-au/blog/dear-documentdb-customers-welcome-to-azure-cosmos-db/

 

And so… let get into the belly of Azure DocumentDB CosmosDB!

Continue reading

PASS 2015 Session Report – Azure SQL DW Integration with the Azure Ecosystem & End of PASS 2015

PASS 2015 continues (and finishes up today!) in Seattle.

Its been an amazing conference this year with a few things really hitting home;

  • Amazing technology announcements around SQL 2016 CTP3
  • Incredible advances in almost every component in Azure Data Services
  • Full and seamless SQL/Azure ecosystem integration – and by that I mean both On-Prem and/or within the Azure Cloud.  The story of either On-Prem or Azure Cloud is compelling enough individually, however the Hybrid story is now a reality for SQL and enables dynamic and flexible architectures well beyond what competitors can offer.
  • BUT what astounds me the most is actually the pace of change – barely a day goes by where I don’t receive a new services or feature update related to SQL 2016 CTP3 or Azure.
  • I don’t recall a time (in recent memory) where the step changes have come so thick/fast – its certainly changed from where I started as a DBA on RDB/VMS back in 1994 where patches arrived by mail on tape cartridge! 🙂
  • (As a quick aside a chief designer on RDB was Jim Gray, the same who joined Microsoft to lead the SQL Server architecture to stardom soon after Oracle bought-out and shelved DEC around 1995+)

 

Enough reminiscing already – moving along – Today I attended 5 back-back sessions, and again I cannot blog about all of them in the time I have (or want to spend), but the one which stands out the most was Azure SQL Data Warehouse and Integration with the Azure Ecosystem by Drew DiPalma of Microsoft.

This session focused specifically on the Azure ecosystem surrounding the Azure SQL Data Warehouse (SQL DW) and how it can seamlessly interact with other Azure components to create different operational solutions.  To me this was very compelling, not necessarily due to the SQL DW technology (which I know well already as the on-prem APS appliance), but more-so as it showed just how easily all parts of Azure can happily work together.

Continue reading