What Exactly is the “Data Platform” Nowadays?

A couple of months ago I was presenting at SQL Saturday Melbourne (582) on Azure Cognitive Services and got chatting with some of the other presenters about our sessions.

I co-presented with Krissy Rumpff from Microsoft Data Platform Team (https://www.linkedin.com/in/krumpff/) – and for those interested our session is here – http://www.sqlsaturday.com/582/Sessions/Details.aspx?sid=56483    …or…    you can look at the recording here – https://channel9.msdn.com/Events/Ignite/Australia-2017/DA321

Anyway, whats interesting is that some of the other presenters were asking why we were presenting on Cognitive Services, when in fact this was SQL Saturday?  And, you know, Cognitive is not the Data Platform?

This is actually an interesting point – and since then I have had a pretty good think about what this means – and so this is the purpose of this blog post!

Continue reading

A Collection of SQL Server Help Scripts

Like probably every SQL DBA, consultant, architect etc etc out there that has ever worked on or used SQL Server they will likely have their own personal collection of SQL Server Help Scripts.

So not unsurprisingly I also have such a collectionand this is the purpose of this post!

 

The below is a link to my GitHub repo of my personal collection of scripts.  BTW this isn’t all of the scripts, but is probably 50% of the ones I have lurking around.

These are either scripts I have developed in my own time over the years, or adapted from various other websites which I found handy at the time for whatever reason.

Where I have remembered that web site (like most people, which is unfortunately almost never) it was typically from sites like stack overflow etc, and if so I have tried to cite it as such to provide credits.

 

Quite a few of my scripts are in fact scripts that generate other scripts!

So in other words – its a script you run on a source SQL Server where the output of that script execution is in fact another script that performs some dynamic operation based on the source server configuration at the time the original script was run.

These are perfect for those times you need to create scripts for some later activity, such as to say backup or restore databases, recreate indexes, change database properties, set login properties, etc etc

 

The below is the link to the repo and a list of the scripts I have there.  Read on!

Continue reading

Azure Cognitive Services Text Analytics – An API Calling Application

Just last week we had the fantastic opportunity to present at Microsoft Ignite 2017 in the Gold Coast on Azure Cognitive Services – and we had an absolute blast of a time!

I co-presented with Kristina Rumpff who works at Microsoft in the Data Platform team as a Solution Architect.  I focused on an overview of the suite of Azure Cognitive Services along with a deep dive into the Text Analytics service, and Krissy focused on the LUIS service coupled together with Azure Bots.

 

Fast Start to Azure Text Analytics Cognitive API’s

Leading up to, and since, the session I had a few people ask if there is anything pre-canned application wise which can call the Text Analytics API’s which they can just use.

The answer to that is kind of yes

However apart from that I didn’t find anything else out there which people can quickly leverage to do this for them… so I wrote one!

UPDATE:  I have since found this app handy myself when I need to quickly process a stack of once-off random text that someone sent to me for some other downstream reporting.  (…AND this was a good chance to learn more c# coding!)

So lets see the application!

Continue reading

SQL Saturday 582 Melbourne (11 Feb 2017) and Ignite 2017 (14-17 Feb 2017)

For those not aware there are 2 cool local and national events coming to both Melbourne and Gold Coast

  • SQL Saturday 582 (Sat 11 Feb 2017).  Melbourne.  For those looking for some great free local SQL (and related!) learning, you simply cannot go past a SQL Saturday anywhere in the world!  And this one right here in Melbourne will again be no exception.  There is a lineup of fantastic local speakers including Microsoft and MVP’s as well as international speakers too.  http://www.sqlsaturday.com/582/EventHome.aspx
  • Ignite 2017 (14-17 Feb 2017).  Gold Coast.  For those wanting more than a day and across a wider variety of topics, then Ignite will also be another you simply cannot miss.  As above there is a fantastic lineup of both local Microsoft and local MVP/experts as well as an impressive list of international speakers.  This is a 4 day back-to-back set of technical sessions. https://msftignite.com.au/

UPDATE: Ignite 2017 has come and gone – and so here’s some links!

 

I want to focus this short blog post on SQL Saturday 582 as there are some great things happening right here locally in Melbourne… 

SQL Saturday 582 – Melbourne

SQL Saturday is an excellent free learning resource for all things SQL Server – all costs are covered by donations and sponsorships.  Some of the excellent sponsors this year are Microsoft, Wardy IT, Idera and PASS.

Some of the session focus areas include SQL 2016 (many deep dives across almost all facets!), SQL DB/DW in Azure, Azure Cognitive Services and Machine Learning, R, Agile Methods, Power BI, Powershell, BIML …and more!

Furthermore – Some of the top Microsoft product group (PG) speakers will also be attending:

  • Lindsey Allen
  • Sunil Agarwal
  • Matt Masson
  • Julie Koesmarno
  • Ajay Jagannathan

Sunil will also be helping to run a SQL Clinic on the day where you can talk directly to Microsoft about your biggest pain points or suggestions for the next versions of SQL Server.

For those wanting to come along here are the links you need to know.  Please go to the website and register to attend.

The event is being held at Monash University (Caulfield Campus, 888 Dandenong Road, Caulfield East, Victoria)

 

The Rise of the Machines…

For those attending – I am co-presenting a pretty cool session with the awesome Krissy Rumpff (from local Microsoft SQL Team) on Azure Cognitive Services.  

Session Details here – http://www.sqlsaturday.com/582/Sessions/Details.aspx?sid=56483

We’ll talk though what they are, why you should care and how to interact with them (via API’s).  We hope to show some pretty fun demos and draw some relevant use cases – and if time permits will look for some unwilling audience participation!

Feel free to pop in and introduce yourself!

 

I hope to see you all in Melbourne at SQL Saturday!


Disclaimer: all content on Mr. Fox SQL blog is subject to the disclaimer found here

Azure Cognitive Services API’s with SQL Server 2016 CLR

Azure Cognitive Services is relatively new functionality within Azure that exposes some truly amazing APIs that have the ability to do some truly amazing things.

Before I dive into SQL and DLL code to make use of Cognitive Services, lets take a second to understand what I am talking about – imagine this;

  • A customer walks to an electronic kiosk in a shopping centre and says “Hi Cortana, I want to book a holiday, I really need a break from this bad weather.  Any ideas on where I should go?
  • The kiosk is run by a “bot” capable of conversing in 30 languages and which has been trained on understanding context and intent.
  • It recognises you as a 35 yo male who looks and sounds unhappy, and because it recognises your face and voice it knows it has talked to you before in another shopping centre last week when you asked for directions to a Surf Shop clothing store.
  • It also recognises a beach image on your T-Shirt, making note of the link between your previously asked directions and your clothing.
  • As you spoke in English, it replies in English – “Good to see you again.  Now, would you consider a beach holiday to Bali or Thailand?” – Why beach? Well the recommendations engine has determined that is where 35yo male surfer types go when the local weather is bad!
  • You negotiate a package using natural language, and close out the conversation.
  • The “bot” visualises your increased sentiment from the initial baseline and says “I’m glad I could make your day better! Enjoy your flight next week!

It may sound futuristic – but this is exactly what Cognitive Services (API’s) can do right now – and in my example I have only used 6 out of the 21 Azure Cognitive Services!  Microsoft Research has built these powerful Azure ML Models and wrapped them up into a single, simple, consumable publicly available API.

Some other amazing deployments for Cognitive Services

  • At a trade show, or even a window display at a shopping centre, a company could use emotion detection to see how people are reacting to their products.
  • Facial recognition could be used to find missing children quickly at an amusement park.
  • The APIs can determine the male:female ratio and ages of patrons at a nightclub, and identify VIPs or banned guests.
  • The object recognition capabilities can enable a blind person to read a menu in a restaurant or have their surroundings described to them

These are just some of the scenarios possible as described by Jennifer Marsman (Microsoft Principal Software Development Engineer).  For those interested in this you can attend the Microsoft Data Science Summit on 26-27 Sep in Atlanta.  See this info link – https://blogs.technet.microsoft.com/machinelearning/2016/09/07/artificial-intelligence-made-easy-cognitive-services-at-the-microsoft-data-science-summit/

For those not familiar with Azure Cognitve Services APIs, check out this link which has online demos you can try – https://www.microsoft.com/cognitive-services/en-us/apis

For those not familiar with the Azure Bot Framework, check out this link – https://dev.botframework.com/

Anyway – despite all this, for this post today we’ll just focus on something pretty simple – making usage of the Text Analytics API right within SQL Server 2016.

So lets get to scoring some sentiment!

Continue reading

Azure SQL IaaS – TempDB Performance on SSD vs Premium Disk

When it comes to deploying SQL Server onto an Azure IaaS VM, it is sometimes difficult to know the best option to deploy the SQL Server tempdb database for your workload.

In many of the SQL templates on the marketplace it is often deployed to the C:\ by default after which you should redeploy the database to either D:\ (local SSD) or to an attached premium disk (P10, P20, P30).  The Microsoft SQL on IaaS Performance Best Practice article states both are possibilities under certain circumstances, however it does not provide empirical workload evidence as which to use when.

For those who have not seen the article – read here – https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sql-performance/

The key comment of interest is this…

For D-series, Dv2-series, and G-series VMs, the temporary drive on these VMs is SSD-based. If your workload makes heavy use of TempDB (e.g. for temporary objects or complex joins), storing TempDB on the D drive could result in higher TempDB throughput and lower TempDB latency.

…and this…

So I thought lets test a OLTP type SQL workload!

AND SO – lets do some testing to validate this puppy!

Continue reading

Azure Cortana Intelligence Suite – with Azure Data Catalog

I presented at our local Melbourne SQL Server User Group (SSUG) this week on Azure Cortana Intelligence Suite – and the integration with the Azure Data Catalog.  If you are interested to see the presentation then have a look here – https://mrfoxsql.wordpress.com/presentations/

One of the things which stood out to me from attendees is that the number of new services and new cloud terms which are being introduced across the industry is sometimes difficult to keep track of – especially when you have a day job to do!  Right now I cannot think of another time in IT where the rate of change is so dramatic.

More-so it can also sometimes be hard to know which services to use together to get the best architecture outcome.

So this blog post is aimed to explain at a high level the Azure Cortana Intelligence Suite of services and one example of how they could be deployed together in a cohesive architecture.

I will talk to each of these services in the context of a business solution to manage a farm of wind turbines!

WindFarms

 

Continue reading

Azure DocumentDB – Preparing, Loading and Querying Data

Recently I have been using Azure DocumentDB to validate several business use cases for a variety of application purposes.

For those SQL DBA’s and others who are new to Azure DocumentDB, its a recent entrant to the NoSQL document database world, and as its a PaaS document database cloud service it has the agility, scalability and availability of the Azure Cloud.

Being a schema-less Azure PaaS “document database” for my use case I wanted to verify…

  • basic costing and performance levels
  • methods to create valid JSON documents from SQL Server
  • methods to load JSON documents into Azure DocumentDB
  • performing basic like-for-like document query comparisons with SQL Server

Some homework reading for those interested…

 

And so… let get into the belly of Azure DocumentDB!

Continue reading

PASS 2016 – A Business Case for Attendance

For those tech professionals who play in the SQL Server and Business Intelligence world (and let me say that’s a huge number!) the PASS Summit provides one of the most fantastic and focused learning and networking experiences available.

I simply cannot think of any other SQL/BI event that brings together such a range of deep expertise and learning, open networking opportunities, knowledge sharing and community involvement than that of PASS.  In fact I know of several of my Aussie counterparts whom attend no other training throughout the year except for the complete 5 day immersion that only PASS can offer.

For those new to PASS – see here http://www.sqlpass.org/summit/2016/Welcome.aspx

I have attended a few PASS events now (and have been lucky enough to present in 2015!) and each event has been better than the last.  With the Call For Speakers now closed I can see that this year’s PASS 2016 in Seattle (Oct 25-28) is shaping up to be yet another fire cracker!

Will I be presenting this year?  Well, I actually didn’t submit any extracts.  I started a job with Microsoft (Azure) and needed to focus on the ever expanding learning curve!

Anyway – I have blogged about my experiences submitting and attending PASS many times…

Unfortunately such amazing resources don’t always come free and these uncertain financial times can create an uncertain training budget for many organisations and individuals alike.

As such for those attendees whom find themselves in this position I have prepared a brief below which can hopefully help you prepare a convincing business case for attendance.  This is one I wrote a few years back after presenting at a few SQL Saturday and PASS virtual chapter sessions and helped me position the key benefits of PASS.

Take from this what you will – and be sure to update this with the most recent content and also to reflect your personal and business circumstances.

GOOD LUCK!  And I hope to see you at PASS 2016!

Continue reading

Compression Performance with Low Key Selectivity

When it comes to enabling data compression in SQL Server I suspect most people don’t do a lot of testing as to the impacts of either enabling compression and its ongoing maintenance.  I assume most are interested in its ongoing performance for queries, and you know that’s damn fair enough too!

If you read the Microsoft Data Compression Whitepaper (and I mean whos got time to do that!) then it has some interesting technical tidbits burried deep in there which affect the impacts of applying and managing data compression.

https://technet.microsoft.com/en-us/library/dd894051(v=sql.100).aspx

Here are some of the ones of interest…

SQL Server uses statistics on the leading column to distribute work amongst multiple CPUs, thus multiple CPUs are not beneficial when creating, rebuilding, or compressing an index where the leading column of the index has relatively few unique values or when the data is heavily skewed to just a small number of leading key values – only limited effective parallelism will be achieved in this case.


And this…

Compressing or rebuilding a heap with ONLINE set to ON uses a single CPU for compression or rebuild. However, SQL Server first needs to scan the table—the scan is parallelized, and after the table scan is complete, the rest of the compression processing of the heap is single-threaded.


And this…

When a heap is compressed, if there are any nonclustered indexes on the heap, they are rebuilt as follows:
(a) With ONLINE set to OFF, the nonclustered indexes are rebuilt one by one.
(b) With ONLINE set to ON, all the nonclustered indexes are rebuilt simultaneously.
You must account for the workspace required to rebuild the nonclustered indexes, because the space for the uncompressed heap is not released until the rebuild of the nonclustered indexes is complete.

 

That first one is a cracker – it hit me once when compressing a SQL Server table (600M+ rows) on a 64 core Enterprise SQL Server.  After benchmarking several other data compression activities I thought I had a basic “rule of thumb” (based on GB data size and number of rows)… of which just happened to be coincidence!

This also begs the question of why would you use low selectivity indexes?  Well I can think of a few cases – but the one which stands out the most is the identification of a small number of rows within a greater collection – such as an Index on TYPE columns (ie; [ProcessingStatusFlag] CHAR(1) = [P]rocessed, [U]nprocessed, [W]orking, [F]ailed, etc)

… AND SO – lets do some testing to validate this puppy!

Continue reading