Business Intelligence Blogs

View blogs by industry experts on topics such as SSAS, SSIS, SSRS, Power BI, Performance Tuning, Azure, Big Data and much more! You can also sign up to post your own business intelligence blog.

«November 2015»

DirectQuery in Power BI Desktop

In the latest Power BI Desktop a new Preview features was released that now allows you to connect using DirectQuery to either SQL Server or Azure SQL Databases.  DirectQuery is a really neat feature that allows you to point to the live version of the data source rather than importing the data into a data model in Power BI Desktop. 

Normally when you want to get an updated dataset in the Power BI Desktop you would have to manually click the refresh button (this can be automated in the Power BI Service), which would initiate a full reimport of your data.  This refresh could take a variable amount of time depending on how much data your have.  For instance, if you’re refreshing a very large table you may be waiting quite a while to see the newly added data. 

With DirectQuery data imports are not required because you’re always looking at a live version of the data.  Let me show you how it works!

Turning on the DirectQuery Preview

Now, because DirectQuery is still in Preview you must first activate the feature by navigating to File->Options and settings->Options->Preview Features then check DirectQuery for SQL Server and Azure SQL Database


Once you click OK you may be prompted to restart the Power BI Desktop to utilize the feature.

Using DirectQuery in Power BI Desktop

Next make a connection either to an On-Premises SQL Server or Azure SQL database.

Go to the Home ribbon and select Get Data then SQL Server.


Provide your Server and Database names then click OK. ***Do not use a SQL statement.  It is not currently supported with DirectQuery***


From the Navigator pane choose the table(s) you would like to use.  I’m just going to pick the DimProduct table for this example and then click Load.  You could select Edit and that would launch the Query Editor where you could manipulate the extract.  This would allow you to add any business rules needed to the data before visualizing it.


Next you will be prompted to select what you want to connect to the data. Again, Import means the data

Read more

The Big Data Blog Series

Over the last few years I’ve been speaking a lot on the subject of Big Data. I started by giving an intermediate session called “Show Me Whatcha’ Workin’ With”. This session was designed for people who had attended a one hour introductory session that showed you how to load data, to look at possible applications … Continue reading The Big Data Blog Series
Read more

SSIS For Each Column in a Data Flow

  • 29 May 2011
  • Author: MikeDavis
  • Number of views: 13746

When you want to transform data in a data flow task in SSIS, the derived column transform performs this very fast due to the batch method. But if you have hundreds of columns that need the same transform performed on them, it can be very time consuming to create each of these derived column expressions. You can use a script task to loop through the columns in a data flow and perform the same task on every input column you select in the script task. Keep in mind, this will degrade performance, but it will make the development faster.

Here are the first few rows of the million row table I am going to use to demonstrate the functions. Let’s say we need get just the first 3 characters on each of these fields.

SSIS For Each Column in a Data Flow

First I will show how to complete this with a derived column. It is just a simple substring command on each column. This is a small example. Imagine if you had hundreds of columns that need this transform. This would be a very time consuming process. This derived column will work and perform very fast.

SSIS For Each Column in a Data Flow

With a script task you will need to import the System.Reflection namespace. This will allow you to refer the columns in the data flow. Then we will create a For Each Loop to loop through each column.  The first thing we need to do is place a check next to each column we want the script task to substring and set them to read write so we can update them.

SSIS For Each Column in a Data Flow

Here is the code in the script task that will perform the substring function needed.

        Dim column As IDTSInputColumn100
        Dim rowType As Type = Row.GetType()
        Dim columnValue As PropertyInfo

        For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
            columnValue = rowType.GetProperty(column.Name)
            Dim strCol As String = columnValue.GetValue(Row, Nothing).ToString()
            strCol = strCol.Substring(0, 3)
            columnValue.SetValue(Row, strCol, Nothing)

Both of the above transforms will give you the below output.

SSIS For Each Column in a Data Flow

The derived column performed this operation on one million rows in 6 seconds. The Script task took 406 seconds, over 6 minutes, on the same one million rows. This is a massive performance loss. I would not suggest using the script task method due to the performance loss. Although if you know the number of rows will always be low and the table is very wide and has a lot of columns that you need to perform the same function, then it can be used with no scalability.

Rate this article:
No rating


Other posts by MikeDavis

Please login or register to post comments.