Business Intelligence Blogs

View blogs by industry experts on topics such as SSAS, SSIS, SSRS, Power BI, Performance Tuning, Azure, Big Data and much more! You can also sign up to post your own business intelligence blog.

«October 2015»

Data Warehouse from the Ground Up at SQL Saturday Orlando, FL on Oct. 10th

SQL Saturday #442SQL Saturday #442 is upon us and yours truly will be presenting in Orlando, Florida on October 10th alongside Mitchell Pearson (b|t). The session is scheduled at 10:35 AM and will last until 11:35 AM. I’m very excited to be presenting at SQL Saturday Orlando this year as it’ll be my first presenting this session in person and my first time speaking at SQL Saturday Orlando! If you haven’t registered yet for this event, you need to do that. This event will be top notch!

My session is called Designing a Data Warehouse from the Ground Up. What if you could approach any business process in your organization and quickly design an effective and optimal dimensional model using a standardized step-by-step method? In this session I’ll discuss the steps required to design a unified dimensional model that is optimized for reporting and follows widely accepted best practices. We’ll also discuss how the design of our dimensional model affects a SQL Server Analysis Services solution and how the choices we make during the data warehouse design phase can make or break our SSAS cubes. You may remember that I did this session a while back for Pragmatic Works via webinar. I’ll be doing the same session at SQL Saturday Orlando but on-prem! ;)

So get signed up for this event now! It’s only 11 days away!

Read more

Create Date Dimension with Fiscal and Time

Here are three scripts that create and Date and Time Dimension and can add the fiscal columns too. First run the Dim Date script first to create the DimDate table. Make sure you change the start date and end date on the script to your preference. Then run the add Fiscal Dates scripts to add the fiscal columns. Make sure you alter the Fiscal script to set the date offset amount. The comments in the script will help you with this.

This zip file contains three SQL scripts.

Create Dim Date

Create Dim Time

Add Fiscal Dates

These will create a Date Dimension table and allow you to run the add fiscal script to add the fiscal columns if you desire. The Create Dim Time will create a time dimension with every second of the day for those that need actual time analysis of your data.

Make sure you set the start date and end date in the create dim date script. Set the dateoffset in the fiscal script.

Download the script here:


Read more

SSIS For Each Column in a Data Flow

  • 29 May 2011
  • Author: MikeDavis
  • Number of views: 13356

When you want to transform data in a data flow task in SSIS, the derived column transform performs this very fast due to the batch method. But if you have hundreds of columns that need the same transform performed on them, it can be very time consuming to create each of these derived column expressions. You can use a script task to loop through the columns in a data flow and perform the same task on every input column you select in the script task. Keep in mind, this will degrade performance, but it will make the development faster.

Here are the first few rows of the million row table I am going to use to demonstrate the functions. Let’s say we need get just the first 3 characters on each of these fields.

SSIS For Each Column in a Data Flow

First I will show how to complete this with a derived column. It is just a simple substring command on each column. This is a small example. Imagine if you had hundreds of columns that need this transform. This would be a very time consuming process. This derived column will work and perform very fast.

SSIS For Each Column in a Data Flow

With a script task you will need to import the System.Reflection namespace. This will allow you to refer the columns in the data flow. Then we will create a For Each Loop to loop through each column.  The first thing we need to do is place a check next to each column we want the script task to substring and set them to read write so we can update them.

SSIS For Each Column in a Data Flow

Here is the code in the script task that will perform the substring function needed.

        Dim column As IDTSInputColumn100
        Dim rowType As Type = Row.GetType()
        Dim columnValue As PropertyInfo

        For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
            columnValue = rowType.GetProperty(column.Name)
            Dim strCol As String = columnValue.GetValue(Row, Nothing).ToString()
            strCol = strCol.Substring(0, 3)
            columnValue.SetValue(Row, strCol, Nothing)

Both of the above transforms will give you the below output.

SSIS For Each Column in a Data Flow

The derived column performed this operation on one million rows in 6 seconds. The Script task took 406 seconds, over 6 minutes, on the same one million rows. This is a massive performance loss. I would not suggest using the script task method due to the performance loss. Although if you know the number of rows will always be low and the table is very wide and has a lot of columns that you need to perform the same function, then it can be used with no scalability.

Rate this article:
No rating


Other posts by MikeDavis

Please login or register to post comments.