SQL - Add Header Row to SSIS Generated CSV Files

After generating a flat file in SSIS I needed to update that file with the column headers as the first row.  In order to do this I used a SQL stored procedure.

STEP 1: Run the code below to create the stored procedure.

USE MIGRATION_DB

IF OBJECT_ID('GenerateBCPforSSIS') IS NOT NULL
DROP PROC GenerateBCPforSSIS
GO

CREATE PROCEDURE GenerateBCPforSSIS
(
@db_name varchar(1000),
@table_name varchar(1000),
@file_name varchar(1000)
) AS

Declare @Headers varchar(MAX),@HeadersRaw varchar(MAX),@sql varchar(MAX), @header_file varchar(MAX), @filename_short varchar(MAX)

--Generate column names as a recordset
Select @Headers = IsNull(@Headers + ',', '') + '""' + Column_Name + '""'
From INFORMATION_SCHEMA.COLUMNS
Where Table_Name = @table_name ORDER BY ORDINAL_POSITION ASC

--Create a dummy file to have header data
select @header_file=substring(@file_name,1,len(@file_name)-charindex('\',reverse(@file_name)))+'\data_file.csv'

set @sql = 'bcp "select ''' + @Headers + '''" queryout "'+@header_file+'" -c -C RAW -t, -S localhost -T'
print @sql

set @sql = 'type "'+@file_name+'" >> "'+@header_file+'"'
print @sql

set @sql = 'del "'+@file_name+'"'
print @sql

set @filename_short = reverse(substring(reverse(@file_name),1,charindex('\',reverse(@file_name)) -1 ) )

set @sql = 'rename "'+@header_file+'" '+'"'+@filename_short+'"'
print @sql

GO




STEP 2: Run the following command which will produce sql commands for all the views and tables in your database.  You can add where clauses to the statements to limit which views/tables commands are created for.  Note: Make sure to update the MIGRATION_DB string to the name of your database

USE MIGRATION_DB

SELECT 'exec GenerateBCPforSSIS ''MIGRATION_DB'',''' + name + ''',''E:\SSIS Output\' + name + '.csv''' AS CommandName
FROM sys.views 

UNION

SELECT 'exec GenerateBCPforSSIS ''MIGRATION_DB'',''' + name + ''',''E:\SSIS Output\' + name + '.csv''' AS CommandName
FROM sys.tables



STEP 3: Copy the output from the command in step 2 and paste it into a new SQL query window.


STEP 4: Copy the output from step 3 and past it into a batch file.


STEP 5: Run the batch file, it will open your SSIS csv file and insert a header row with the column names. 


Notes:



  • The CVS file output from SSIS must match the names of your Views/Tables

  • If you change, add, or remove any of the columns in your table/view you will have to run steps 2-5 again.

Comments

  1. Chris LesnarJune 18, 2021 at 5:21 AM

    Wow, that is a very good article about SSIS Upsert and its related aspects.
    SSIS Upsert

    ReplyDelete
  2. noraFebruary 14, 2022 at 8:07 AM

    "SQL Injection" is subset of the unsubstantiated/unsanitized client input weakness ("cradle floods" are an alternate subset), https://onohosting.com/

    ReplyDelete
  3. HendryFebruary 15, 2022 at 5:29 AM

    This is a job which is basic to the accomplishment of your undertaking so assuming there is somebody you in your association who meets your prerequisites secure them for your venture by distinguishing them as a basic asset in your task sanction.https://hostinglelo.in/

    ReplyDelete

Post a Comment

Popular posts from this blog

Add User As Local Administrator On Domain Controller

I recently was settting up a new Microsoft SharePoint 2010 machine and had promoted the machine to a domain controller before creating my SharePoint admin accounts.  I needed to add several of my accounts to the local Administrators group.  Unfortunately after you promote a server to a domain controller you can no longer access the GUI for Local Users and Groups.  Instead I had to use the command line to add the users. Open a command promt using the "Run as administrator" function and then run the following command. net localgroup Administrators /add {domain}\{user} Note: do not include the {} brackets.
Read more

Calling Dataverse Web API in PowerShell using Client Credentials

Image
Connecting to Dataverse using PowerShell can be very helpful for data migrations and use within Azure DevOps. Connecting to an instance in a non-interactive way can be tricky though. This article will provide you the links you need for creation and App registration and adding the app user to your environment. You can then utilize the script provided to call Web API requests including ones you define using the new Custom API functionality now available. The script was written so that it is not dependent on any outside libraries such as the Microsoft.Xrm.Tooling connector. This is helpful in situation where involving an outside library will slow down your deployment time by having to be approved in a change control board. If utilizing an outside code library is not a concern you can create a connection to Dataverse utilizing the Microsoft.Xrm.Tooling connector Get-CrmConnection cmdlet. Create App Registration The first thing to do is create an App Registration within Azure AD for ...
Read more

How to Create SharePoint Items with Power Automate Desktop

Image
Overview Power Automate Desktop is a great way to automate many of your daily task so you can focus on real work. A prime example of this is getting data from one place to another, especially when those data sources do not have an API such as a legacy desktop application or a file. In this example I demonstrate several ways in which an Excel sheet containing fictitious customer data could be loaded into a SharePoint list. In order to do this I first generated some fake data for my customers using Mockaroo . I then created a new SharePoint list and added some columns to match my spreadsheet. Methods My original plan was to just use the PAD recorder , which I did, but after creating that Flow I decided to find other possible ways to accomplish the task. The list below is not a complete list but should provide some ideas. The flexibility of PAD allows for an even wider range of possibilities for carrying out tasks such as these. At the end of each section I have placed the s...
Read more