178 Hands-On Microsoft SQL Server 2008 Integration Services You may want to test the connection by clicking Test; once you get the “Test connection succeeded” message, click OK twice to return to the Message Queue Task Editor. 13. Leave the Send Message option selected in the Message field. The settings in General page should be like the ones shown in Figure 5-11. 14. Go to the Send page. Leave UseEncryption set to False. This field can be used to encrypt your messages using an encryption algorithm (RC2 or RC4) specified in the EncryptionAlgorithm field. Figure 5-11 Message Queue task configured to send messages Chapter 5: Integration Services Control Flow Tasks 179 15. Select Variable message in the MessageType field, as you will be sending a variable value in the message. Select User::txtfname in the VariableMessage field. Click OK to close the editor window. 16. Save the package by pressing --. 17. Using Windows Explorer, move the DealerSales01.zip and DealerSales02.zip files from the C:\SSIS\downloads\Archive folder to the C:\SSIS\downloads folder that have been moved to the Archive folder in earlier exercise. 18. Switch to BIDS and press 5 on the keyboard to start debugging. When the package runs, notice that the tasks change color from yellow to green twice, indicating that the package has moved two files and sent out two messages. When all the tasks turn green, stop debugging from the Debug menu or by pressing - 5. Close the project by choosing File | Close Project. 19. Switch to the Server Manager and notice two messages in the Queue messages folder under SSISprivQ. (You might have to refresh the folder if the window was already open.) Exercise (Build Importing Expanded Files Package) In this part of the exercise, you will build a new package, which will pick up variable messages sent by the Archiving Downloaded Files package earlier from the SSISprivQ messaging queue. 20. In the Solution Explorer, add a new package and rename it Importing expanded files.dtsx. 21. Start building the package by dropping the For Loop Container on the Control Flow Designer surface. 22. Right-click anywhere on the Designer surface and choose Variables from the context menu. Create three variables: filename, maxcounter, and mincounter with the details shown in the Figure 5-12. Figure 5-12 Creating variables for the text filenames and For Loop Container 180 Hands-On Microsoft SQL Server 2008 Integration Services Be sure to specify a value of C:\\SSIS\\RawFiles\\DealerSales_placeholder.txt to the filename variable and a value of 2 to the maxcounter variable. The mincounter and maxcounter variables will be used in the For Loop Container, while the filename variable will be used to receive the variable value from the messages sent by Archiving Downloaded Files package using the Message Queue task. 23. Double-click the For Loop Container icon to open the For Loop Editor. Type the following in the General page of the editor: Name Import Loop Description This For Loop Container imports DealerSales files one by one. InitExpression @mincounter = 0 EvalExpression @mincounter < @maxcounter AssignExpression @mincounter = @mincounter + 1 As the initial value assigned to the mincounter variable in the InitExpression field is 0 and the maximum value specified for maxcounter variable while creating it is 2, the Import Loop will loop twice at run time, after which the expression defined in EvalExpression field will become false and the loop will stop running. Click OK to close the For Loop Editor window. 24. From the Toolbox, drag and drop the Message Queue task within the Import Loop (For Loop) Container. Open the Message Queue Task Editor window by double-clicking its icon and type the following on the General page: Name Receive variable message Description This task receives variable messages sent by ‘Archiving downloaded files’ package 25. Leave the Use2000Format field set to the default False value. 26. Create an MSMQ connection manager in the MSMQConnection field exactly as specified in Step 3 of the preceding series of steps. 27. Select Receive Message in the Message field. Note that Send changes to Receive on the left pane of the window. Go to the Receive page. 28. On the Receive page, select True in the RemoveFromMessageQueue field. This value lets you delete the message from the queue after the Message Queue task has received it. You may prefer to keep the message in the queue in situations when you have multiple subpackages to read a message. 29. You can choose to display an error message if the task fails with timeout in the ErrorIfMessageTimeout field. For now, leave it set to False. However, if you select True in this field, you will be able to specify a time-out value in seconds in the TimeoutAfter field. Chapter 5: Integration Services Control Flow Tasks 181 30. Choose Variable Message in the MessageType field and No Filter in the Filter field. If you select the From Package option in the Filter field, you can specify the package in the Identifier field. 31. Specify User::filename in the Variable field, as shown in Figure 5-13. 32. Click OK to close the Message Queue Task Editor. 33. Drag and drop the Bulk Insert task from the Toolbox in the Import Loop Container below the Receive variable message task. Connect the Receive variable message task to the Bulk Insert task using the on success (green colored line) precedence constraint. Figure 5-13 Configuring the Message Queue task in receive mode 182 Hands-On Microsoft SQL Server 2008 Integration Services 34. Open the Bulk Insert Task Editor and type the following in the General page: Name Importing DealerSales Description This task imports DealerSales files into Campaign DealerSales table. 35. Go to the Connection page, click in the Connection field under the Destination Connection group of options, and choose <New Connection…> to open the Configure OLE DB Connection Manager window. Choose an OLE DB connection manager from the Data Connections list to connect to Campaign database, which you created in an earlier Hands-On exercise. The good thing about the OLE DB Connection Managers is that they are available for reuse in other packages, as you have seen here. If you skipped the earlier Hands-On exercises and do not see an OLE DB Connection Manager in this window, you will have to create a new connection manager by clicking New. For details on how to create an OLE DB Connection Manager, refer back to the “Contacting Opportunities” Hands-On exercise in Chapter 4. 36. Click in the DestinationTable field and then click the drop-down arrow to see the list of tables in the Campaign database. OLE DB Connection Manager provides this list by establishing a connection to the database using the settings specified in the connection manager. Select the [Campaign] .[dbo].[DealerSales] Table from the list. 37. You can choose to specify the format of the file to be imported either directly in the task or by using a file. If you choose Use File in the Format field, you have to specify the name and location of the file in the FormatFile field, which appears on selection of the Use File option. For this exercise, choose Specify in the Format field. Leave the RowDelimiter field set to {CR}{LF} and choose Vertical Bar {|} from the drop-down list in the ColumnDelimiter field. 38. Under Source Connection options group, choose <New Connection…> to specify a File Connection Manager. As you will be using filenames provided by the Receive Variable Message task in the form of variables, you will specify a filename here as a placeholder. Specify C:\SSIS\RawFiles\DealerSales_ placeholder.txt in the File field of the File Connection Manager Editor (see Figure 5-14). You can create a blank DealerSales_ placeholder.txt file in case you’re having difficulty specifying it in the File Connection Manager. 39. Go to the Options page, where you can specify the code page of the text file in the CodePage field. Leave it set to the default RAW. 40. When the data is coming from various database systems, the data type may be in different formats. The Bulk Insert task provides four default data formats that can be imported. You can specify the data type of the input file using the DataFileType field. The options are Character, Unicode (wide) Character, Native, and Unicode (wide) Native formats. Leave it set to char. Chapter 5: Integration Services Control Flow Tasks 183 41. You can specify the number of rows in a batch in the BatchSize field. The rows specified in a batch are treated together and copied to the server as one transaction. The Bulk Insert task copies files by committing all the rows in the batch as one transaction and moving over to the next batch for another transaction. In case of an error and failure of the task, all the rows in the batch will be rolled back. For example, if you set the BatchSize equal to 10,000 rows for a table of 50,000 rows and the task fails at row number 25,001, the task will fail with 20,000 rows inserted to the table. A default value of 0 implies that all the rows will be treated as a single batch—i.e., fail or commit totally in one transaction. Figure 5-14 Configuring the Bulk Insert task to import text files 184 Hands-On Microsoft SQL Server 2008 Integration Services 42. In the LastRow field, you can specify the value of last row at which the task should stop inserting data to the table. Leave it set at the default value of 0 that means all the rows from the file will be inserted to the specified SQL table. 43. The FirstRow field is quite useful in situations where you have a large amount of data to import and the quality of data results in failing the process in between. By specifying where to start inserting the rows, you can avoid re-importing the rows that have already been imported. Using this option in conjunction with the BatchSize option helps in achieving high levels of input performance with less rework, even though the data quality may not be good. Leave the option selected to the default value of 1. 44. Moving to the Options section, you can choose any of the five options. You can also select more than one option here, which then will be listed in a comma- delimited list. Check Constraints c Checks the table and column constraints. Keep Nulls c Imports blank columns from the text file as Null values. Enable Identity Insert c Inserts explicit values into the identity column of the table. Table Lock c Locks the table during import process. Fire Triggers c Fires any existing triggers on the table while importing the data. Though none of these options will be of much help in this case, you can select Keep Nulls for this exercise. 45. You can specify the names of columns on which to sort the data in the SortedData field. This is effectively the ORDER BY clause in the bulk insert SQL statement. Leave it blank, which means do not sort on any column. 46. The rows that cannot be imported by Bulk Insert task are counted as errors. You can specify a maximum number of errors—i.e., the number of rows to fail before the task fails—in the MaxErrors field. Click OK to close the task. You have configured all the tasks and options within the tasks apart from specifying which files to import. You can do this by mapping the connection string of the placeholder File Connection Manager to the value of the variable received by the Receive variable message task. Let’s see how to do this. 47. Right-click the DealerSales_placeholder.txt File Connection Manager and choose Properties from the context menu. In the Property window, click in the Expressions field and then click the ellipsis button on this field. In the Property Expressions Editor, click in the field below the Property column and select ConnectionString from the drop-down list. Then click the ellipsis button under the Expression field to open the Expression Builder. Chapter 5: Integration Services Control Flow Tasks 185 48. In the Expression Builder, drag User::filename from the Variables list to the Expression field, as shown in Figure 5-15. Click OK twice to finish the configurations. 49. As a last step before you run the package, make sure that you have two messages queued, which were sent by the Archiving Downloaded Files package, in the message queue folder in SSISprivQ. Also, check that the DealerSales table in the Campaign database does not have any record. 50. In a real-life scenario, you will be running such a package under a schedule that wakes up after the message has been dropped in the queue just to read the message. For our exercise, press  5 to run the package. Notice that the Import Loop task turns yellow, followed by the Receive Variable Message task turning yellow and then green, indicating that it has successfully received the first variable message. Then Importing DealerSales turns yellow and stays yellow for some time before turning to green, indicating that it has successfully imported DealerSales01.txt file (Figure 5-16). This completes the first iteration of the Import Loop task and the process repeats itself for the second iteration, after which all the tasks turn green and stop processing. Figure 5-15 Using Expression Builder to create a property expression for dynamically altering the connection string of the file connection manager 186 Hands-On Microsoft SQL Server 2008 Integration Services To close the loop, check the Queue messages folder under SSISprivQ by switching over to the Server Manager and notice that the messages have been deleted. Go to SQL Server Management Studio and run the following query against the DealerSales table to see the imported data and total number of records: SELECT * FROM DealerSales You should see 242,634 rows displayed. 51. Press -- to save all the files in this solution and then choose File | Close Project. Review In this exercise, you used the Message Queue task to send and receive variables from one package to another using the Windows Message Queuing service. This can be quite useful for enterprise-wide implementations in which data and servers are scattered Figure 5-16 Executing the Importing expanded files package Chapter 5: Integration Services Control Flow Tasks 187 all over the network, and it makes sense to use all the processing power underutilized in those servers to help you complete the nightly processes within the allocated time. You also used the Bulk Insert task to import files to an SQL Server table. This is the fastest method of importing data to an SQL Server table. In this exercise, you had encountered the For Loop Container and used it to iterate twice to receive two messages using the Message Queue task and then import two text files into the SQL Server table using Bulk Insert task. Last but not the least, and perhaps the most important thing you’ve learned in this exercise, you used property expressions to change ConnectionStrings of the File Connection Managers dynamically at run time with the help of variables. Execute Package Task The Execute Package task brings the benefits of modular design to SSIS packages. It allows an SSIS (parent) package to run the other (child) packages as part of a workflow. This task can be used to run packages stored either in the SQL Server MSDB database or on the file system. While developing a solution for a business problem, you tend to build smaller packages meeting specific needs. This approach is recommended, as it helps your solution to be modular, which is easier to debug. This also helps you achieve quick results on some of the specific requirements. Once you have built all the modules of your complete solution, you can use the Execute Package task to fit all the smaller packages together to form an enterprise-wide solution. Another benefit of following a modular design for SSIS packages is less work when you have to modify a child package that is used in multiple parent packages. For example, if you want to modify a child package that is used in five parent packages, you can modify your child package only once and all the five parent packages will pick up the modified child package. While developing your packages with the modular design concept, you can reuse packages developed as modules to perform specific functions in other packages. You may, for example, develop a send mail package that reads from a table and sends a broadcast mail to all the members in the table. This functionality can be used in any package that requires sending mails. Another example could be auditing for which you can develop a package that records environment values and variable values to an audit table. You can then attach this package as a child package in all of your packages wherever you want to record auditing information. Using the Execute Package task, you can have better security controls in place. You can control access to sensitive data by dividing your package into packages that can have public access and other packages that can be accessed by administrators or management only. For example, you can separate out salary processing package from your main SSIS . connection manager 186 Hands-On Microsoft SQL Server 2008 Integration Services To close the loop, check the Queue messages folder under SSISprivQ by switching over to the Server Manager and notice. 5-12 Creating variables for the text filenames and For Loop Container 180 Hands-On Microsoft SQL Server 2008 Integration Services Be sure to specify a value of C:\SSIS\RawFiles\DealerSales_placeholder.txt. constraint. Figure 5-13 Configuring the Message Queue task in receive mode 182 Hands-On Microsoft SQL Server 2008 Integration Services 34. Open the Bulk Insert Task Editor and type the following in