148 Hands-On Microsoft SQL Server 2008 Integration Services Review In this Hands-On exercise, you learned to use the FTP task to download multiple files from a remote FTP server. You have hard-coded file paths for the RemotePath and LocalPath fields in this exercise. However, you can use variables to change these folders dynamically during run time and download or upload files to different folders each time the package is run. For example, you can download files to the folders derived on the basis of date—for instance, the folder name includes the date as a part of the name (such as Zip20091011). So, in that case, when the package is run, the path variable is calculated dynamically based on the current date and loads the files to the current date folder. Variables are not the only means to derive values dynamically. SSIS enables you to derive values dynamically at run time, based on the outcome or values generated by other tasks, using property expressions as well that do not necessarily contain variables in their expressions. You have been introduced to the property expressions in Chapter 3 and will learn more about them in Chapter 8. Execute Process Task Generally enterprises have home-grown or custom-developed applications to apply business processes to the information they deal with. These applications might be to make sure that the information goes to right people in the right format after corporate standards are applied on to it, or it might be a custom application to transform a mainframe encrypted and compressed (such as EBCDIC format) file into a standard text format file. The company might wish to reuse the functionality that already exists and has been accepted within the corporate setting. Integration Services allows you to leverage your existing investment and bring those processes in the workflow within the packages by using the Execute Process task. With this task, you can run a business application or a batch file to perform specific business functions that already exist or can’t be developed easily in SSIS; the good thing about this is that the output from running the business application or batch file can then be included in the SSIS package workflow. You could unzip files, call custom programs to break a huge file into smaller files that can be easily handled for transformation operations, run a custom report generation application from within SSIS, and then use a Send Mail task to distribute those reports. To keep it simple, we will use this task to unzip the files you have downloaded from the FTP server in the preceding Hands-On exercise. You have begun your journey to understand the SSIS workflow. In the last Hands-On exercise, you downloaded zipped files from an FTP server in the C:\SSIS\ downloads folder. In the next exercise, you will unzip the downloaded files using the Execute Process task, which will run a batch file to do the trick. Chapter 5: Integration Services Control Flow Tasks 149 Hands-On: Expanding Downloaded Files Adventure Works Bikes dealers’ reports need to be downloaded and extracted so that they can be imported into the SQL Server database. You have already downloaded these files; here you will extract flat files from the downloaded files. Method You have downloaded zipped files DealerSales01.zip and DealerSales02.zip in the C:\SSIS\downloads folder but can’t use these files, as SSIS can’t read zipped files. You need to extract flat files from the zipped files to be able to import them into SQL Server using SSIS. You will run a batch file to extract the flat files from the zipped files and will deploy an Execute Process task to run this batch file at run time. Because you want to unzip more than one file, you will enumerate the files you want to unzip; an enumeration function will be provided by a Foreach Loop Container. In the last chapter, you used a Foreach Loop Container with the Foreach ADO enumerator to enumerate over rows; now you will be using the Foreach File enumerator to enumerate over files. The steps for this will be: Create a batch file to expand the downloaded Zip file. c Use a Foreach Loop Container to enumerate over multiple Zip files. c Add an Execute Process task to call a batch file from within the SSIS package to c extract the flat files. Exercise (Creating Batch File) The downloaded files have been compressed using a Freezip freeware package that has been included in the distribution software provided for this book. Go to C:\SSIS\ Freeware\Freezip to find the software files there. However, if you use some other software utility to compress files, you can use a command-line interface of that utility to unzip files in the same manner as shown in this exercise. Open a blank text file using Notepad and type the following commands: DEL C:\SSIS\downloads\%1.txt C:\SSIS\Freeware\Freezip\UNZIP %1.zip %1.txt The first line deletes the previously present text file, and the second line extracts a text file from the Zip file specified as an argument in the command. Save the file as C:\SSIS\RawFiles\ExtractFiles.bat. Make sure that the file has been saved with the .bat extension and not with the .txt extension. This file should already be available in the RawFiles folder, as it has been included in the distribution software provided for this book. 150 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Enumerating Multiple Zipped Files) You will add and configure a Foreach Loop Container to enumerate over the Zip files in the downloads folder and populate the fname variable with the filename. 1. Start BIDS and open the Control Flow Tasks project. Go to Solution Explorer and right-click the SSIS Packages folder in the Control Flow Tasks project and choose New SSIS Package from the context menu. This will add a new package called Package1.dtsx in the project. 2. Right-click Package1.dtsx and rename it as Expanding downloaded files.dtsx. 3. Drop a Foreach Loop Container from the Toolbox onto the designer surface and open the editor by double-clicking it. Type the following in the General page: Name Enumerating Zip Files Description This task enumerates Zip files in the C:\SSIS\downloads folder. 4. Go to Collection page and make sure Foreach File Enumerator is selected in the Enumerator field. 5. In the Enumerator configuration area, specify C:\SSIS\downloads in the Folder field either by typing it in directly or by using the Browse button to select the folder. You can enumerate multiple files of the same type with the Foreach Loop Container at one time. For this, you can use the Files field to limit or select precisely the files of one type you want to access in a folder containing different types of files. In the Files field, type *.zip to specify the types of files you want the Foreach Loop Container to enumerate on (refer Figure 5-3). Use the Files field to limit or select precisely the files you want to access in a folder containing different types of files. 6. The most interesting bit in this page is the Retrieve File Name section. This is where you will choose how the filename should be retrieved from three options. Before you jump into selecting one of the options, you must understand what these options are and how they work. The filename is treated in SSIS as consisting of three parts: the path pointing to the location where the file might be stored, the name portion of the file, and the extension of the file indicating its type. Fully Qualified This option will return the path, the filename, and the extension all as a single string. The path portion of the filename can be a full folder path in a universal naming convention (UNC) or absolute form. For example, the fully qualified name retrieved can be \\ComputerName\Sales\DealerSales01.zip, or it can be in the form C:\SSIS\downloads\DealerSales01.zip, depending on what you have specified in the Folder field. Name And Extension Choosing this option returns the name portion of the filename along with its extension—for example, DealerSales01.zip. Name Only Choosing this option will return only the name portion of the filename—for example, DealerSales01. Chapter 5: Integration Services Control Flow Tasks 151 Select Name only, as you will need only the name of the file to be passed as an argument to the batch file. 7. Go to the Variable Mappings page, click in the Variable field, and select <New Variable…> to create a new variable. Name the variable fname in the Expanding Download Files Container with the string type value and return to the Foreach Loop Editor. Note that the Index field has automatically been allocated a value of 0 for this variable. Make sure that you type the variable name in lowercase— fname—as the variables are case sensitive. Click OK to complete the Foreach Loop Container’s configuration. Figure 5-3 Configuring the Foreach Loop Container with the Foreach file enumerator 152 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Calling Batch File Using Execute Process Task) Now you will add Execute Process task inside the Foreach Loop Container. This task will use the fname variable populated by Foreach Loop Container to pass as an argument to the ExtractFiles.bat file. 8. Drag and drop the Execute Process task from the Toolbox within the Foreach Loop Container and double-click it to open the Execute Process Task Editor. Type the following in the General page of the task editor: Name Call ExtractFiles batch file Description This task extracts DealerSales flat files. 9. Go to Process page to configure options for this task. Verify that the RequireFullFileName field has the default True selected. This means that the task will fail if it cannot find the batch file at specified location. 10. In the Executable field, specify C:\SSIS\RawFiles\ExtractFiles.bat to point the task to the batch file. 11. You need to specify the filename as an argument to the batch file. The filename argument has been populated in the fname variable in the earlier part of this exercise. However, Arguments field doesn’t allow use of a variable, so you will be using property expressions in the next step to attach the fname variable to the Arguments field. Leave the Arguments field blank for the moment and in the WorkingDirectory field, specify C:\SSIS\downloads, as shown in Figure 5-4. In the next three fields, you can use variables for the input, output, or error values as explained here: c StandardInputVariable Specify a variable to provide the input to the process. �c StandardOutputVariable Specify a variable to capture the output of the process. c StandardErrorVariable Specify a variable to capture the error output of the process. You are not using these fields for this exercise, so leave them blank. Using the FailTaskIfReturnCodeIsNotSuccessValue field, you can configure the task to fail if the process exit code does not match the value you provide in the SuccessValue field. Also, you can specify a time-out value in seconds and choose to terminate the task after time-out period. Finally, if it is important, you can choose the window style in which the task starts a process. Leave these fields set at their default values and move on to the Expressions page. Chapter 5: Integration Services Control Flow Tasks 153 12. In this step, you will pass the value of the fname variable to the Arguments field using property expressions. Property expressions are explained in detail in Chapter 8, but here for a quick introduction, property expressions will apply the run-time values of the fname variable on to the Arguments property. Click the ellipsis button in the Expressions field to open the Property Expressions Editor. Click in the Property field and select Arguments from the drop-down list. Click the ellipsis button in the Expression field to open an Expression Builder dialog box. Expand Variables in the left pane of the window and locate the User::fname variable in the list. Drag this variable and drop it in the Expression box. Click OK three times to close the task editor. 13. Press 5 to start debugging the task. You should see the tasks turning quickly from yellow to green (see Figure 5-5). Figure 5-4 Configuration of the Execute Process task 154 Hands-On Microsoft SQL Server 2008 Integration Services 13. Press -- to save all the files in this project and then choose File | Close Project. 14. Go to the downloads folder and see that the DealerSales01.txt and DealerSales02 .txt files have been extracted. Review You have learned how to use the Execute Process task to expand the zipped files in this Hands-On exercise. Now you know that SSIS provides an option to run the custom-built applications that you need to use for your day-to-day functions. Also, not only can you run custom-built applications, but you can also use the results of running such an application by using the StandardOutputVariable property and use this value in other places in the package such as building a workflow expression. Figure 5-5 Executing the Expanding downloaded files.dtsx package Chapter 5: Integration Services Control Flow Tasks 155 File System Task The File System task allows you to perform operations on the files and directories in the file system. You will be using this task in your packages to perform operations such as copy, delete, and move on files and folders. If you have used DTS 2000, you might have written ActiveX code to rename, copy, or move files or folders. Though you may still have to write a piece of code in the Script task to perform some of the complex operations, most of your requirements for performing operations on files and folders can be met by using the File System task and hence less code to maintain. In the Task Editor of File System task, you will see an Operation field that offers a drop-down list of operations it can perform. Following are the brief descriptions of the operations that the File System task can perform: Copy directory c Copies a folder from one location to another. Copy file c Copies a file from one location to another. Create directory c Creates a directory in a specified location. Delete directory c Deletes a directory from a specified location. Delete directory content c Deletes all the files and folders in the specified directory. Delete file c Deletes the specified file. Move directory c Moves a folder from one location to another. Move file c Moves a file from one location to another location. Rename file c Renames a file in the specified location. You can actually move the file to a new location while renaming by specifying a new path in the name, though you cannot rename the file while moving it to a new location. Set attributes c Sets Archive, Hidden, ReadOnly, and System attributes on the files and folders. If you do not specify any attribute in your selections, the Normal attribute will be applied, which is exclusive to all the other attributes. The File System task has a dynamic layout in which the available fields change depending on the operation you choose. For example, the Delete Directory, Delete Directory Content, and Delete File Operations do not use a destination, and hence the DestinationConnection field doesn’t appear when you choose any of these operations. The SourceConnection, however, is used by all of the operations. The SourceConnection and the DestinationConnection can be specified by using a File Connection Manager to refer to a file or a folder or by providing a variable that contains the connection strings. 156 Hands-On Microsoft SQL Server 2008 Integration Services The following Hands-On exercise for this task will help you understand the configuration options. In the earlier exercises, you have downloaded and unzipped the dealer sales files and hence have compressed Zip files and uncompressed text files in the C:\SSIS\downloads folder. Your next problem is to archive the Zip files for historical purposes. Hands-On: Archiving Downloaded Files The downloaded reports of dealers of Adventure Works Bikes are to be archived for historical purposes. You want to keep the downloaded zipped files in the Archive folder. Method In this exercise, you will use the Foreach Loop Container and the File System task to copy these files one by one to the C:\SSIS\downloads\Archive folder. The step-by-step method is as follows: Configure the Foreach Loop Container to enumerate over files and house the File c System task in it. Your File System task will copy one file with each iteration to the Archive folder. Configure the File System task to move files to the Archive folder one by one with c each iteration of the Foreach Loop Container. Exercise (Configure Foreach Loop Container) In first part of the exercise you will configure the Foreach Loop Container to enumerate files using Foreach File enumerator and populate a variable with the filename. This variable will be used by File System task to determine the name of the file to be copied over. 1. Open the Control Flow Tasks project in BIDS. Go to the Solution Explorer, right-click the SSIS Packages folder, and choose New SSIS Package from the context menu. This will add a new package, Package1.dtsx, to the project. 2. Right-click Package1.dtsx and rename it as Archiving downloaded files.dtsx. 3. From the Toolbox, drag and drop the Foreach Loop Container onto the Control Flow panel. 4. Make sure that the Foreach Loop Container is selected; open the Variables window from the default position on the left side of the BIDS interface. Alternatively, click the Foreach Loop Container and choose View | Other Windows | Variables. Chapter 5: Integration Services Control Flow Tasks 157 Click Add Variable, the left-most button in the toolbar of the Variables window. Specify the following for this variable: Name Fname Scope Foreach Loop Container Data Type String Value Xyz The value you’ve assigned to fname variable will be changed at run time when the Foreach Loop Container reads the filename and populates this variable during each iteration. The value assigned is just a placeholder value, so you can realize the Fname variable changes values dynamically. Last, be careful with the case of the Name field, as SSIS variables are case-sensitive. 5. Right-click the Foreach Loop Container and choose Edit from the context menu to open the Foreach Loop Editor. In the General page of the container, type the following: Name Enumerating Files Description This container enumerates the files and populates the fname variable. 6. Go to the Collection page. Make sure it shows the Foreach File Enumerator in the Enumerator field and specify the options shown in Figure 5-6. Enumerator Foreach File Enumerator Folder C:\SSIS\downloads Files *.zip Retrieve file name Fully qualified 7. Move on to Variable Mappings page. You will map fname variable to the filename retrieved by the Foreach Loop Container so that later on in the package you can get the filename by the fname variable. Click in the highlighted area under the Variable column and then click the drop-down arrow that appears. From the drop- down list, select the variable User::fname, configured earlier in the exercise. A 0 will appear under the Index field. Index indicates the position of the string within the collection. As you will be accessing only one filename at a time, Index will be set to 0. Click OK to close the Foreach Loop Editor. . 148 Hands-On Microsoft SQL Server 2008 Integration Services Review In this Hands-On exercise, you learned to use the FTP task to download multiple files from a remote FTP server. You. providing a variable that contains the connection strings. 156 Hands-On Microsoft SQL Server 2008 Integration Services The following Hands-On exercise for this task will help you understand the. Configuring the Foreach Loop Container with the Foreach file enumerator 152 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Calling Batch File Using Execute Process Task) Now you