.ExceptionFileOptions = 1 ‘DTSExcepFile_SingleFile70 .ExceptionFileRowDelimiter = “{CR}{LF}” .ExceptionFileTextQualifier = “” .FetchBufferSize = 1 .FirstRow = 0 .InputGlobalVariableNames = “” .LastRow = 0 .MaximumErrorCount = 0 .ProgressRowCount = 1000 .SourceSQLStatement = “” End With pkg.Tasks.Add tsk ‘Create step for task Set stp = pkg.Steps.New With stp .Name = “stp” & sBaseName .Description = sBaseName .TaskName = tsk.Name End With pkg.Steps.Add stp fctCreateDataDrivenQueryTask = stp.Name Set conSource = Nothing Set conDest = Nothing Set tsk = Nothing Set cus = Nothing Set stp = Nothing ProcExit: Exit Function ProcErr: MsgBox Err.Number & “ - “ & Err.Description GoTo ProcExit End Function Conclusion The Data Driven Query task is a useful tool when you need to specify multiple results for a data transformation. The next chapter explains how to use multiple phases with the transforma- tion tasks. DTS Connections and the Data Transformation Tasks P ART II 226 L ISTING 8.2 Continued 11 0672320118 CH08 11/13/00 4:56 PM Page 226 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 9 The Multiphase Data Pump IN THIS CHAPTER • Enabling the Multiphase Data Pump 228 •Programmatic Flow with Multiple Phases 230 • Using the Phases 233 •Creating a COM Object with Visual C++ to Program the Phases 243 •Creating a Multiphase Data Pump in Code 243 12 0672320118 CH09 11/13/00 5:03 PM Page 227 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. DTS Connections and the Data Transformation Tasks P ART II 228 The multiphase data pump option allows you to write code at several different points in the data transformation process. You can use this option with the Transform Data task, the Data Driven Query task, and the Parallel Data Pump task. If you haven’t enabled the multiphase data pump, you can write code for only one point of the data transformation process—the point at which each row is being transformed. After enabling this option, you can write code for all these phases and subphases: •Pre Source Phase—Before the source query is executed. •Row Transform Phase—Each row of data is processed. • On Transform Failure—Subphase of the Post Row Transform phase. Occurs when there is an error in the transformation. • On Insert Failure—Subphase of the Post Row Transform phase. Occurs when a record fails to be inserted. • On Insert Success—Subphase of the Post Row Transform phase. Occurs when a record is successfully inserted. • Batch Complete Phase—A batch of records is successfully inserted. • Post Source Data Phase—The rows have all been processed. • Pump Complete Phase—The transformation task has completed its work. Enabling the Multiphase Data Pump The last section of this chapter explains how to create a multiphase data pump in code. In the DTS Designer, you can enable the multiphase data pump by doing the following: 1. Right-click on the Data Transformation Services node in the Enterprise Manager. 2. Select Properties from the pop-up menu. 3. Select Show multi-phase pump in DTS Designer on the Package Properties dialog, as shown in Figure 9.1. After selecting the multiphase option, you will see a Phases filter on the Transformation tab of the Transform Data Task Properties dialog, as shown in Figure 9.2. Each transformation can implement one or more of the phases. By selecting one of the phases, you can see which trans- formations have implemented a particular phase. The Phases tab of the ActiveX Script Transformation Properties dialog shows the eight phases and subphases where you can write code, as shown in Figure 9.3. You enable a phase by select- ing the check box beside it. You can set the default entrance function and create the default code for all the phases you have selected by clicking the Auto Gen button. 12 0672320118 CH09 11/13/00 5:03 PM Page 228 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. F IGURE 9.1 You select the multiphase data pump option in the Package Properties dialog. The Multiphase Data Pump C HAPTER 9 9 T HE M ULTIPHASE D ATA P UMP 229 F IGURE 9.2 You can use the Phases filter to show the transformations that are using a particular phase. Even if you remove the multiphase option from the Package Designer, multiple phases in a transformation will remain. However, you will not be able to view all the properties of those transformations in the Package Designer without using Disconn- ected Edit. If any of your transformations do not include a Row Transform phase, you will not be able to access that phase in the Transform Data Task Properties dialog. N OTE 12 0672320118 CH09 11/13/00 5:03 PM Page 229 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. DTS Connections and the Data Transformation Tasks P ART II 230 F IGURE 9.3 You can choose which phases you want to use in the ActiveX Script Transformation Properties dialog. You can enable the multiphase data pump option in code by using the Application object. Here’s the VBScript code to do so: Function Main Dim app Set app = CreateObject(“DTS.Application”) app.DesignerSettings = DTSDesigner_ShowMultiPhaseTransforms Main = DTSTaskExecResult_Success End Function Programmatic Flow with Multiple Phases Figure 9.4 shows the programmatic flow of the phases and subphases as the Transform Data task is executed. You can choose to implement one, all, or any combination of these phases and subphases. When you implement a phase, the specified entry function is called and the code in that function is executed. The Pre Source phase is usually executed just once. You can execute it more than once by set- ting DTSTransformStat_SkipFetch as the return value from the entry function. The Row Transform phase is executed once for each record in the data source. 12 0672320118 CH09 11/13/00 5:03 PM Page 230 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. F IGURE 9.4 The programmatic flow in the multiphase data pump. The Post Row Transform phase is often executed once for each record in the data source. It is not executed if the Row Transform phase is completed successfully, and it does not attempt to insert a record into the destination. The Transform Failure subphase is executed whenever there is an error in the Row Transform phase. Either the Insert Success subphase or the Insert Failure subphase is executed after an attempt to insert a record into the destination. These subphases are mutually exclusive. The On Batch Complete phase is executed once each time a batch of records is committed. This phase is only used when Fast Load is being used. The Post Source Data and Pump Complete phases are each executed once after all the records have been processed. Information about the progress of the transformation is available through the properties of the DTSTransformPhaseInfo object. This object can be referenced both from the code of the trans- formation and from code in a transformation ActiveX script. The Multiphase Data Pump C HAPTER 9 9 T HE M ULTIPHASE D ATA P UMP 231 On Insert Failure On Insert Success OK/SkipFetch Skip Row/Skip Insert Error/Exception No Error On Transform Failure Row Transform Batch Complete Pump Complete Pre Source Post Source Skip Fetch All Rows Processed Follow Next Row in Batch/Skip Fetch Follow Next Row after Batch Post Row 12 0672320118 CH09 11/13/00 5:03 PM Page 231 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. All six properties of the DTSTransformPhaseInfo object are read-only. They are as follows: • CurrentPhase —The phase whose code is currently being executed. This property uses the values of the DTSTransformPhaseEnum , which are listed in the last section of this chapter. • CurrentSourceRow —The number of the source row currently being processed. The source rows are numbered consecutively, beginning with 1. • DestinationRowsComplete —The number of rows that have been successfully inserted into the data destination. For a Data Driven Query task, the value in this property is the number of data-driven queries that have been executed. • ErrorRows —The number of rows that have generated an error. • ErrorCode —The error code returned by the phase immediately preceding the current phase. • TransformStatus —The transformation status returned by the transformation immedi- ately preceding the current transformation, when that transformation was executed on the same row of source data. Three of these properties (CurrentSourceRow, DestinationRowsComplete, and ErrorRows) use the vt_decimal data type. When you reference these properties in VBScript, you have to con- vert them to long integers: lCurrentSourceRow = CLng(DTSTransformPhaseInfo.CurrentSourceRow) Listing 9.1 shows a function that uses several of these properties to record the state of the transformation at the time of an error. You can find this code in a file called UseAllPhases.dts on the book’s CD. L ISTING 9.1 The Properties of the DTSTransformPhaseInfo Object Can Help Determine What Happened at the Time of an Error Function fctError Dim msg, sPhase sPhase = fctPhaseName If bDisplayErrMsg Then msg = “Error in “ & sPhase & vbCrLf msg = msg & Err.Number & “ “ & Err.Description msgbox msg End If If bRecordErr Then DTSLookups(“InserttblOrderErrors”).Execute _ DTS Connections and the Data Transformation Tasks P ART II 232 12 0672320118 CH09 11/13/00 5:03 PM Page 232 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. DTSSource(“OrderID”), _ DTSSource(“TranType”), _ DTSSource(“TranDate”), _ DTSDestination(“OrderID”), _ DTSDestination(“OrderDate”), _ DTSDestination(“RequiredDate”), _ DTSDestination(“ShippedDate”), _ DTSTransformPhaseInfo.CurrentPhase, _ CLng(DTSTransformPhaseInfo.CurrentSourceRow) , _ DTSTransformPhaseInfo.ErrorCode, _ CLng(DTSTransformPhaseInfo.DestinationRowsComplete) , _ DTSTransformPhaseInfo.TransformStatus, _ Now, _ Err.Number, _ Err.Description End If End Function Using the Phases The phases and subphases differ in their access to the source and destination columns. They also have different return values that are valid for the entry function. Table 9.1 has an overview of these differences. (Where my testing varies from Books Online, the information from Books Online appears in parentheses.) T ABLE 9.1 The Columns and the Return Values Available for Each of the Phases Phase/Subphase Source Columns Destination Columns Return Values Pre Source None Write All Row Transform Read Write All Transform Failure Read Write All Insert Failure Read Write(None) OK or AbortPump Insert Success Read Write(None) OK or AbortPump Batch Complete Read(None) Read(None) OK or AbortPump Post Source Data None Write All Pump Complete Read(None) Read(None) OK or AbortPump The Multiphase Data Pump C HAPTER 9 9 T HE M ULTIPHASE D ATA P UMP 233 L ISTING 9.1 Continued 12 0672320118 CH09 11/13/00 5:03 PM Page 233 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Books Online states that the Pre Source phase, the Transform Failure subphase, and the Post Source Data phase can use all the transformation values that can be returned from a Row Transform phase. DTS Connections and the Data Transformation Tasks P ART II 234 The transformation statuses have some different behavior when used for the various phases: DTSTransformStat_SkipFetch can be used as the return value for the Pre Source phase, the Transform Failure subphase, and the Post Source Data phase, as well as for the Row Transform phase. For all of these phases, this transformation status causes the phase’s entry function to be called again immediately. DTSTransformStat_SkipInsert has the same effect as DTSTransformStat_OK when used for the Pre Source phase, the Transform Failure subphase, and the Post Source Data phase. DTSTransformStat_SkipRow can be used only for the Row Transform phase. It gener- ates an error when used for any other phase. N OTE Chapter 7, “Writing ActiveX Scripts for a Transform Data Task,” has a transformation script example that uses two of the transformation phases. I have extended that example here to use all eight of the phases and subphases. You can find it on the CD in a file called UseAllPhases.dts. The transformation script has a function called fctPhase that displays a message box telling which phase is currently active. The message is only displayed if the user sets the value of the variable bDisplayPhase to TRUE in the Pre Source phase. This function calls another function, fctPhaseName , which finds the current phase number and converts it into a phase name. These functions are shown in Listing 9.2. L ISTING 9.2 Functions That Display the Current Phase Function fctPhase Dim msg If bDisplayPhase = True Then msg = fctPhaseName msgbox msg End If End Function 12 0672320118 CH09 11/13/00 5:03 PM Page 234 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Function fctPhaseName dim sName Select Case DTSTransformPhaseInfo.CurrentPhase Case 1 sName = “Pre-Source Data Phase” Case 2 sName = “Post-Source Data Phase” Case 4 sName = “Row Transform Phase” Case 8 sName = “On Transform Failure Phase” Case 16 sName = “On Insert Success Phase” Case 32 sName = “On Insert Failure Phase” Case 64 sName = “Batch Complete Phase” Case 128 sName = “Pump Complete Phase” End Select fctPhaseName = sName End Function Pre Source Phase You can use the Pre Source phase to initialize variables that are used throughout the script. You can use either global variables or script variables declared outside a function for this purpose. If you use script variables, they will be visible only to functions in this particular script. They will not be visible outside the task or to other transformations in the same task. If you are using a text file as the destination for your transformation, you could use the Pre Source phase to write a header to the file. If you want to execute the code in this phase more than once, you can return the DTSTransformStat_SkipFetch transformation value from the entry function of the Pre Source phase. The Multiphase Data Pump C HAPTER 9 9 T HE M ULTIPHASE D ATA P UMP 235 L ISTING 9.2 Continued 12 0672320118 CH09 11/13/00 5:03 PM Page 235 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... property for the Transformation2 object in SQL Server 2000 This property contains a bitmap that indicates which phases are enabled for that particular transformation The values for the DTSTransformPhaseEnum constants are as follows: • 0—DTSTransformPhase_None • 1—DTSTransformPhase_PreSourceData 9 THE MULTIPHASE DATA PUMP Creating a Multiphase Data Pump in Code 244 DTS Connections and the Data Transformation. .. multiphase data pump option allows you to extend the capabilities of your transformation tasks It is especially useful for error handling and monitoring the progress of the task The next chapter introduces the new transformation task in SQL Server 2000 the Parallel Data Pump 9 THE MULTIPHASE DATA PUMP The Parallel Data Pump Task IN THIS CHAPTER • Hierarchical Rowsets 249 • When to Use the Parallel Data Pump... the Transform Data task or the Data Driven Query task But instead of having one collection of Transformation objects for a task, the Parallel Data Pump task has multiple collections—one for each separate rowset embedded in the hierarchical rowset Each of these collections of transformations is called a TransformationSet object NOTE The Parallel Data Pump task is a new feature in SQL Server 2000 It’s so... flattened transformation should be used • Whether the Parallel Data Pump task should function like a Transform Data task or a Data Driven Query task These factors are combined in the following ways: • Hierarchical mode—Hierarchical transformation functioning like a Transform Data task • Flattened mode—Flattened transformation functioning like a Transform Data task • Data driven query mode—Flattened transformation. .. creates a transformation set for each of the Transform Data or Data Driven Query tasks Any existing transformation sets in the Parallel Data Pump task are deleted before the new ones are added 5 Disable the Transform Data tasks or the Data Driven Query tasks, and then test the package 6 If you want to modify the transformation sets in the future, you can modify the Transform Data tasks or the Data Driven... each row of data The Parallel Data Pump task used in data driven query mode has more flexibility than the Data Driven Query task because you can create a set of four queries for each of the transformation sets The TransformationSetOptions Property • 0—DTSTranSetOpt_Flattened • 1—DTSTranSetOpt_Hierarchical • 4—DTSTranSetOpt_DataDrivenQueries 10 THE PARALLEL DATA PUMP TASK The mode of a Parallel Data Pump... an option in the Parallel Data Pump task 10 THE PARALLEL DATA PUMP TASK What you gain with the Parallel Data Pump task is the ability to more closely tie together the transformation logic for the transformation of the set of tables 252 DTS Connections and the Data Transformation Tasks PART II The Collections and the Properties of the Parallel Data Pump Task The Parallel Data Pump task has only three... You can use the CreateParallelDataPumpTask application to begin creating a Parallel Data Pump task 10 The five Transform Data tasks are used to create the five transformation sets for the Parallel Data Pump task THE PARALLEL DATA PUMP TASK FIGURE 10.2 260 DTS Connections and the Data Transformation Tasks PART II CAUTION I have tested this strategy for creating a Parallel Data Pump task and verified that... Transform Data tasks or Data Driven Query tasks for each of the rowsets in the hierarchical query, as shown in Figure 10.2 Create and test the transformations for those tasks THE PARALLEL DATA PUMP TASK 1 Use the CreateParallelDataPumpTask application to create a Parallel Data Pump task and its two connections in a new DTS package, as shown in Figure 10.1 10 258 DTS Connections and the Data Transformation. .. DestinationCommandProperties, are also collections of the Transform Data task and the Data Driven Query task The third collection, TransformationSets, is unique to the Parallel Data Pump task The TransformationSet objects in this collection hold the other collections that are found in the Transform Data task and the Data Driven Query task—Transformations, Lookups, DestinationColumnDefinitions, InsertQueryColumns, . and the Data Transformation Tasks P ART II 228 The multiphase data pump option allows you to write code at several different points in the data transformation. for the Transformation2 object in SQL Server 2000. This property contains a bitmap that indicates which phases are enabled for that particular transformation.