288 Hands-On Microsoft SQL Server 2008 Integration Services assign a writer role to user accounts or groups. In the last part, you assigned the Reader role directly to the user-defined role and then linked that role to a fixed database-level role. Alternatively, you could have added users directly to the fixed database-level role, though that would not be a recommended approach. Whichever way you choose to go, make sure you assign a reader or writer role to the user or user-defined role and link them to the fixed database-level roles. Another point worth mentioning here is that any metadata or configuration files that are saved outside the package cannot be protected solely by the database-level roles. Carefully consider where you want to store your configuration files or other miscellaneous files so that they are properly protected. Examples of such files that can be saved outside the package are configuration files, checkpoint files, cache files, or log files. If you are saving these files to SQL Server (but outside the package) or to the file system, you need to secure them with appropriate security measures. For more details on this, refer to the next section. Last but not least, you also need to assign permissions to user-defined roles on the objects (such as tables, views, and so on) being accessed by your package; otherwise, the package will run but will fail for not being able to access the objects. Considerations for Different Storage Areas You need to protect the Integration Services packages and the package metadata from unauthorized access or hardware failures. Integration Services packages may have associated configuration files that have been saved outside the package. Figure 7-7 Assigning a Reader role to the Team role Chapter 7: Securing Integration Services Packages 289 These configuration files are used to update values of properties at run time, making it easier to deploy packages from a development environment to a production environment. While both the package and the package configurations can be saved in the SQL Server or file system, they need to be protected irrespective to where they have been saved. Considerations for Saving to SQL Server When you save SSIS packages to SQL Server, you actually save them to the sysssispackages table in the MSDB database. If you are using a legacy package—i.e., the DTS 2000 package—Integration Services saves it to the sysdtspackages table in the MSDB database. This means when you save your packages in the MSDB database, they are protected by server, database, and table-level permissions. Packages developed from the ground up using BIDS (and not using the DTS 2000 package)—i.e., the packages saved in the sysssispackages table—are also protected by fixed database-level roles as you have studied earlier. Also, you can associate a reader role or a writer role to a package and can link these roles to user-defined roles. You need to consider the protection of miscellaneous files attached to a package, e.g., package configurations that can also be saved to any SQL Server database table, not necessarily the MSDB database. When saved in an SQL Server database table, package configurations can be protected with server-, database-, and table-level permissions. To protect your packages from hardware failures or miscellaneous types of data losses, you need to develop a backup plan that includes backup of the MSDB database, as this will back up sysssispackages and sysdtspackages tables stored in the MSDB database. For package configurations, you need to make sure that the table containing package configurations is also included in the backup plans to provide complete protection from hardware failures. On the operational side, you know that the packages used and saved in BIDS, by default, go to the file system. When you want to work on a package in BIDS that has been saved to SQL Server, you’ll need to import package into an Integration Services project that will create a local copy on the file system before you can edit it. After editing, when you want to save the package back to SQL Server, you will have to use the Save Copy Of packageName.dtsx As option to save it. You can argue that if you save your packages to the file system, you wouldn’t really need to have SQL Server database engine installed on the server. Your storage decision may be affected by the fact that most of the SSIS packages end up being scheduled using the SQL Server Agent, which in turn requires the SQL Server database engine. While thinking of having only Integration Services on a machine and avoiding SQL Server engine installation, you may think to use the SQL Server Express Edition as a database engine. However, it is worth mentioning that the SQL Server Express Edition doesn’t support the SQL Server Agent. 290 Hands-On Microsoft SQL Server 2008 Integration Services Considerations for Saving to the File System When saved to the file system, an Integration Services package is stored as an XML file (.dtsx), and package configurations can also be saved to XML file types. If you decide to save your packages and package configurations to the file system, you need to secure them using file system permissions. It’s better to create a folder hierarchy for a solution to keep all the projects, their packages, configuration files, and other miscellaneous files if possible and protect this folder hierarchy using file system permissions. This also helps in making it easy to back up all the code and the files for related projects in the same backup job. One consideration must be given while deciding the storage that how and where the packages will be deployed to production finally. As a developer, you can’t decide about the production storage, as this is generally in the control of DBAs. It’s not that you can’t deploy to an alternate storage in production than what you’ve got in the development environment, but it’s about making things easier for yourself, as you may find testing certain package configurations and access permissions difficult if the environments differ. Summary Security is the most often discussed topic these days in the IT industry. As a DBA or information analyst, you need to make sure that the contents of SSIS packages are secure. You have seen that the Integration Services provides a facility to sign a package to check for its integrity and provides six package protection levels, out of which four are for encrypting packages, one is for excluding sensitive information, and one is for using database-level roles. You have learned how to exclude sensitive information from your packages and encrypt sensitive information in packages using a user key or password. You have also used database-level roles to protect the package with read or write access controls. You should understand the various issues involved in deciding the storage location of packages. Remember to consider carefully where to store each file used in an SSIS solution and make sure these files are also protected. Advanced Features of Integration Services Chapter 8 In This Chapter c Logging and Log Providers in SSIS c Transactions in Integration Services Packages c Restarting Packages with Checkpoints c Expressions and Variables c Handling Events at Package Run Time c As a Data Source for Reporting Services Report c Summary 292 Hands-On Microsoft SQL Server 2008 Integration Services B y now, you’ve worked with most of the control flow tasks and have used their features and properties to create a work flow within your package to perform a job. In this chapter you will learn about some of the advanced functionalities built into Integration Services that you can leverage to extend functionality of your packages. First, you will learn about logging and log providers in Integration Services, which help you to log the run-time events and provide auditing abilities. Then you will work with transactions and will learn how they can be used to maintain data consistency in the data stores. You will also do a Hands-On exercise on transactions and will learn about the three main scenarios in which you can deploy transactions. Use of checkpoints in packages provides the facility to restart a package from the point of failure and reduce the work that needs to be done to complete the regular data processing jobs. This is where you will do your next exercise on checkpoints and their usability and will restart a failed package. Though you’ve already been introduced to variables and property expressions and have used them to a basic level, you will be using them extensively in this chapter. Then you will use event handlers to learn how you can make your packages respond to the events occurring during run time and take appropriate actions. SSIS is quite extensible and can interface with several applications; for instance, it can act as a data source for Reporting Services reports or for an ASP.NET application. SSIS can also work with source control applications such as Visual SourceSafe or Team Foundation Server to allow you to adopt standard software development lifecycle practices. We do not cover these topics here, as these are features of the main software products rather than of SSIS; however, this chapter does cover how SSIS acts as a data source for the Reporting Services Reports feature to demonstrate that SSIS can be configured as an interface to external applications quite easily. Let’s start with the easiest and the most basic requirement of logging events and activities at run time. Logging and Log Providers in SSIS You can use logging in Integration Services to log information about events happening at run time for auditing and troubleshooting purposes. You can add the logs at the package level for which you want to log information. Integration Services provides detailed levels of logging within packages with five preconfigured log providers; plus, you can create your own custom log providers. The log providers specify the format and the storage type of the log. You can log information to the text files, XML files, SQL Server Profiler, or Windows event log; or else to the sysssislog system table in an SQL Server database. Other than the Windows Chapter 8: Advanced Features of Integration Services 293 event log provider, which writes log entries to the Application log of the Windows Event log on the local computer, you will need to connect to the log stores to write log entries into them. Depending on the type of log provider you choose, you may need to add a connection manager to the package. To add a log to a package, first select the log provider and then, if applicable, specify a connection manager to specify the location of the log. SSIS containers expose their properties as log entries to enable you to log information from them. From an architecture standpoint, an Integration Services package is a container that can enclose other containers such as the Sequence container, the For Loop container, and the Foreach Loop container. These containers can further enclose tasks. And a task is enclosed inside a container called the task host container enabling the task to exhibit the container behavior. This allows you to capture log information at the package level, at the container level, or even at the task level. Once you have enabled logging at the package level, you can then configure logging at the package, container, and task levels. Let’s explore the types of log providers available in Integration Services: SSIS Log Provider for SQL Server When you want to write the logging information into an SQL Server database so that you can query for certain events, you will select this log provider. It writes the log entries into the sysssislog system table that it creates when the package is run the first time with this log provider selected and configured. An OLE DB Connection Manager is used to specify the database to which you want it to create a sysssislog system table. SSIS Log Provider for SQL Server Profiler In SQL Server 2008, you have the ability to link Windows performance logs with the Profiler traces so that you can match the events happening inside SQL Server against the overall performance behavior of the server. For such scenarios, when you want to analyze the execution of the package step-by-step against the performance logs, you will select this log provider to log information in a format that can be read by the SQL Server Profiler. The log file name extension must be .trc for it to be used with the SQL Server Profiler. This provider uses a file connection manager to write log entries to a file with .trc extension. SSIS Log Provider for Text files This log provider adds the simplest and most used type of logs in the form of text files that can be transferred anywhere and can be read by several other applications. This log provider writes the logging information into a CSV (comma-separated value) formatted text file that you specify using a file connection manager. 294 Hands-On Microsoft SQL Server 2008 Integration Services SSIS Log Provider for Windows Event Log When you use this log provider, your package will log entries into Windows Application Event Log on the local computer. This is the simplest form of log provider to work with, as you do not even need to configure a connection manager. SSIS Log Provider for XML Files When you want to write the logging information in XML format, you will use this log provider. You will then use a file connection manager to specify a file with an .xml extension into which it writes the log entries. A few words about log providers in general: You can add more than one log of each type of log providers in your package. For example, you can add the SSIS log provider for text files twice in your package to contain different set of information in the different files. Once you’ve added the logs and have specified their connection managers, you can then enable logging for the tasks and containers in your package and select the events to log to one or more types of logs that you’ve added into your package. By this time, you can well imagine that you can write different events to different logs with the ability to write events happening within a container to multiple logs. Also, with different levels of containers and tasks within the package, you can select relevant events to be logged based on the role or level of the container within the package. For example, you may want to log package start time and end time at the package level but may want to log much more information at the task level. By default, Integration Services provides several types of information for logging against each event or log entry. This information, called the Integration Services log schema, consists of the following elements: Event C Name of the event or log entry. Computer C Name of the computer where the event occurred. Operator C Name of the user under whose context the package is being executed. SourceName C Name of the container or task where the event occurred. SourceID C Each task or container in the Integration Services package is assigned a unique ID, which is displayed in the ID field in the Properties window. is field refers to that unique ID of the task or container where the event occurred. ExecutionID C Each time a package is executed, it is assigned a globally unique execution ID that is recorded in the logs by this element. MessageText C Some events generate messages such as “Beginning of package execution” during package execution, which are logged in this element. DataBytes C Specifies a byte array (BLOB) that is specific to the log entry. You will find some additional elements available in the logs depending on the type of log, such as ID, StartTime, EndTime, and DataCode. Other features are associated Chapter 8: Advanced Features of Integration Services 295 with logging functionality, such as inheritance of logging options and a facility to apply templates of logging configurations to keep consistency across the organization. These features are best understood visually while working with them, so let’s do a Hands-On exercise to configure logging in a package. Hands-On: Configuring Logging in a Package The objective of this Hands-On exercise is to enable and configure logging in a package and study the log results after executing the package. Method To implement logging, we will be using the Contacting Opportunities package that you created in Chapter 4. However, to avoid version conflict and confusion among these two packages, we will create a new Integration Services project and add the Mailing Opportunities.dtsx package in to it. Exercise (Create a New Integration Services Project) Here you will create a new Integration Services project and add the Mailing Opportunities.dtsx package in to it. 1. Create an Integration Services project in BIDS with the following details: Name Contacting Opportunities with Logging Project C:\SSIS\Projects 2. After the project has been created, go to the Solution Explorer window and delete the Package.dtsx package. 3. Right-click the SSIS Packages node and select Add Existing Package. In the Add Copy Of Existing Package dialog box, choose SQL Server in the Package Location field if it is not already selected. In the Server field, type localhost and leave Windows Authentication selected in the Authentication Type field. Click the ellipsis button next to the Package Path field to open the SSIS Package window, which should display the Mailing Opportunities package under SSIS Packages folder, which you imported to MSDB in Chapter 6. However, if you skipped Chapter 6 and this package is not there, add this package from the file system instead. Select Mailing Opportunities from the list and click OK. Close the Add Copy Of Existing Package dialog box by clicking OK. You will see the Mailing Opportunities.dtsx package added under SSIS Packages node in the Solution Explorer. 4. Double-click the Mailing Opportunities.dtsx package in the Solution Explorer to load it in the Designer. 296 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Enable and Configure Logging) This is the part in which you will enable logging and configure log entries at various levels in the package. 5. Right-click anywhere on the blank surface in the Control Flow tab and select Logging from the context menu. In the Containers pane of the Configure SSIS Logs dialog box, click the check box to the left of Mailing Opportunities to enable logging at the package level. 6. On the right pane of the Providers And Logs tab, click in the Provider Type field and select “SSIS log provider for SQL Server” from the list. Click Add to add this log provider. Similarly, add another log of SSIS log provider for Text Files type at the package level. You have added two different types of log providers to your package, as you will be logging the package level information to SQL Server and the Send Mail task level logging information to the text file. 7. Click in the Name field of the SSIS log provider for SQL Server log and rename it Package level log to SQL Server. Similarly rename the SSIS log provider for Text files log name as SMTP task level log to Text file. 8. Click in the Configuration field of Package level log to SQL Server and select localhost.Campaign Connection Manager. 9. Click in the Configuration field of SMTP task level log to Text file and select <New Connection . . .>. This will open a File Connection Manager Editor. Click in the Usage Type field and select Create File from the drop-down list. Type C:\ SSIS\Projects\Contacting Opportunities with Logging\ SMTPTaskLevel.log in the File field and click OK. At this stage, your Configure SSIS Logs dialog box should look like the one shown in Figure 8-1. 10. Now that you have enabled logging for your package, you can select the log entries against which you want to log information. As you will be logging package-level information to the SQL Server table, select the check box on left of the Package level log to SQL Server log. 11. Go to the Details tab and select the check box provided to the left of Events column to select all the events listed below it, as shown in Figure 8-2. You can select the desired events from this list provided for the container or the task. Some objects in SSIS also display custom log entries here as well that are used to log special information related to that particular object. Spend some time here reading the description provided for each event. 12. After enabling logging and the log entries at package level, you can move on to configure subcontainers and tasks. Click Iterating October Opportunities. The Details tab will gray out and an alert will ask you to enable logging if you want to implement unique logging options for this container. Also note that the check in the check box on the left of this container is grayed out. This means that the logging options will be inherited from the immediate parent container—i.e., Chapter 8: Advanced Features of Integration Services 297 you do not need to define logging options specifically for child containers or tasks. This makes logging easy and quick to implement. However, if you want to control the logging level for each container or task, the option is available. Each container or task in the package can have three levels of logging modes that are defined by a property called LoggingMode: UseParentSetting, Enabled, or Disabled. You can set the logging mode on an object either from the Properties window of the selected object or by using the check boxes provided for the object in the Configure SSIS Logs dialog box. If you click the Iterating October Opportunities Container check box to clear the check mark, this disables the LoggingMode, and if you click again to apply a check mark in the check box, it will enable the LoggingMode property, which means that you can define unique logging options for this container. As we do not want to set separate logging options for this container, click again to gray out the check box to imply that the parent logging options will be used. Figure 8-1 Enabling logging in an Integration Services package . However, it is worth mentioning that the SQL Server Express Edition doesn’t support the SQL Server Agent. 290 Hands-On Microsoft SQL Server 2008 Integration Services Considerations for Saving to. requires the SQL Server database engine. While thinking of having only Integration Services on a machine and avoiding SQL Server engine installation, you may think to use the SQL Server Express. 288 Hands-On Microsoft SQL Server 2008 Integration Services assign a writer role to user accounts or groups. In the last part, you assigned the Reader role directly