Chapter 11 Open Grid Services for Envisat and Earth Observation Applications Luigi Fusco, European Space Agency Roberto Cossu, European Space Agency Christian Retscher, European Space Agency Contents 11.1 Introduction 239 11.2 ESA Satellites, Instruments, and Products 239 11.2.1 ERS-2 240 11.2.2 Envisat 240 11.3 Example of Specialized User Tools for Handling ESA Satellite Data 242 11.3.1 BEST 243 11.3.2 BEAM 243 11.3.3 BEAT 245 11.4 Grid-Based Infrastructures for EO Data Access and Utilization 246 11.4.1 Service Support Environment 249 11.4.2 GeoNetwork 249 11.4.3 CCLRC DataPortal and Scientific Metadata Model 250 11.4.4 Projects@ReSC 250 11.4.5 OPeNDAP 251 11.4.6 DataGrid and Follow-up 251 11.4.7 CrossGrid 252 11.4.8 DEGREE 253 11.5 ESA Grid Infrastructure for Earth Science Applications 254 11.5.1 Infrastructure and Services 254 11.5.2 The GRID-ENGINE 255 11.5.3 The Application Portals 256 11.5.3.1 An Example of an Application Portal: Computation and Validation of Ozone Profile Calculation Using the GOME NNO Algorithm 258 237 © 2008 by Taylor & Francis Group, LLC 238 High-Performance Computing in Remote Sensing 11.6 EO Applications Integrated on G-POD 259 11.6.1 Application Based on MERIS and AATSR Data and BEAM Tools 259 11.6.1.1 MERIS Mosaic as Displayed at EO Summit in Brussels, February 2005 259 11.6.1.2 MERIS Global Vegetation Index 260 11.6.1.3 MERIS Level 3 Algal 1 260 11.6.1.4 Volcano Monitoring by AATSR 261 11.6.2 Application Based on SAR/ASAR Data and BEST Tools 261 11.6.2.1 A Generic Environment for SAR/ASAR Processing 261 11.6.2.2 EnviProj – Antarctica ASAR GM Mapping System 263 11.6.2.3 ASAR Products Handling and Analysis for a Quasi Systematic Flood Monitoring Service 263 11.6.3 Atmospheric Applications Including BEAT Tools 265 11.6.3.1 GOME Processing 265 11.6.3.2 3D-Var Data Assimilation with CHAMP Radio Occultation (RO) Data 265 11.6.3.3 YAGOP: GOMOS Non-operational Processing 266 11.6.3.4 GRIMI-2: MIPAS Prototype Dataset Processing 268 11.6.3.5 SCIA-SODIUM: SCIAMACHY Sodium Retrieval 268 11.7 Grid Integration in an Earth Science Knowledge Infrastructure 270 11.7.1 Earth Science Collaborative Environment Platform and Applications – THE VOICE 271 11.7.2 Earth Science Digital Libraries on Grid 272 11.7.3 Earth Science Data and Knowledge Preservation 273 11.7.4 CASPAR 274 11.7.5 Living Labs (Collaboration@Rural) 275 11.8 Summary and Conclusions 275 11.9 Acknowledgments 277 References 277 The ESA Science and Application Department of Earth Observation Programmes Directorate at ESRIN has focused on the development of a dedicated Earth Science grid infrastructure, under the name Earth Observation Grid Processing On-Demand (G-POD). This environment provides an example of transparent, fast, and easy access to data and computing resources. Using a dedicated Web interface, each application has access to the ESA operational catalogue via the ESA Multi-Mission User Inter- face System (MUIS) and to storage elements. It furthermore communicates with the underlying grid middleware, whichcoordinates all the necessary steps to retrieve, pro- cess, and display the requested products selected from the large database of ESA and third-party missions. This makes G-POD ideal for processing large amounts of data, developing services that require fast production and delivery of results, comparing scientist approaches to data processing, and permitting easy algorithm validation. © 2008 by Taylor & Francis Group, LLC Open Grid Services for Envisat and Earth Observation Applications 239 11.1 Introduction Following the participation of the European Space Research Institute (ESRIN) at ESA in DataGrid, the first large European Commission funded grid project [1], the ESA Science and Application Department of Earth Observation Programmes Directorate has focusedon thedevelopment of a dedicated Earth Science gridinfrastructure, under the name Earth Observation Grid Processing on-Demand [2]. This generic grid-based environment (G-POD) ensures that specific Earth Observation (EO) data handling and processing applications can be seamlessly plugged into the system. Coupled with high performance and sizeable computing resources managed by grid technologies, G-POD provides the necessary flexibility for building a virtual environment that gives applications quick access to data, computing resources, and results. Using a dedicated Web interface, each application has access to a catalogue like the ESA Multi-Mission User Interface System (MUIS) and storage elements. It furthermore communicates with the underlying grid middleware, which coordinates all the necessary steps to retrieve, process, and display the requested products selected from the large database of ESA and third-party missions. Grid On-Demand provides an example of transparent, fast, and easy access to data and computing resources. This makes G-POD an ideal environment for processing large amounts of data, developing services that require fast production and deliv- ery of results, comparing approaches, and fully validating algorithms. Many other grid-based systems are being proposed by various research groups using similar and alternative approaches, although sharing the same ambition for improved integration of the emerging Information and Communication Technologies (ICT) technologies exploitable by the Earth Science community. In the Sections 11.2 and 11.3 we give an overview of selected ESA Earth Ob- servation missions and related software tools that ESA provides for facilitating data handling and analysis. In Section 11.4 we describe how the EO community can ben- efit from grid technology for data access and sharing. In this context, some examples of ESA and EU projects are described. Section 11.5 describes in detail the G-POD environment, its infrastructure, the intermediary layer developed to interface with the application, and the grid computer and storage resources, the Web portals. Differ- ent examples of EO applications integrated in G-POD are described in Section 11.6. Section 11.7briefly documents theuse ofgrid technologyin Earth Science Knowledge Infrastructures. Conclusions are drawn in Section 11.8. 11.2 ESA Satellites, Instruments, and Products This section briefly overviews the ESA European Remote Sensing satellite (ERS) and Envisat missions and the sensors on-board these satellites, with special attention to the data used in the context of ESA’s activities on grids. © 2008 by Taylor & Francis Group, LLC 240 High-Performance Computing in Remote Sensing 11.2.1 ERS-2 The ERS-2 Earth Observation mission [3] has been operating since 1995. The ERS-2 satellite carries a suite of instruments to provide data for scientific and commercial ap- plications. ERS-1, the ERS-2 predecessor, was launched in July 1991 and was ESA’s first sun-synchronous polar-orbiting remote sensing mission, operated until March 2000. It continued to provide excellent data, far exceeding its nominal lifetime. ERS- 2 is nearly identical to ERS-1. The platform is based on the design developed for the French SPOT satellite. Payload electronics are accommodated in a box-shaped hous- ing on the platform; antennas are fitted to a bearing structure. On-board ERS-2 there are seven instruments to support remote sensing applications: RA, ATSR, GOME, MWR, SAR, WS, and PRARE. In particular we wish to refer to: r SAR: Synthetic Aperture Radar (SAR) wave mode provides two-dimensional spectra of ocean surface waves. For this function the SAR records regularly spaced samples within the image swath. The images are transformed into di- rectional spectra providing information about wavelength and the direction of the wave systems. Automatic measurements of dominant wavelengths and di- rections will improve sea forecast models. However, the images can also show the effects of other phenomena, such as internal waves, slicks, small-scale vari- ations in wind, and modulations due to surface currents and the presence of sea ice. r GOME: The GOME instrument, which stands for Global Ozone Monitoring Experiment, is a newly developed passive instrument that monitors the ozone content of the atmosphere to a degree of precision hitherto unobtainable from space. This highly sophisticated spectrometer was developed by ESA in the record time of five years. GOME is a nadir-scanning ultraviolet and visible spectrometer for global monitoring of atmospheric ozone. It was launched on- board ERS-2 in April 1995. Since the summer of 1996, ESA has been delivering to users three-day GOME global observations of total ozone, nitrogen dioxide, and related cloud information, via CD-ROM and the Internet. A key feature of GOME is its ability to detect other chemically active atmospheric trace gases as well as the aerosol distribution. r ATSR: The Along-Track Scanning Radiometer consists of an InfraRed Ra- diometer (IRR) and a Microwave Sounder (MWS). On-board ERS-1, the IRR is a four-channel infrared radiometer used for measuring sea-surface tempera- tures (SST) and cloud-top temperatures, whereas on-board ERS-2 the IRR is equipped with additional visible channels for vegetation monitoring. 11.2.2 Envisat The Environmental Satellite (Envisat) [4] is an advanced polar-orbiting Earth Ob- servation satellite that provides measurements of the atmosphere, ocean, land, and ice. The Envisat satellite has an ambitious and innovative payload that ensures the © 2008 by Taylor & Francis Group, LLC Open Grid Services for Envisat and Earth Observation Applications 241 continuity of the data measurements of the ERS satellites. The Envisat data sup- port Earth Science research and allow monitoring of the evolution of environmental and climatic changes. Furthermore, they facilitate the development of operational and commercial applications. On-board Envisat there are ten instruments: ASAR, MERIS, AATSR, GOMOS,MIPAS, SCIAMACHY,RA-2 (RadarAltimeter 2), MWR (Microwave Radiometer), DORIS (Doppler Orbitography and Radio-positioning), LRR (Laser Retro-Reflector). In particular we wish to refer to: r ASAR: ASAR is the Advanced Synthetic Aperture Radar. Operating at C-band, it ensures continuity with the image mode (SAR) and the wave mode of the ERS-1/2 AMI (Active Microwave Instrument). It features enhanced capability in terms of coverage, range of incidence angles, polarization, and modes of operation. This enhanced capability is provided by significant differences in the instrument design: a full active array antenna equipped with distributed transmit/receive modules that provide distinct transmit and receive beams, a digital waveform generation for pulse ‘chirp’ generation, a block adaptive quantization scheme, and a ScanSAR mode of operation by beam scanning in elevation. r MERIS: MERIS is a programmable, medium-spectral resolution imaging spectrometer operating in the solar reflective spectral range. Fifteen spec- tral bands can be selected by ground command, each of which has a pro- grammable width and a programmable location in the 390 nm to 1040 nm spectral range. The instrument scans the Earth’s surface by the so-called push- broom method. Linear CCD arrays provide spatial sampling in the across- track direction, while the satellite’s motion provides scanning in the along- track direction. MERIS is designed so that it can acquire data over the Earth whenever illumination conditions are suitable. The instrument’s 68.5 ◦ field of view around nadir covers a swath width of 1150 km. This wide field of view is shared between five identical optical modules arranged in a fan-shape configuration. r AATSR: The Advanced Along-Track Scanning Radiometer (AATSR) is one of the Announcement of Opportunity (AO) instruments on-board Envisat. It is the most recent in a series of instruments designed primarily to mea- sure Sea Surface Temperature (SST), following on from ATSR-1 and ATSR- 2 on-board ERS-1 and ERS-2. AATSR data have a resolution of 1 km at nadir and are derived from measurements of reflected and emitted radiation taken at the following wavelengths: 0.55 μm, 0.66 μm, 0.87 μm, 1.6 μm, 3.7 μm, 11 μm, and 12 μm. Special features of the AATSR instrument include its use of a conical scan to give a dual view of the Earth’s sur- face, on-board calibration targets, and use of mechanical coolers to main- tain the thermal environment necessary for optimal operation of the infrared detectors. r GOMOS: The Global Ozone Monitoring by Occultation of Stars instrument is a medium-resolution spectrometer covering the wavelength range from 250 nm © 2008 by Taylor & Francis Group, LLC 242 High-Performance Computing in Remote Sensing to 950 nm. The high sensitivity down to 250 nm required the design of an all- reflective optical system for the UVVIS part of the spectrum and the functional pupil separation between the UVVIS and the NIR spectral regions. Due to the requirement of operating on very dim stars (magnitudes ≤ 5), the sensitivity requirement for the instrument is very high. Consequently, a large telescope with 30 cm ×20 cm aperture had to be used in order to collect sufficient signals. Detectors with high quantum efficiency and very low noise had to be developed to achieve the required signal to noise ratios (SNR). r MIPAS: The Michelson Interferometer for Passive Atmospheric Sounding is a Fourier transform spectrometer for the detection of limb emission spectra in the middle and upper atmosphere. It observes a wide spectral interval through- out the mid infrared with high spectral resolution. Operating in a wavelength range from 4.15 μm to 14.6 μm, MIPAS detects and spectrally resolves a large number of emission features of atmospheric trace gas constituents playing a major role in atmospheric chemistry. Due to its spectral resolution capabili- ties and low-noise performance, the detected features can be spectroscopically identified and used as input to suitable algorithms for extracting atmospheric concentration profiles of a number of target species. r SCIAMACHY: The Scanning Imaging Absorption Spectrometer for Atmo- spheric Cartography instrument is an imaging spectrometer whose primary mission objective is to perform global measurements of trace gases in the tro- posphere and in the stratosphere.The solar radiation transmitted, backscattered, and reflected from the atmosphere is recorded at high resolution (0.2 μmto 0.5 μm) over the range 240 nm to 1700 nm, and in selected regions between 2.0 μm and 2.4 μm. The high resolution and the wide wavelength range make it possible to detect many different trace gases despite low concentrations. The large wavelength range is also ideally suited for the detection of clouds and aerosols. SCIAMACHY has three different viewing geometries: nadir, limb, and sun/moon occultations, which yield total column values as well as distri- bution profiles in the stratosphere and even the troposphere for trace gases and aerosols. 11.3 Example of Specialized User Tools for Handling ESA Satellite Data To facilitate users in accessing ERS and Envisat instrument’s data products, ESA has developed a set of software utilities with the contribution and validation of key instrument scientists. All these tools can be downloaded for free at [5]. Among these tools,some ofthemhave beenintegratedin theESAgrid environment, and for this reason we briefly describe them in the following. Greater details can be obtained from the aforementioned Website. © 2008 by Taylor & Francis Group, LLC Open Grid Services for Envisat and Earth Observation Applications 243 Figure 11.1 The BEST Toolbox. 11.3.1 BEST The Basic Envisat SAR Toolbox (BEST) is a collection of executable software tools that has been designed to facilitate the use of ESA SAR data. The purpose of the Toolbox is not to duplicate existing commercial packages, but to complement them with functions dedicated to the handling of SAR products obtained from ASAR and AMI on-board Envisat, ERS-1, and ERS-2, respectively. BEST has evolved from the ERS SAR Toolbox (see Figure 11.1). The Toolbox operates according to user-generated parameter files. The interface does not include a display function. However, it includes a facility to convert images to TIFF or GeoTIFF format so that they can be read by many commonly available visualization tools. Data may also be exported in the BIL format for ingestion into other image processing software. The tools are designed to achieve the following functions: data import and quick look, data export, data conversion, statistical analysis, resampling, co-registration, basic support for interferometry, speckle filtering, and calibration. 11.3.2 BEAM The Basic ERS & Envisat (A)ATSR and MERIS Toolbox is a collection of executable tools and APIs (Application Programming Interfaces) that have been developed to fa- cilitate theutilization, viewing, and processing ofERS andEnvisat MERIS,(A)ATSR, and (A)SAR data. The purpose of BEAM is to complement existing commercial pack- ages with functions dedicated to the handling of MERIS and AATSR products. The main components of BEAM are: r A visualization, analyzing, and processing software (VISAT). r A set of scientific data processors running either from the command line or invoked by VISAT. © 2008 by Taylor & Francis Group, LLC 244 High-Performance Computing in Remote Sensing Figure 11.2 The BEAM toolbox with VISAT visualization. r A data product converter tool allowing a user to convert raw data products to RGB images, HDF-5, or the BEAM-DIMAP standard format. r A Java API that provides ready-to-use components for remote sensing related application development and plug-in points for new BEAM extensions. r MERIS/(A)ATSR/(A)SAR product reader API for ANSI C and IDL, allowing read access to these data products using a simple programming model. VISAT (see Figure 11.2) and the scientific data processors use a simple data input/ output format, which makes it easy to import ERS and Envisat data in other imaging applications. The format is called DIMAP and has been developed by SPOT-Image in France. The BEAM software uses a special DIMAP profile called BEAM-DIMAP, which has the following characteristics: r A single product header (XML) containing the product metadata. r An associated data directory containing ENVI-compatible images for each band. Each image in the directory is composed of a header file (ASCII text) and an image data file (flat binary) source code. The complete BEAM software has been developed under the GNU public license and comes with full source code (Java and ANSI C). All main components of the toolbox are programmed in pure Java for maximum portability. The product reader API for C has been developed exclusively with the ANSI-compatible subset of the C programming language. The BEAM software has been successfully tested under MS Windows 9X, NT4, 2000, and XP, as well as under Linux and Solaris operating systems. BEAM is intended to also run on other Java-enabled UNIX derivates, e.g., Mac OS X. © 2008 by Taylor & Francis Group, LLC Open Grid Services for Envisat and Earth Observation Applications 245 11.3.3 BEAT The Basic ERS and Envisat Atmospheric Toolbox aims to provide scientists with tools for ingesting, processing, and analyzing atmospheric remote sensing data. The project consists of several software packages, with the main packages being BEAT and VISAN. The BEAT package contains a set of libraries, command line tools, and interfaces to IDL, MATLAB, FORTRAN, and Python for accessing data from a range of atmospheric instrument product files. The VISAN package contains an application that can be used to visualize and analyze data retrieved using the BEAT interface. The primary instruments supported by BEAT are GOMOS, MIPAS, SCIAMACHY (Envisat), GOME(ERS-2), OMI,TES, andMLS (Aura), as well as GOME-2 and IASI (MetOp). BEAT, VISAN,and an MIPASprocessor called GeoFitare provided as Open Source Software, enabling the user community to participate in further development and quality improvements. The core part of the toolbox is the BEAT package itself. This package provides data ingestion functionalities for each of the supported instruments. The data access functionality is provided via two different layers, called BEAT-I and BEAT-II: r BEAT-I: The first layer of BEAT provides direct access to data inside each file that is supported by BEAT. The supported instruments include GOMOS, MIPAS, SCIAMACHY, GOME, OMI, TES, and MLS. All product data files are accessible via the BEAT-I C library. On top of this C library there are several interfaces available to directly ingest product data using, e.g., FORTRAN, IDL, MATLAB, and Python. Furthermore, BEAT also comes with a set of command line tools (beatcheck, beatdump, and beatfind). r BEAT-II: The second layer of BEAT provides an abstraction to the product data to make it easier for the user to get the most important information ex- tracted. Using only a single command you will be able to ingest product data into a set of flexible data types. These predefined data types make it easier to compare similar data coming from different instruments and also simplify the creation of general visualization routines. Furthermore, the BEAT-II layer provides some additional functions to manipulate and import/export these spe- cial data types. The layer 2 interface is built on top of the BEAT-I C library, but BEAT-II also supports reading of additional products that are stored in, e.g., ASCII, HDF4, or HDF5 format. As for BEAT-I, all BEAT-II function- ality is accessible via the BEAT-II C. Moreover, BEAT contains interfaces of BEAT-II for FORTRAN, IDL, MATLAB, and Python, and a command line tool. r VISAN: VISAN (see Figure 11.3) is a cross-platform visualization and anal- ysis application for atmospheric data, where the user can pass commands in Python language. VISANprovides powerfulvisualization functionality fortwo- dimensional plotsand worldplots.The Python interfaces for BEAT-I andBEAT- II are included so one can directly ingest product data from within VISAN. By using the Python language and some additional included mathematical packages it is possible to perform an analysis on selected data. © 2008 by Taylor & Francis Group, LLC 246 High-Performance Computing in Remote Sensing Figure 11.3 The BEAT toolbox with VISAN visualization. r GeoFit: BEAT also contains the GeoFit software package, which is used to process MIPAS special mode measurements. 11.4 Grid-Based Infrastructures for EO Data Access and Utilization While conducting their research, Earth scientists are often hindered by difficulties lo- cating andaccessing theright data, products, and other information needed to turndata into knowledge, e.g., interpretation of the available data. Data provision services are far from optimal forreasons related both to scienceand infrastructurecapabilities. The process of identifying and accessing data typically takes up the most time and money. Of the different base causes of this, those most frequently reencountered relate to: r The physical discontinuity of data. Data are often dispersed over different data centers and local archives distributed all over Europe and abroad and, inher- ent to this, the different policies applied (e.g., access and costs), the variety of interoperability, confidentiality, and search protocols as well as the diversity of data storage formats. To access a multitude of data storage systems, users need to know how and where to find them and need a good technical/system background to interface with the individual systems. Furthermore, often only the metadata catalogues can be accessed online, while the data themselves have to be retrieved offline. © 2008 by Taylor & Francis Group, LLC [...]... read files stored on 8-inch floppy disks that were popular just 25 years ago Vast amounts of digital information from just 25 years ago are lost for all practical purposes [8] © 2008 by Taylor & Francis Group, LLC 248 High- Performance Computing in Remote Sensing r The many different actors involved Science is becoming increasingly international and interdisciplinary, resulting in an increased total number... high resolution SAR observations provides useful input to crisis and damage mapping This is particularly relevant for flood monitoring, and SAR is considered a useful information source for river plain flooding events, a © 2008 by Taylor & Francis Group, LLC 264 High- Performance Computing in Remote Sensing Figure 11. 9 ASAR mosaic obtained using G-POD considering GM products acquired from March 8 to 14, 2006... infrastructure) of the DataGrid project have been included in the follow-up EU grid project called Enabling Grids for E-sciencE (EGEE) [18], already introduced in Chapter 10 of this book EGEE, funded by the EC Framework Programme (FP), aims to develop a European-wide service grid © 2008 by Taylor & Francis Group, LLC 252 High- Performance Computing in Remote Sensing infrastructure available to scientists 24... documented in a standard and consistent way This has enhanced the data exchange and sharing between the organizations, avoiding © 2008 by Taylor & Francis Group, LLC 250 High- Performance Computing in Remote Sensing duplication, and has increased the cooperation and coordination of efforts in collecting data The data are made available to benefit everyone, saving resources and at the same time preserving data... vegetation, chlorophyll, or water vapor from independent measures stored in different files) By dividing the spatial domain (e.g., continents or latitude/longitude boxes), a straightforward division of the corresponding process is achieved © 2008 by Taylor & Francis Group, LLC 256 High- Performance Computing in Remote Sensing The applications are then submitted to the computing elements and their state is automatically... LLC 260 High- Performance Computing in Remote Sensing Figure 11. 6 MERIS mosaic at 1.3 km resolution obtained in G-POD from the entire May to December 2004 data set automatically updated and registered each day from the ground segment, can be selected over user-defined areas and temporal coverage for producing public-relations material The final image is a mosaic made up of true color images using four... achieved: r r r Disseminate, promote uptake of grid in a wider ES community, and integrate newcomers Reduce the gap between ES users and grid technology Explain and convince ES users of grid benefits and capability to tackle new and complex problems © 2008 by Taylor & Francis Group, LLC 254 11. 5 High- Performance Computing in Remote Sensing ESA Grid Infrastructure for Earth Science Applications In previous sections... Mission (SRTM) Digital Elevation Model (DEM) v3 [31], and mosaicking These capabilities are obtained by using different toolboxes such as BEST and inhouse developed software © 2008 by Taylor & Francis Group, LLC 262 High- Performance Computing in Remote Sensing Figure 11. 7 The ASAR G-POD environment The user browses for and selects products of interest (upper left panel) The system automatically identifies... Science KI The main contribution is in integrating a common infrastructure In this context, ESA is active in demonstrating extensive use of applications in stable grid environments In the summer of 2006, ESA opened an ‘Announcement of Opportunity’ to provide online access to large ESA Earth Observation archives, computing and storage elements, and user tools for handling data Finally, the e-collaboration... (YAGOP) is running on Grid on-Demand [42] This processor not only enables the derivation of temperature profiles; the calculation of non-operational GOMOS Level 2 ozone products was also implemented, which is a processing option based on an optimal estimation technique, inverting ozone and NO2 profiles simultaneously © 2008 by Taylor & Francis Group, LLC 268 High- Performance Computing in Remote Sensing The . avoiding © 2008 by Taylor & Francis Group, LLC 250 High- Performance Computing in Remote Sensing duplication, and has increased the cooperation and coordination of efforts in collect- ing data Francis Group, LLC 238 High- Performance Computing in Remote Sensing 11. 6 EO Applications Integrated on G-POD 259 11. 6.1 Application Based on MERIS and AATSR Data and BEAM Tools 259 11. 6.1.1 MERIS Mosaic. data processors running either from the command line or invoked by VISAT. © 2008 by Taylor & Francis Group, LLC 244 High- Performance Computing in Remote Sensing Figure 11. 2 The BEAM toolbox