Image Databases: Search and Retrieval of Digital Imagery Edited by Vittorio Castelli, Lawrence D. Bergman Copyright  2002 John Wiley & Sons, Inc. ISBNs: 0-471-32116-8 (Hardback); 0-471-22463-4 (Electronic) 4 Medical Imagery STEPHEN WONG and KENT SOO HOO, JR. University of California at San Francisco, San Francisco, California 4.1 INTRODUCTION Medical imaging has its roots in the accidental discovery of a new class of electromagnetic radiation, X rays, by Wilhelm Conrad Roentgen in 1895. The first X-ray radiograph ever taken was of his wife’s hand, revealing a picture of the living skeleton [1]. In the subsequent decades, physicians refined the art of X-ray radiography to image the structural and physiological state of internal organs such as the stomach, intestines, lungs, heart, and brain. Unlike the gradual evolution of X-ray radiography, the convergence of imaging physics and computers has spawned a revolution in medical imaging practice over the past two decades. This revolution has produced a multitude of new digital imaging modalities: film scanners, diagnostic ultrasound, computed tomography (CT), magnetic resonance imaging (MRI), digital subtraction angiography (DSA), single-photon-emission computed tomography (SPECT), positron-emission tomography (PET), and magnetic source imaging (MSI) to name just a few [2,3]. Most of these modalities are routinely being used in clinical applications, and they allow in vivo evaluation of physiology and anatomy in ways that conventional X-ray radiography could never achieve. Digital imaging has revolutionized the means to acquire patient images, provides flexible means to view anatomic cross sections and physiological states, and frequently reduces patient radiation dose and examination trauma. The other 70 percent of radiological examinations are done using conventional X rays and digital luminescent radiography. These analog images can be converted into digital format for processing by using film digitizers, such as laser scanners, solid-state cameras, drum scanners, and video cameras. Medical images are digitally represented in a multitude of formats depending on the modality, anatomy, and scanning technique. The most outstanding feature of medical images is that they are almost always displayed in gray scale rather than color, with the exception of Doppler ultrasound and pseudocolor nuclear medicine images. A two-dimensional (2D) medical image has a size of 83 84 MEDICAL IMAGERY Table 4.1. Dimensions and sizes of biomedical images Modality Image Gray Level Average Dimension (Bits) Size/Exam. Nuclear medicine 128 × 128 8 or 16 2 MB MRI 256 × 256 12 8–20 MB Ultrasound 512 × 512 8 5–8 MB Doppler ultrasound 512 × 512 24 15–24 MB DSA 512 × 512 8 4–10 MB CT 512 × 512 12 20 MB Spiral or helical CT 512 × 512 12 40–150 MB Digital electronic microscopy (DEM) 512 × 512 8 varies Digital color microscopy (DCM) 512 × 512 24 varies Cardiac catheterization 512 × 512 or 8 500–1000 MB 1024 × 1024 Digitized X-ray films 2048 × 2048 12 8 MB Computed radiography 2048 × 2048 12 8–32 MB Digitized mammogram 4096 × 4096 12 64 MB (a pair) M × N × k bits, where M is the height in pixels and N is the width, and where there are 2 k gray levels. Table 4.1 lists the average number of megabytes (MB) per examination generated by medical imaging technologies, where a 12-bit image is represented by 2 bytes in memory. The size of an image and the number of images taken in one patient examination varies with the modality. As shown in Table 4.1, except for digital electronic microscopy (DEM) and digital color microscopy (DCM), which are pathological and histological images of microscopic tissues, all the modalities are classified as radiological images (that broadly include images for use in other medical disciplines such as cardiology and neurology) and used for diagnosis, treatment, and surgery-planning purposes. Each radiological examination follows a well-defined procedure. One examination (about 40 image slices) of X-ray CT with uniform image slice size of 512 × 512 × 12 bits is around 20 MB, whereas one digital mammography image usually generates 32 MB of data. Digital imaging modalities produce huge amounts of image data that require the creation of new systems for visualization, manipulation, archiving, and transmission. The traditional method of handling images using paper and films cannot possibly satisfy the needs of the modern, digitally enabled radiological practice. Picture archiving and communication systems (PACS) have been developed in the past decade to handle the large volume of digital image data generated in radiology departments, and proponents envision an all- digital, filmless radiology department in the near future. Today’s managed care environment further demands the reduction of medical costs, and computer systems can help to streamline the process of handling all patient data, including images. Telemedicine enables physicians to consult with regional expert centers APPLICATIONS 85 using wide area networks and telephones, improving the quality of care and also eliminating the cost of maintaining such expertise on-site at smaller clinics or rural hospitals. In addition, there is great interest in integrating all the health care information systems into one computerized patient record (CPR) in order to reduce costs and to provide full access to longitudinal patient data and history for care providers. 4.2 APPLICATIONS The most prevalent clinical application of medical image database systems (MIDS) is acquiring, storing, and displaying digital images so that radiologists can perform primary diagnosis. These systems are responsible for managing images from the acquisition modalities to the display workstations. Advanced communication systems are enabling doctors to exchange voice, image, and textual data in real time, over long distances in the application known as teleconsultation. Finally, researchers are utilizing MIDS in constructing brain atlases for discovering how the brain is organized and how it functions. 4.2.1 Display Workstations Clinicians interact with MIDS through display workstations. Clinicians interpret images and relevant data using these workstations, and the results of their analysis become the diagnostic report, which is permanently archived in hospital and radiology information systems (HIS and RIS). Generally, the clinician enters the patient name or hospital identification into the display station’s query field to survey which image studies are available. The clinician selects only those images that need to be transferred from the central storage archive to the display workstation for the task at hand. The six basic types of display workstations support six separate clinical appli- cations: diagnosis, review, analysis, digitization and printing, interactive teaching, and desktop applications. Radiologists make primary diagnoses using diagnostic workstations. These workstations are constructed using the best hardware avail- able and may include multiple high-resolution monitors (having a significantly higher dynamic range than typical displays and a matrix of 2,000 × 2,000 or 2,500 × 2,000 pixels) for displaying projection radiographs. Redundant arrays of inexpensive disks (RAID) are used for local storage to enable rapid retrieval of images with response time on the order of 1 to 2 seconds. In addition to the primary diagnosis, radiologists and referring physicians often review cases in the hospital wards using a review workstation. Review workstations may not require high-resolution monitors, because the clinician is not generating a primary diag- nosis and the referring physicians will not be looking for every minute detail. Analysis workstations differ from diagnostic and review workstations in that they are used to extract useful parameters from images. An example of a useful parameter might be the volume of a brain tumor: a clinician would then perform a region of interest (ROI) analysis by outlining the tumor on the images, and the 86 MEDICAL IMAGERY workstation would calculate its volume. Clinicians obtain hard copies (printouts) of digital medical images at digitizing and printing workstations, which consist of a paper printer for pictorial report generation. When a patient is examined at other hospitals, the workstation’s laser film scanner allows the radiology depart- ment technician to digitize hard copy films from outside the department and store the digitized copy into the local image archival system. An interactive teaching workstation is used to train radiologists in the art of interpreting medical images; a software program leads the student through a series of images and multiple- choice questions that are intended to teach him/her how to recognize various pathologies. Finally, physicians or researchers need to generate lecture slides for teaching and research materials from images and related data in the MIDS. The desktop workstation uses everyday computer equipment to satisfy requirements that are outside the scope of daily clinical operations. As examples, a pair of multimedia physician workstation prototypes developed at University of California, San Francisco (UCSF) is described using an object- oriented multimedia graphical user interface (GUI) builder. Age assessment of pediatric bone images and Presurgical planning in epilepsy are the two supported applications. In the first application, a pediatrician assesses bone age and compares it with the chronological age of the patient based on a radiological examination of the skeletal development of a left-hand wrist. A discrepancy indicates abnormalities in skeletal development. Query of the database for pediatric hand bone images can be by image content, for example, by radius bone age or ratio of epiphyseal and metaphyseal diameters; by patient attributes, for example, by name, age, and exam date; or by a combination of these features. Programs for extracting features of hand bone images were discussed in Refs. [4,5]. The sliders in the “Query-by-Image Attributes” window can be used to specify the range of the image attributes for data retrieval. The Image Database System (IDBS) returns with a list of five patients and representative thumbnail images satisfying the combined image- and patient-attribute constraints. The user can click on any thumbnail image to retrieve, visualize, and analyze the original digitized hand radiographs (Fig. 4.1). The second application, assisting the presurgical evaluation of complex partial seizure is illustrated in Figure 4.2. Here, the user specifies the structural, func- tional, and textual attributes of the MRI studies of interest. The IDBS returns a list of patients satisfying the query constraints and a set of representative images in thumbnail form. The user then clicks on one of the thumbnail images to zoom to full size or to retrieve the complete three-dimensional (3D) MRI data set for further study. After studying the retrieved images, the user can update the database with new pictures of interest, regions of interest, image attributes, or textual reports. 4.2.2 An Application Scenario: Teleconsultation Consolidation of health care resources and streamlining of services has motivated the development of communication technologies to support the remote diagnosis, APPLICATIONS 87 Figure 4.1. Content-based retrieval of MRI images based on ranges, structural volume, and functional glucose count of the amygdala and hippocampus. A color version of this figure can be downloaded from ftp://wiley.com/public/sci tech med/image databases. Figure 4.2. Content-based retrieval for hand-bone imaging based on hand-bone age and epiphyseal and metaphyseal diameter ratio. A color version of this figure can be down- loaded from ftp://wiley.com/public/sci tech med/image databases. 88 MEDICAL IMAGERY consultation, and management of patient cases. For the referring physician to access the specialist located in an expert medical center, the specialist must have access to the relevant patient data and images. Telemedicine is simply the delivery of health care using telecommunications and computer technologies. Teleradiology adds radiological images to the information exchange. In the past, textual and image information was exchanged on computer networks and the consultation between doctors was carried out over conventional phone lines. Teleconsultation enables the real time interaction between two physicians and improves the mutual understanding of the case. Both physicians see the exact image on their computer monitors, and each of them can see the mouse pointer of the other. When one physician outlines an area of interest or changes a window or level setting, the other physician’s computer monitor is automatically updated with the new settings. A neuroradiological teleconsultation system has been implemented between the UCSF main hospital and Mt. Zion hospital for emergency consultations and cooperative readouts [6]. Images are transferred from the referring site (Mt. Zion) to the expert center at UCSF over local area network using digital imaging and communications in medicine (DICOM) protocols and transmission control protocol/Internet protocol (TCP/IP). During the consultation, information is exchanged over both TCP (stream) and UDP (datagram) channels for remote control and display synchronization. Conversation is over regular telephone lines. 4.2.3 Image Archives for the Research Community: Brain Atlases In addition to being used for diagnostic purposes, imagery finds an important application as reference for clinical, research, and instructional purposes. Brain atlases provide a useful case in point. In this section, the construction of brain atlases [7] and their use is described briefly. Historically, brain maps have relied almost exclusively on a single anal- ysis technique, such as analysis at the cellular level [8], 3D tomography [9], anatomic analysis [10], PET [11], functional MRI [12], and electrophysiology [13]. Although each of these brain maps is individually useful for studying limited aspects of brain structure and function, they provide far more information when they are combined into a common reference model such as a brain atlas. The problem of combining data from different sources (both from different patients and from different modalities) into a single representation is a common one throughout medical imagery and is central to the problem of brain atlas construction. Brain atlases typically employ a common reference system, called stereotaxic space, onto which individual brains are mapped. The deformable atlas approach assumes that there exists a prototypical template of human brain anatomy and that individual patient brains can be mapped onto this template by continuous deformation transformations. Such mappings include piecewise affine transformations [14], elastic deformations [15], and fluid-based warping transforms [16,17]. In addition to geometric information, the atlas can also contain anatomic models to ensure the biological validity of the results of the mapping process [18]. CHALLENGES 89 As an alternative to a single deformable model, the probabilistic approach employs a statistical confidence limit, retaining quantitative information on inter- subject variations in brain architecture [19]. Since no “ideal” brain faithfully represents all brains [19,20], probabilistic models can be used to capture vari- ations in shape, size, age, gender, and disease state. A number of different techniques for creating probabilistic atlases have been investigated [21–24]. Brain atlases have been used in a number of applications including automatic segmentation of anatomy to measure and study specific regions or structures [25], [26,27]; statistical investigation of the structural differences between the atlas and a subject brain to detect abnormal pathologies [28]; and automatic labeling of neuroanatomic structures [28]. 4.3 CHALLENGES An MIDS stores medical image data and associated textual information for the purpose of supporting decision making in a health care environment. The image data is multimodal, heterogeneous, and changing over time. Patients may have different parts of the body imaged by using any number of the available imaging modalities, and disease progression is tracked by repeating the imaging exams at regular timely intervals. A well-designed imaging database can outperform the capabilities of traditional film library storage and compensate for limita- tions in human memory. A powerful query language coupled with an easy-to-use graphic user interface can open up new vistas to improve patient care, biomedical research, and education. Textual medical databases have attained a high degree of technical sophistica- tion and real-world usage owing to the considerable effort expended in applying traditional relational database technology in the health field. However, the inclu- sion of medical images with other patient data in a multimodal, heterogeneous imaging database raises many new challenges, owing to fundamental differences between the information acquired and represented in images and that in text. The following have been identified as key issues [29,30]: 1. Large Data Sets. The sheer size of individual data sets differentiates imaging records from textual records, posing new problems in informa- tion management. Images acquired in one examination can range from one or two megabytes in nuclear medicine modalities to around 32 megabytes each in mammograms and digital radiographs. A major hospital typically generates around one terabyte of digital imaging data per year [31]. Because of the large volumes, traditional methods employed in textual databases are inadequate for managing digital imagery. Advanced algorithms are required to process and manage multimodal images and their associated textual information. 2. Multimodality. Medical imaging modalities are differentiated by the type of biomedical information, for example, anatomic, biochemical, physiolog- ical, geometric, and spatial, that they can reveal of the body organ under 90 MEDICAL IMAGERY study in vivo, for example, brain, heart, chest, and liver. Modalities are selected for diagnosis depending on the type of disease, and it is the job of the radiologist to synthesize the resulting image information to make a decision. Features and information contained in multimodal images are diverse and interrelated in complex ways that make interpretation and corre- lation difficult. For example, Figure 4.3 shows both a CT scan and an MRI scan of the torso, and despite imaging the same part of the body, the two images look very different. CT is especially sensitive to hard tissue such as bone, but it presents soft tissue with less contrast. On the other hand, MRI renders soft tissue with very high contrast but does not image bone as well as CT. Scans of PET and CT look entirely different from one another and are also distinct from other modalities, such as computed radiography (CR) and ultrasound. PET acquires images of different body parts from those of mammographic images (Fig. 4.4). Even within the same modality and for the same anatomy, two sets of medical images can vary greatly in slice thickness, data set orientation, scanning range, and data representation. Geometric considerations, such as location and volume, are as important as organ functionality in the image interpretation and diagnosis. 3. Data Heterogeneity. Medical image data are heterogeneous in how they are collected, formatted, distributed, and displayed. Images are acquired Bone Soft tissue (a) Figure 4.3. (a) Single image slice from a CT scan of the body. Note that bone appears as areas of high signal intensity (white). The soft tissue does not have very good contrast. (b) Single image slice from a MRI scan of the body. Unlike CT, bone does not show up as areas of high intensity; instead, MRI is especially suited to imaging soft tissue. (Courtesy of A. Lou). CHALLENGES 91 (b) Figure 4.3. (Continued) from the scanners of different modalities and in different positions, repre- sented in internal data formats that vary with modality and manufacturer, and differ in appearance, orientation, size, spatial resolution, and in the number of bits per pixel. For example, the CT image of Figure 4.3 is 512 × 512 pixels in size, whereas the MRI image contains 256 × 256 pixels. It is worth noting that, with the exception of Doppler ultrasound, diagnostic images are acquired and displayed in gray scale. Hence issues pertaining to color, such as the choice of color space, do not arise for medical images. Color images are edited only for illustration purposes, for example, in pseudocolor nuclear medicine; physicians rarely use color images in diagnosis and therapy workups. 92 MEDICAL IMAGERY 4. Structural and Functional Contexts. Structural information in a medical image contributes essential knowledge of the disease state as it affects the morphology of the body. For example, the location of a tumor, with respect to its adjacent anatomic structures (spatial context), has profound implica- tions in therapeutic planning, whereas monitoring of growth or shrinkage of that tumor (geometric context) is an important indicator of the patient’s progress in therapy. However, what distinguishes medical images from most other types of digital images is the representation of functional infor- mation (e.g., biochemistry and physiology) about body parts, in addition to their anatomic contents and structures. As an example, fluorodeoxyglucose PET scans show the relative oxygen consumption of brain tissue — areas of low oxygen consumption (i.e., dark areas in the PET image) corre- spond to tissue that is hypometabolic and may be dead or dying. The PET findings can then be compared with MRI findings in expectation that areas of hypometabolism in PET correspond to areas of tissue atrophy in the MRI. The preceding example demonstrates the power of utilizing more than one imaging modality to bolster the clinical decision-making process. 5. Imprecision. Because of limited spatial resolution and contrast and the presence of noise, medical images can only provide the physician with an approximate and often imprecise representation of anatomic structures and physiological functionalities. This phenomenon applies to the entire (a) Figure 4.4. (a) FDG-PET image of the brain, coronal plane, 128 × 128 × 8 bits, (b) Mammography image, 4096 × 4096 × 12 bits.