Telescience for Advanced Tomography Applications

PROJECT MANAGER
Martin Hadida-Hassan, UC San Diego/SDSC

xamining the 3-D structure of the smallest components of nerve cells has the potential to reveal fundamental insights into how the brain and nervous system work. But to study the complexities of these tinystructures in their natural environment, at the highest resolutions, and in three dimensions, neuroscientists need access to very scarce advanced imaging instruments--such as high-energy electron microscopes--and computer resources--such as parallel supercomputers and distributed workstations. The Telescience alpha project is creating a set of tools to allow researchers to go seamlessly through the steps required to create images on a remote high energy microscope, use elaborate software programs to compute the 3-D structures--called electron tomographic volumes--and deposit the most useful representations of these data into a database forming a library of computerized cell-level brain structures. As a result of this work, scientists and students in any location, using a reasonable network connection and a modern Web browser, will be able to harness these advanced capabilities.

"The Telescience alpha project is developing a telescience infrastructure that will accommodate future data sets and support future application scenarios," said Mark Ellisman, leader of both the Telescience alpha project and the Neuroscience thrust area. "The project will demonstrate an end-to-end solution for the specific problem of telemicroscopy for biological specimens. With further elaboration, this remote data acquisition and tomography environment will find use in other disciplines ranging from Engineering to Earth Systems Science."

A number of ongoing collaborations between Neuroscience researchers at UC San Diego's National Center for Microscopy and Imaging Research (NCMIR), directed by Ellisman, and computer scientists at UC San Diego, the Information Sciences Institute (ISI) at the University of Southern California, and elsewhere, have been attacking the various components of teleinstrumentation. The Telescience alpha project is integrating those collaborations to create a seamless and transparent Grid-based discovery environment.

REMOTE INSTRUMENTATION

DESKTOP RESOURCE

GRID-BASED TOMOGRAPHY

DISTRIBUTED DATA

THE FINAL FRONTIER

REMOTE INSTRUMENTATION

To study a cell, such as a neuron, with an electron microscope requires slicing a specimen into portions about 1,000 times thinner than a human hair, placing one such slice into the microscope under high vacuum, then sending a beam of high energy electrons through it. The electrons penetrate the sample, and the microscope's electromagnetic lens system forms an interior image that can be viewed with video cameras or recorded on film with high-resolution digital camera systems.

In telemicroscopy, this process can occur with a researcher located far from the microscope's physical location. Ellisman pioneered the practice and established the Collaboratory for Microscopy and Digital Anatomy--part of NCMIR--at UC San Diego in 1992. "The NCMIR has worked with SDSC for more than a decade," he said. "Telemicroscopy makes it possible for more researchers to utilize our 400,000-volt electron microscope and to benefit from the high-
performance computing capabilities at SDSC."

The NCMIR group has also had an ongoing collaboration with scientists at Argonne National Laboratory. In 1998, the telemicroscopy code was modified to run on Globus--a program for distributed computing environments that is part of the Metasystems thrust area--and supported a microtomography demonstration at the Advanced Photon Source (APS) at Argonne.

APS scientists made high-resolution X-ray observations of complex but tiny structures, including micromechanical gears and the head of an ant, and reconstructed them as volume renderings using the Globus-enabled code. "While in our alpha project we're initially focusing on telemicroscopy access to electron microscope resources, the APS experiment demonstrated how the concept can be expanded to encompass other unique instruments," Ellisman said. "It also demonstrated the benefits of distributed computing with Globus."

Another 1998 demonstration allowed Japanese and American scientists present at UC San Diego to operate the world's largest and most powerful transmission electron microscope, located at the Research Center for Ultra High Voltage Electron Microscopy (UHVEM) in Osaka, Japan. In this first successful demonstration of trans-Pacific telemicroscopy, the power of the UHVEM instrument allowed specimens 4.0 micrometers thick to be used--dense enough to allow capture of long pieces of a neuron's dendrite in one longitudinal section. Such images enable scientists to count the spines--synaptic specializations--on dendrites and to characterize their shape, something that was previously very difficult to do.

DESKTOP RESOURCE

Telemicroscopy brings these resources and benefits to a researcher's desktop. Remotely controlling the microscope stage over the Internet they can focus on an area of a specimen that they want to view in 3-D. For example, an Alzheimer's investigator might be interested in rendering the fibrous masses within the cell the scientists believe indicate the disease. The researcher then initiates a tilt series--high-resolution 2-D images that are taken as the stage rotates between –60 and +60 degrees. The position of each new image is compared against the coordinates of its predecessor, and the projected image is repositioned automatically by electronic means. The tilt series forms the basis for tomographic reconstruction, the second component of the Telescience alpha project.

The tomography code, collaboratively developed by the researchers and groups now involved with the alpha project, is in use daily at the NCMIR lab and has been run on up to 128 processors of the 256-processor T3E at SDSC, producing volume-renderings quickly, but not immediately. Steve Lamont, the late Steve Young, and Jamie Frey from NCMIR, Shava Smallen and Walfredo Cirne from computer scientist Fran Berman's group at UC San Diego, and Mei-Hui Su and Carl Kesselman of ISI, all participated in the code's development.

GRID-BASED TOMOGRAPHY

"Tomography pushes the limits of supercomputing resources," Ellisman said. "With optimized code we've been able to reconstruct a 3-D image in under an hour on the SDSC Cray T3E, but even that isn't fast enough for real-world applications--especially since the queue for the T3E can often run an hour or longer. The collaboration with Globus, AppLeS, and NWS lets us bypass such obstacles."

A goal of the alpha project is to have nearly instantaneous, automated tomographic feedback. In this scenario, as the researcher views a specimen on the microscope through a Web device, they can also simultaneously view the 3-D reconstruction of a given spot on the sample. "The intent is to help the researcher more rapidly acquire their data and to do so in fewer microscope sessions," Ellisman said. "If the rendering doesn't show anything of significance, they can immediately move on to another area, rather than having to initiate the process again at a later date, after they've waited and received a 3-D image back. Globus, AppLeS, and NWS are making this possible."

Globus, led by Kesselman and Ian Foster of Argonne National Laboratory, dynamically distributes the parallel tomography code out to available processors on networked resources, drawing on adaptivity provided by AppLeS--the Application Level Scheduler--and the dynamic forecasting ability of NWS--the Network Weather Service. Significantly, this happens transparently, without the user having to log in to individual machines or manipulate or transfer files. Globus tools allow scientists to log in securely at any access point and then interactively use applications programs, scientific instruments, databases, and tools for scientific visualization. "Globus takes advantage of the ubiquitous access and uniform interface elements of the Grid," Kesselman said.

AppLeS, which is also a Metasystems thrust area project, is led by Berman and Rich Wolski of the University of Tennessee. The software has adaptive schedulers that allow applications to achieve performance in the decentralized, dynamic, and heterogeneous Grid environment. Wolski is also the developer of the NWS, a monitoring and prediction tool used by AppLeS and others to keep track of available load
and bandwidth on Grid resources. NWS is another Metasystems project.

"Applications typically target either batch environments, as on supercomputers, or interactive environments, as on PCs or other workstations," Berman said. "We have given the tomography code the flexibility to achieve even better performance by targeting both batch and interactive resources simultaneously and by using all of the resources available at execution time."

DISTRIBUTED DATA

Future objectives for the Telescience project will extend its discovery capabilities using products and resources of the Data-Intensive Computing Environments (DICE) thrust area. Specifically, the SDSC Storage Resource Broker (SRB) will help federate databases populated with raw and refined data acquired by the telemicroscopy system. Cell-level data from tomography will be integrated with a multi-scale database of brain structures that derives from other Neuroscience projects, including the brain-mapping project at UCLA and neuron modeling efforts at
Montana State University.

Eventually, as scientists use the microscope and get feedback about their data volumes, they will also be able to query the federated databases for comparisons against similar structures, and to visualize elements of the cell--or the cell in its entirety--for more thorough analysis. As envisioned, the whole process will occur quickly and transparently enough to inform and guide researchers' decision-making process regarding specimen acquisition. The cell-level data sets from automated tomography will be available to researchers using computational modeling tools such as MCell, GENESIS, and NEURON. As part of the Neuroscience thrust area work, GENESIS and MCell are being expanded to operate in conjunction with the PACI computational infrastructure.

THE FINAL FRONTIER

"We're in pursuit of a great mystery--an understanding of the nervous system," Ellisman said. "We progress in stages and attack from many fronts. This alpha project is a concentrated step forward. It harnesses what we already do well--remote instrumentation--to even greater power, likely resulting in increased knowledge at a pace we couldn't imagine just a few short years ago." --AF *