AppLeS: Picking the Best Mix of Resources

PROJECT LEADERS
Francine Berman, Professor, Department of Computer Science and Engineering, UC San Diego
Rich Wolski, Research Scientist, Department of Computer Science and Engineering, UC San Diego

igh-performance networks make possible the construction of virtual computers, or metasystems, that tap the resources of the machines on a network to run demanding applications. But just connecting computers together won't make this happen. An application needs to know what resources are available and how busy they are to ensure that the application can achieve the best possible performance. Researchers in the Heterogeneous Computing Group at UC San Diego are working to make this a reality. "Scheduling is the key to performance," said computer science professor Francine Berman, the group's co-leader. "Our goal is to give individual users a way to schedule their applications efficiently."

Collaborators Berman, professor in UC San Diego's Computer Science and Engineering Department, and Rich Wolski, soon to be an assistant professor of Computer Science at the University of Tennessee, believe that the best scheduling strategy is adaptive. They have developed a methodology they call "application-level scheduling," in which everything in the system is evaluated in terms of its impact on the application. This seemingly simple paradigm provides a powerful way to use adaptation to promote the best schedule for the application in a dynamic metasystem.

Metasystems, such as NPACI's Metasystems thrust projects Legion and Globus, can provide a single computational environment in which processor cycles, communication, and data are shared by many users. However, all of the metasystem components affect the performance of applications. In their most general form, metasystems consist of various kinds of resources with different performance characteristics, distributed over large geographic areas and linked by networks of various speeds and bandwidths. Performance is time-dependent--a different set of resources may be available at any given time, and the resource requirements of executing applications will vary, too. Schedules must utilize both application-specific and dynamic information to adapt to variations in performance due to other applications in the system.

ADAPTATION TO DELIVER PERFORMANCE

CORE OF THE METASYSTEM

ADAPTATION TO DELIVER PERFORMANCE

Berman and Wolski call their schedulers AppLeS, for Application Level Schedulers. "AppLeS scheduling agents use adaptation to automatically determine a performance-efficient schedule in much the same way as an intelligent user would for their program--only at machine speeds and with more information," Berman said. AppLeS agents determine and evaluate schedules, coordinate the transference of data and computation, forecast the capacity of resources at execution time, and integrate the results.

To determine the best schedule for its application, an AppLeS agent evaluates resources in terms of their load and availability and in terms of how they will be used by the application. Candidate schedules are determined for potential target resource sets and the performance of the application is forecasted. Critical information is provided by the Network Weather Service (NWS), developed by Wolski, which gives accurate forecasts for dynamic system parameters (p. 13). Finally, the AppLeS agent selects the best potential schedule and executes the application using that schedule.

CORE OF THE METASYSTEM

AppLeS are being built for a variety of NPACI applications, and results show that the adaptive approach used by AppLeS outperforms other approaches used for scheduling metasystem applications. Collaborative efforts by the group focus on developing AppLeS for applications developed by UC San Diego's Collaboratory for Microscopy and Digital Anatomy project, the Computational Center for Macromolecular Structure's DOT molecular interaction program, Caltech's Synthetic Aperture Radar Atlas project, and other projects.

In the near future, the AppLeS "corps" will also work with collaborators to incorporate AppLeS and NWS methodology into NetSolve from the University of Tennessee and Oak Ridge National Laboratory (a project in NPACI's Programming Tools and Environments thrust area), the University of Wisconsin's Condor high-throughput job scheduling system, the NASA Information Power Grid program, and Japan's Ninf Global Network Computing Infrastructure project. AppLeS applications and the NWS also target NCSA as well as NPACI resources.

"Performance is the point," Berman said, "and AppLeS agents provide a critical boost to the achievement of high performance for metasystem applications on heterogeneous, networked computers." --MG