Migration of Monte Carlo Simulation of High Energy Atmospheric Showers to GRID Infrastructure Adolfo Vazquez1,2, Ignacio de la Calle2, Jose Luis Contreras1, Aitor Ibarra2, Daniel Tapiador2 1

Grupo de Altas Energias Departamento de Física Atómica, Molecular y Nuclear Universidad Complutense de Madrid Avenida Complutense s/n, 28040 Madrid – Spain 2

INSA. Ingeniería y Servicios Aeroespaciales S.A. Paseo Pintor Rosales 34, 28008 Madrid – Spain [email protected] Abstract. A system to run Monte Carlo simulations on a Grid environment is presented. The architectural design proposed uses the current resources of the MAGIC Virtual Organization on EGEE and can be easily generalized to support the simulation of any similar experiment, such as that of the future European planned project, the Cherenkov Telescope Array. The proposed system is based on a Client/Server architecture, and provides the user with a single access point to the simulation environment through a remote graphical user interface, the Client. The Client can be accessed via web browser, using web service technology, with no additional software installation on the user side required. The Server processes the user request and uses a database for both data catalogue and job management inside the Grid. The design, first production tests and lessons learned from the system will be discussed here.

1. Introduction The MAGIC (Major Atmospheric Gamma Imaging Cherenkov) telescope [1] (http://wwwmagic.mppmu.mpg.de), a 17-meter Cherenkov telescope located 2200 meters above sea level on La Palma (Canary Islands), is dedicated to the study of the universe in Very High Energy gamma rays [2]. These particles arrive at the Earth's atmosphere producing atmospheric showers of secondary particles that can be detected on ground with the appropriate detectors. The MAGIC telescope is one of such detectors, sensitive to the Cherenkov radiation produced by the charge component of the showers. MAGIC relies on a large number of Monte Carlo simulations for the characterization and identification of the recorded events. Simulations are used to evaluate detector efficiencies and identify patterns to distinguish between genuine gamma-ray events and unwanted background events. Up to now, these simulations were executed on local queuing systems, resulting in large execution times and a complex organizational task. Due to the parallel nature of these simulations, and the uprising technological advances that Grid computing has had in the past few years, a Grid-based simulation system seems like the natural solution.

The main objectives of the project described here can be summarizes as follows: •

• •

• •

To re-launch the MAGIC Virtual Organization (VO) on EGEE, which supports users and research centres of the MAGIC Collaboration by providing resources. Re-launching the MAGIC VO involves setting up the required Grid services for this VO and testing the environment. To port the MAGIC simulation software to the Grid environment and develop a procedure for its remote installation on the nodes of the Grid environment associated to the MAGIC VO. To development an architecture that will ultimately provide users with a web-based graphical interfaces for the configuration and execution of Monte Carlo simulations on the Grid environment. The system should provide at the same time the user with a unique access point for the execution and control of simulation jobs. To install and configure a local Grid node to be in charge of submitting jobs received from a user through a web-based graphical interface to the MAGIC VO Grid nodes. To design a metadata database for data catalogue and management of the simulation jobs.

2. The MAGIC Telescope The first studies towards the design and later construction of the MAGIC telescope started in 1995, and finished with the final operational configuration in October 2003 (see figure 1). The main components of the telescope are the mount, able to change its observing location to any point in the sky in less than 30 seconds, the reflector, with 956 individual facet mirrors (covering a total area of 234 m2), and the camera (made out of 577 photomultiplier tubes) with associated fast readout The MAGIC Figure 1. The MAGIC Telescope, Observatorio del Roque de Los electronics. telescope is a member of the Muchahos, La Palma, Canary Islands (Spain). Cherenkov telescope family, which observes Cherenkov radiation flashes (on the visible and ultraviolet part of the electromagnetic spectrum) with a very short time interval (nanoseconds), produced by the interaction of the gamma rays with the upper atmosphere. Gamma-ray astronomy is based on the indirect observation of highenergy gamma rays emitted by objects from within and outside our galaxy, like pulsars or AGNs (see e.g. [3], [4]). Gamma rays interact in the upper atmosphere with atmospheric particles (mainly O2 and N2), which brake the primary particle giving origin to an air shower of secondary particles which travel to ground where they can be detected by the appropriate devices. The charged component of these showers emits Cherenkov light, which is captured by the mirrors of the telescope and focus onto the camera, generating specific patterns called shower images [2]. The MAGIC telescope was designed and constructed by an international collaboration of around 150 scientists and engineers associated to 19 institutions from Europe, America and Oceania.

3. The Simulation Flow in High Energy Gamma-ray Astronomy One of the obstacles that Cherenkov astronomy has to tackle is the suppression of the large number of different particles (Cosmic Rays) that arrive to the Earth’s atmosphere. This unwanted background makes necessary the development of techniques to distinguish between genuine gamma-ray generated showers from those generated by the particle background. This is done through the study of the image pattern generated on the camera by the Cherenkov flashes. In the case of gamma-ray produced showers, and due to geometrical effects, the pattern generated is elliptical and oriented towards the centre of the camera, as long as the telescope points to the putative gamma-ray source. Particles coming from other directions in the sky leave random orientated patterns on the camera, and due to the nature of the primary particle, random in shape and size. Here lies the importance of Monte Carlo simulations for Cherenkov astronomy, since most of the data recorded by Cherenkov telescopes is originated by particle background showers. Simulations allow creating libraries of gamma-ray and background generated images which are used to characterize the different events [5]. This characterization is later used to determine the nature of the primary particles received at the telescope. The MAGIC collaboration uses the code MMCS (MAGIC Monte Carlo Simulation), a customized version of the CORSIKA [6] simulation software, widely used by the high energy gamma-ray community for air shower simulations. This program takes into account different aspects of the environment where the shower is created, like atmospheric conditions or the Earth’s magnetic field. MMCS distinguishes four different simulation phases: the first one generates the traces of the shower particles while a second stage simulates particle interactions within the showers. A third stage considers electromagnetic interactions, to end in the fourth stage with the simulation of the generation of Cherenkov light. As a result of this simulation process, in practical terms, MMCS generates a file containing information of all photons that arrive to the ground near the telescope. The next step on the simulation chain is to include the detector (detector reflector and camera), starting with the reflection of Cherenkov photons on the telescope mirror. Reflector is software developed by the MAGIC Collaboration for this purpose, taking also into account issues like the attenuation of the light on the atmosphere. Reflector gets as input the photon data file generated by MMCS together with all the parameters that characterize the mirrors, including their geometry. The output of this code is another data file containing information of all the photons that will reach the camera. This information is fed in the last simulation step to the Camera software, also developed by the MAGIC Collaboration, to simulate the fast readout electronics behind the camera, and the generation of the shower image on the camera from the photons reflected on the mirror. Additional simulations include the telescope’s field of view and the night sky background to include in the whole process real observation conditions.

4. The MAGICGrid Project The simulation process described so far has been until now carried out using local resources, with no sharing of computational and storage resources among member institutions of the MAGIC Collaboration. Also, local resources are not necessarily exclusively reserve for the simulation process. At the same time, between 500 to 600 GBs of real data are produced by the telescope each night, which is received at different institutes for processing. This will soon be doubled as the second MAGIC telescope becomes fully operational during the second half of 2009. As an illustration of the kind of computer power required to run these simulations, an exercise carried out by Kornmayer et al. [7] concluded that an event takes around 6,5 seconds on a Pentium 2Ghz to be generated. This of course is dependant on the nature and energy of the events to generate. Other studies carried out by the same author noted that a cluster based on Condor middleware generates no more than 105 events per day, of which just 7% of the gamma-ray events and 0.1% of the proton events were useful for its later processing. From these results one can conclude that in order to get a set of events equivalent to a

night’s observation (around three and a half million events) 288 days and 19200 days of computer power would be required to generate the gamma-ray and background event sets respectively. Although the computational power needed for the event generation is a major issue, it is not the only bottleneck in the simulation process. The management of the size of the datasets generated by these simulations is not trivial, and since the analysis of the datasets does not necessarily take place at the production site, the transfer from the local storage to its analysis place can generate another bottleneck on the simulation analysis chain. The Grid presents itself as an alternative solution to both these problems, by providing a computational environment were all the computational resources associated to a given project can be joined, including storage, avoiding the transfer of large amounts of data. 4.1. Previous Work The work developed by Kornmayer et al. [7] represented in 2004 the first step inside the MAGIC Collaboration towards adapting Monte Carlo simulations of high energy showers to a Grid environment. A test production environment was designed using three institutions where the MAGIC simulation software (MMCS) and the MAGIC data analysis software (MARS) were running locally on each site, with the data stored locally with a metadata database to provide a global management of the system. This initiative led to the creation of the MAGIC VO on EGEE, where the work was left. 4.2. Re-launch of the MAGIC VO Picking up from the work started by Kornmayer et al., the MAGICGrid projects aims at designing a system, using modern software technology, to allow users to launch and control massive production of Monte Carlo simulations for the MAGIC Telescope on a Grid environment. The first work item for the MAGICGrid project was the re-launch the MAGIC VO into EGEE. At the end of this first stage, the MAGIC VO was formed by nodes of around 14 institutions, with around 45 computing nodes. A first test environment was set up by using nodes within the MAGIC VO at the PIC (Tier-1, Spain), CIEMAT (Tier-2, Spain) and TU Dortmund (Germany) Grid computational centers. The PIC is also the center that receives, stores and centralizes the data taken by the MAGIC Telescope. Also, a local Grid node, based both on the Globus (http://www.globus.org) and Gridway ([8], http://www.gridway.org) installations, was set up to be in charge of interacting with these Grid centers and be responsible for managing and submitting, jobs received from a given user. 5. MAGICGrid Project Architecture The MAGICGrid project is based on a Service Oriented Architecture that has been applied successfully in previous projects like [9], where configuration parameters of analysis jobs and their location and status in the Grid are exchange via web services. The MAGICGrid project distinguishes two components in its architectural design: a client side, where the user configures the simulation jobs, launches and monitors them, and receives the results (or its location via url inside the Grid), and a server side where an application manages the interaction with the Grid by launching the simulations to the different Grid nodes available. Figure 2 shows the UML diagram of the MAGICGrid architecture.

Figure 2. UML diagram of the MAGICGrid architecture, where a client and sever side are identified.

5.1. The MAGICGrid Client

Figure 3. A screen shot of one of the menus that constitute the MAGICGrid Client GUI.

To avoid any software installation, and the complexity associated with it, the user is presented, via web, with a client based on the Java Web Start technology. This way, the user only needs a web browser with the java runtime environment installed to run the client. The client interface provides all the configuration options needed by the MMCS software to run (figure 3). The configuration values are saved on a XML file which is submitted using web services to be processed by the server for its submission to the Grid for execution.

5.2. The MAGICGrid Server Once the XML configuration file is received by the server from the client, the server parses it and creates the required input cards for the simulation software. This processing is done by a Java application deployed on a Tomcat web application server. At the same time that the appropriate input cards are created, the metadata corresponding to the simulations is saved on a database, which will serve as a metadata repository to control the status of the simulations (see section 6). On this machine a Grid metascheduler provides the possibility of submitting jobs to the Grid without worrying about internal issues associated to a Grid environment, like error handling or scheduling policies. In order to send jobs using this metascheduling tool, a template has to be created for each simulation job in order to specify the computing requirements and the data location for that particular simulation job. MAGICGrid makes use of the Gridway metascheduler, included on the standard distribution of Globus and compatible with the Glite middleware, because of its support to the DRMAA API for Java Grid job submission. Throughout its execution, the status and metadata associated to the simulation job is updated automatically on the database, which can be accessed via public web interface. Once the simulation job has finished, the results can be both be retrieved in the local storage system or stored on a remote storage system, ideally the one nearest to the computing element where the simulation software was ran and where the subsequent data analysis would take place, avoiding data transfer and minimizing this bottleneck. 6. Database Two of the bottlenecks in a massive Monte Carlo simulation system are the large amounts of time required and computational resources availability to run simulations. A solution to both of these problems is to promote reusing existing simulations. As in other similar projects [10], MAGICGrid has implemented a metadata database for managing this issue. This database assigns a series of

metadata to the simulations when those are launched to the Grid environment for execution. The database allows saving information on the data generated by the simulation chain at three stages (shower generation, reflector and camera), so a later simulation job can be picked up at a certain stage of the process and be executed easily. In this way it implements a use case required by the MAGIC Collaboration, to allow the execution of a set of simulations with very similar simulation parameter values with minimum duplication of jobs to be ran. A prototype of the database compliant with this requirement has already been tested, along with a web interface that allows accessing the information on the database. So one of the key figures of the database is to control the status of the different simulation runs, keeping track of the version number of the different software components that make up the simulation chain. A case example would be that of wanting to produce new shower images to reflect an upgrade in the configuration of the camera software. In this case, only the camera part of the simulation chain needs to be run again, where the information available from the reflector part of the chain can be reused. 7. Conclusions The MAGICGrid project provides a user friendly system to run Monte Carlo simulations in a controlled Grid environment. The design system provides a web client, including a web-based graphical user interface for the configuration and execution of the simulation jobs, a web server to manage the submission of jobs to the Grid, and a metadata database to manage and control the simulation jobs produced. The designed system has been successfully applied to the generation of Monte Carlo simulations of high energy showers and telescope detection for the case of the MAGIC Telescope. The system proves easy to use and transparent to the user in terms of knowledge of Grid infrastructure or software installation, and provides at the same time a unique access point for the execution, control and storage of large amounts of Monte Carlo simulations in a Grid environment. Figure 4 shows a summary schematic representation of the MAGICGrid system. The achievements of the MAGICGrid project can be summarized as follows: • •

• • •

Successful testing, configuration and administration of a Grid VO (MAGIC VO), including the development of a workable test bed environment for simulations. Successful configuration of a Grid local system to allow research members of the MAGIC Collaboration to gain fast and easy access to Grid resources with minimal software configuration on their side, avoiding the complexity commonly associated to Grid environment tools. Successful porting of the MAGIC simulation software (MMCS) to a Grid environment, including the automatization of its installation procedure on the Grid computing elements. Development of a prototype metadata database for the management of large amount of simulation jobs. Development of a massive Monte Carlo simulation architecture that can be easily adapted to run simulations for projects with similar needs, like the future Cherenkov Telescope Array (CTA, http://www.cta-observatory.org/) with minimal changes.

Figure 4. Schematic representation of the MAGICGrid architecture.

8. Future work Throughout the course of this work, several areas and technologies have been investigated, and although not yet included in the MAGICGrid project, its future implementation is planned. In particular: • • • • • •

Grid-enable the remaining of the MAGIC simulation pipeline (reflector + camera). The use of multi-agent systems for job management. Easy and fast way to store and access the large amounts of simulations produced (e.g. no download required).. Expansion of the services available through the database. Virtual Observatory (VO) compliance of output data to allow the use of standard VO tools for data processing. Adapting and porting the Grid infrastructure developed for MAGIC for the simulation environment required by the future European project CTA.

Acknowledgements A. Vazquez would like to acknowledge support to work on this project by means of a fellowship granted by INSA S.A., a Spanish-based company in the Aerospace sector. References [1] [2] [3]

Cortina, J. 2005; Astrophysics & Space Science, 297, 245. Ong, R. A. 1998; Very high-energy gamma-ray astronomy, Physics Review, 305, 93. Aliu, E. et al. 2008; Observation of Pulsed γ-Rays Above 25 GeV From the Crab Pulsar with MAGIC, Science 322, 1221. [4] Albert, J. et al. 2007; Discovery of Very High Energy gamma-rays from 1ES1011+496 at z=0.212, Astrophys. J. Lett. 667, L21. [5] Hillas, A. M., 1985; Cerenkov light images of EAS produced by primary gamma, 19th International Cosmic Ray Conference, LA Jolla, 3, 445. [6] D. Heck, J. Knapp, J.N. Capdevielle, G. Schatz, T. Thouw, CORSIKA: A Monte Carlo Code to Simulate Extensive Air Showers (http://www-ik.fzk.de/corsika/). [7] Kornmayer, H. Hardt, M., Kunze, M. Bigongiari, C. Mazzucato, M. deAngelis, A. Cabras, G. Forti, A. Frailis, M. Piraccini, M. Delfino, M. 2004; A distributed, Grid-based analysis system for the MAGIC telescope. Proccedings of the CHEP 2004. [8] Huedo, E. Montero, R.S. and Llorente, I.M. 2005; The GridWay framework for adaptive scheduling and execution on Grids. Scalable Computing: Practice and Experience 6 (3): 1-8. [9] Ibarra, A. et al. 2005; Remote Interface to Science Analysis Tools for Grid Architecture: The XMM-Newton SAS Case. Procceedings of the Astronomical Data Analysis Software and Systems XVI. [10] Lefébure, V. and Andreeva, J. 2003; RefDB: The Reference Database for CMS Monte Carlo Production. Proceedings of CHEP 2003.