The costs of a free, open-source software ­implementation: building ICCROM, a web-based archival catalogue using ICA-AtoM* María Mata Caravaca and Anne-Marie Viola

To meet the needs of archives around the world whose budgets do not allow for expenditure on software licenses, the International Council on Archives (ICA) supported development of the Access to Memory (AtoM) system, a free, open-source processing and access tool. For archives facing tighter budgets than ever, the appeal is obvious. So what’s the catch? This is the story of the hidden organisational costs of an open-source software implementation project at the Archives of the International Centre for the Study of the Preservation and Restoration of Cultural Property (ICCROM).

Background The International Centre for the Study of the Preservation and Restoration of Cultural Property (ICCROM) is an intergovernmental autonomous organisation created in 1956 by UNESCO and dedicated to the conservation of cultural heritage [Figure 1]. It exists to serve the international community as represented by its 130 Member States. The dual aims of the organisation are to improve the quality of conservation practice and raise awareness of the importance of preserving cultural heritage. Its five main areas of activity are: Training, Information, Research, Cooperation and Advocacy. ICCROM currently has 35 people on its staff [Figure 1]. The ICCROM Archives are part of the Knowledge and Communication Services and are located at ICCROM’s headquarters in Rome. Established in 2002, the Archives arranges, María Mata Caravaca has been the Archivist of ICCROM (International Centre for the Study of the Preservation and Restoration of Cultural Property) since 2002. She graduated in Fine Arts from the University of Granada, Spain. She also trained in conservation and restoration in Florence and at ICCROM in Rome, and received her professional archival degree from the Vatican School. From 2008 to 2012, she has been involved in InterPARES (The International Research on Permanent Authentic Records in Electronic Systems), leading a case study on the design and implementation of a records management system at ICCROM. Anne-Marie Viola is a metadata specialist and graduate of the Pratt Institute School of Information and Library Science with a background in developing online tools for description and discovery using Encoded Archival Description. She worked as Project Archivist at ICCROM between 2010–2011, during which period she implemented ICA’s Access to Memory (AtoM) open-source archival description software and migrated the institution’s Excel-based inventory. She is now working at the Image Collections and Fieldwork Archives at Dumbarton Oaks Research Library and Collection, an institute of Harvard University based in Washington D.C. *  This article, written in October 2011, presents the implementation of open-source archival description software ICA-AtoM, release 1.1. ICA-AtoM has recently released new software upgrades with significant enhancements. One is the improvement of the indexing process, mentioned in this article as a system problem. The upgrades make this process speedier and much more efficient. This article has also been published in Italian in Archivi & Computer, 1, 2012, pp.26–41. doi:10.3828/comma.2011.2.06

74

María Mata Caravaca and Anne-Marie Viola

Figure 1:  ICCROM’s headquarters in Rome

describes and preserves records generated by ICCROM’s administration and activities, and provides reference assistance to internal users and external researchers who can demonstrate a need for use [Figure 2]. The Archives contain institutional records related to ICCROM programmes and projects, courses, missions by staff, and the financial and administrative work of the General Assembly, Council and the Office of the Director-General. The fonds comprises 550 linear metres of textual paper records, drawings and plans, photographs, audio and video recordings, and digital material. There are around 4,500 drawings and plans, more than 200,000 photographs (among which are slides, photographic prints, negatives and digital images), and around 1,000 units of audiovisual material.1 Archival records were originally described and located using an Excel (XLS) inventory maintained by the organisation’s lone archivist, the sole staff member responsible for all reference requests and for processing. This inventory was not made available to researchers, which meant that users had no way of independently searching the Archives’ records. Virtual access to records was limited to an image database, which contains ICCROM’s drawings and photographs series and is available through the ICCROM website. Wishing to produce descriptive records based on international standards, to use a database management system for archival description, and to improve records access, the ICCROM Archives decided to undertake a feasibility study of ICA-AtoM, the open source archival description software supported by the International Council on Archives. As a member of the ICA, the ICCROM Archives was aware of the AtoM project and the support it received from the international community.

Feasibility study The feasibility study was designed to obtain a better understanding of the software’s characteristics, to test its functionalities and usability, and to verify its possible implementation at ICCROM and the ability to import the existing inventory data into the new system. The system was analysed and installed locally for testing. Following an initial review and trial import it was determined that ICA-AtoM fitted many of ICCROM’s needs: 1  http://www.unesco.org/archives/sio/Eng/presentation.php?idOrg=1042.

The costs of a free, open-source software implementation

Figure 2:  The Archives containing institutional records related to ICCROM programmes and projects

• First and foremost, it is a free and open-source software. This would allow utilisation of existing resources and minimise expenses. • The system is also user-friendly for end-users and administrators. • It allows different levels of access for a variety of users groups. It restricts access to personnel files, financial statements and other confidential documents, and hides all physical location information from users. • It creates and makes digital surrogates available to users via query and browse functionalities. ICA-AtoM adheres to international descriptive standards and offers ‘multi-lingual interfaces with content translation features’, both relevant benefits for an international organisation such as ICCROM. AtoM documentation also states that the software supports ‘multiple collection types’ as well as multi-repository implementations for organisations ‘small and large.’2 In short, the system provides a simple, point-and-click interface, which enables archivists to create hierarchical descriptive records findable via a basic search engine and browseable 2  http://ica-atom.org/about. For a more technical description, visit https://www.ica-atom.org/doc/What_is_ ICA-AtoM%3F.

75

76

María Mata Caravaca and Anne-Marie Viola

directory. By providing fields for each element of description, archivists are relieved of the need to know EAD, although they still need a firm grasp of the descriptive standards in order to understand how to complete each field and build an effective record. AToM appeared to require little in the way of finances or technical expertise, and perhaps most importantly, it also provides both an export and import functionality which would enable ICCROM to leverage its existing inventory data, which if converted could become the basis of the new database. With a stated purpose of providing ‘free and open-source software that enables institutions to make their archival holdings available online, especially those who could not otherwise afford to do so’, the organisation was confident that AtoM would meet their needs and cost little in the way of implementation.

Resource allocation The expenses of this project were real if indirect. As is often the case, implementations are impeded or otherwise delayed by failing to recognise expenses in the form of time, expertise and existing resource allocation. The challenge of such expenses is in how they affect project planning – or more correctly, how they may fail to be fully anticipated and thus appropriately allocated. ICCROM began this project with a feasibility study, both to evaluate the AtoM software and to outline a methodology for its implementation at ICCROM. The project team allowed one and a half months for planning. This included researching and testing out the data conversion method subsequently used to capitalise on the existing inventory’s data. The project was expected to take six months. One of the very first steps was to identify internal and external resources. As is often the case with such initiatives, the organisation sought outside assistance because of the need for both a particular expertise and additional manpower. The greatest expense of the ICCROM project was a project archivist working full-time to complete the metadata conversion process and build the system. In addition to coordinating this effort, the person served as EAD expert and was responsible for authoring all project documentation, managing communications with the AtoM community, and designing initial usability testing. The project also required significant contributions of time and expertise from both the institutional archivist and the organisation’s software programmer. The archivist, who had intimate knowledge of the Archives and archival description standards, played a key decision-making role, while the programmer supplied all technical knowledge, executing server installation, manual search indexing and all necessary adjustments and programming solutions described later in this article. Documentation and the support of the online user community were two more essential resources. Both AtoM and its underlying software, Qubit, provide web-based documentation.3 But users would be lost without the expertise communicated and archived in the forums managed by developers of both software systems and enhanced with user feedback. ICA-AtoM maintains its user discussion group,4 where both programmers and users regularly 3  The ICA-AtoM manual is available at https://www.ica-atom.org/doc/User_manual [accessed 21 Dec. 2012]. Qubit’s wiki can be found at https://www.qubit-toolkit.org/wiki/Main_Page [accessed 21 Dec. 2012]. 4 At https://groups.google.com/forum/?fromgroups#!forum/ica-AtoM-users [accessed 21 Dec. 2012].

The costs of a free, open-source software implementation

post. This forum is an invaluable resource for learning how other users have resolved implementation issues. A separate discussion group is maintained about Qubit.5 Other helpful sites often referenced include the Library of Congress EAD tag library,6 crosswalks for EAD and ISAD(G),7 and Qubit’s EAD import mapping.8

Existing data: rearrangement and conversion With a cursory understanding of AtoM and having identified resources, the next priority was to understand how ICCROM’s existing data would function within this new tool and what amount of work would be required to begin using it. ICCROM’s inventory had been created and was maintained using Excel. Records were described using eight fields: Document Number, Office of Creation, Current Location, Span Dates, Title, Description, Volume, Value. The fonds was intellectually arranged into six subfonds. Each subfonds was composed of series and often subseries, which communicated the organisation’s different activities. In its arrangement, ICCROM specified up to seven or eight levels of description (fonds, subfonds, series, subseries, file, subfile1, subfile2, item), two more than AtoM’s default list of values (fonds, subfonds, series, subseries, file, item, plus collections to distinguish records assembled inorganically from fonds). Fortunately, AtoM’s developers thought to make record level labels customisable through the system’s administrative module [Figure 3].

Figure 3:  Taxonomy of record levels in AtoM’s administrative module 5 At http://code.google.com/p/qubit-toolkit/ [accessed 21 Dec. 2012]. 6 At http://www.loc.gov/ead/tglib/element_index.html [accessed 21 Dec. 2012]. 7  At http://www.loc.gov/ead/tglib/appendix_a.html [accessed 21 Dec. 2012]. 8 At https://www.qubit-toolkit.org/wiki/EAD [accessed 21 Dec. 2012].

77

78

María Mata Caravaca and Anne-Marie Viola

The AtoM system also affords the possibility of significantly greater description than ICCROM had been providing. An archival description in AtoM specifies ten areas of description, each of which comprises three to five (G) elements and addresses a different facet of the record (e.g. ‘Name of Creator’ or ‘Archival History’ in the ‘Context’ element).9 According to its documentation, while ICA-AtoM is designed around the ICA’s suite of descriptive standards (ISAD(G), ISAAR, ISDIAH and ISDF), ‘it is intended to be flexible enough to accommodate other practices based on other (national or local) descriptive standards.’10 Of the 26 elements defined in ISAD(G), the system only requires six specific elements for any individual archival description to be in compliance with ISAD-G: reference code, title, creator, date(s), extent and level of description. In the team’s assessment of ICCROM’s existing description, the current inventory was meeting all six ISAD(G) requirements, along with providing location data and, often, details relevant to other fields, such as the catchall ISAD(G) field, ‘Scope & Content’ which had been labeled generically as ‘Description’. Upon closer review of this field, we found that the data in the ‘Description’ field actually belonged in a number of other ISAD(G) fields. Although not present for all records, we identified six additional fields: ‘Immediate source of acquisition or transfer’, ‘Appraisal, destruction and scheduling information’, ‘Conditions governing access’, ‘Physical characteristics and technical requirements’, ‘Existence and location of copies’ and ‘Related units of description’. New columns were created in the original Excel file for these fields and where applicable, data was then moved from ‘Description’ to its more appropriate field. To leverage AtoM’s import functionality, it was necessary to convert the existing data from Excel (XLS) to XML. Research completed during the feasibility study identified comprehensive conversion documentation published by the Northwest Digital Archives.11 NWDA’s process makes creative use of the Microsoft Word mailmerge function and necessitated only the creation of an EAD template and minor modifications to the existing files. Although the conversion process is clearly outlined and uses simple software, and AtoM itself features a very user-friendly interface with straightforward documentation, a working knowledge of archival description and encoding standards as well as mark-up languages was essential. The project archivist constantly referenced documentation to ensure that data was being crosswalked correctly, according to the mappings set up in AtoM. The workflow of the data conversion from Excel (XLS) to XML required the following steps: 1. Re-arrangement of the Excel file to make computer-readable the description levels of the different subfonds (series, subseries, files, subfiles, items). This mainly consisted of indicating the position of the description units in the hierarchy of the subfonds, naming and numbering each record level, and breaking out description across new description fields [Figures 4 and 5]. 2. XLS-XML metadata conversion, which consisted of: a. Creating the EAD template (pictured in Figure 6), which included the Excel description fields enclosed within their equivalent EAD XML tags. 9  http://ica-AtoM.org/doc/RS-1#3.0 [accessed 21 Dec. 2012]. 10  http://ica-atom.org/doc/Descriptive_standards [accessed 21 Dec. 2012]. 11 At http://orbiscascade.org/index/northwest-digital-archives-tools [accessed 21 Dec. 2012].

The costs of a free, open-source software implementation

b. Merging the Excel data into the EAD template using the mailmerge function of MS Word so that the resultant XLS data appeared enclosed between the EAD tags. c. Adding EAD XML structure, including the Document Type Definition (DTD) and other top-level descriptive elements, and then nesting the descriptive components using Replace Text, a simple piece of freeware that enabled users to perform simultaneous search and replace operations. d. Saving as XML and validating these files, which can be done simply by using a web browser such as Internet Explorer.

Figure 4:  ICCROM’s original Excel inventory

Figure 5:  ICCROM’s inventory restructured to indicate each record level and its relationship to the records above it

This step-by-step documentation was developed based on the NWDA process, and ultimately, 56 different files were produced as a result of the series and sub-series conversion, totaling 11.38 MB and over 12,000 records that describe the totality of the ICCROM fonds. In summary, restructuring ICCROM’s Excel inventory was the bulk of the work and took approximately four months. In contrast, the actual XML conversion process can be completed in about half an hour per file.

79

80

María Mata Caravaca and Anne-Marie Viola

Figure 6:  The EAD template created using Microsoft Word’s mailmerge function

Managing access Expanding access to any institution’s archival holdings naturally raises security concerns. ICCROM’s records include personnel files, financial statements and other confidential documents to which the organisation wanted to restrict access. So, before proceeding with implementation, the team spent time researching AtoM’s functionalities to be sure they understood how the system controls access to sensitive data. AtoM enables administrators to create individual user accounts as well as user groups each with their own set of permissions. In addition to assigning permissions to specific user groups for all archival descriptions, the system enables administrators to set permissions for specific records under ‘permissions by archival description,’ instead of ‘permissions for all archival descriptions.’ This function allows ICCROM to ‘hide’ certain confidential records from all users. One of the first options that ICCROM researched was the possibility of differentiating public users from internal users (e.g. staff, interns) in order to provide internal users with access to ICCROM’s recent records. As defined in the Archive’s access rules, records are not made available to the public until they are at least 20 years old. ICCROM also wanted to restrict access to the physical location of its records. In AtoM 1.1, all users are able to see the name and the type of container for each archival description, displayed under the heading ‘Physical Storage’ at the lower right hand corner [Figure 7]. However, to administrators this same string of text appears as a hyperlink, enabling them to access the corresponding storage record that lists the ‘Location’ for the said container

The costs of a free, open-source software implementation

Figure 7:  Physical storage information, indicating container type and identifier, as it appears to public users

Figure 8:  The actual physical storage record for Box TRN1-90 indicates its location and other contents

along with a list of ‘Related resources’ (which are links to all other archival descriptions with the same location). (It should be noted that if the record does not have location data, this section does not appear) [Figure 8]. Although built to provide the necessary security, this feature was complicated by our EAD import and ultimately caused a significant problem. Despite Qubit mappings which

81

82

María Mata Caravaca and Anne-Marie Viola

Figure 9:  Physical storage information as it appears to logged-in users, includes location and a hyperlink to the related storage record

spelled out how to crosswalk each data element using the appropriate EAD field, the team’s initial import resulted in records lacking physical location data. This data had been encoded in the correct EAD fields, and , in a valid XML file, yet while container data appeared, physical location data would not display within AtoM. To resolve this problem, ICCROM’s programmer devised a workaround, which required moving data inside of tags. Following import, ICCROM’s programmer used a SQL instruction to move the data to the proper field for location data within the database. The end result is that location data displays immediately following the container data under ‘Physical Storage’ on the archival description record, which appears to all authenticated users [Figure 9]. Because of this, this information could not be hidden from authenticated users with read-only privileges as had been hoped, and so it was not possible to create an internal staff user group which would maintain the necessary data security.

System limitations ICCROM’s implementation installed Release 1.1, the first non-beta release of ICA-AtoM.12 The team’s programmer handled the installation as well as all other technical adjustments of the system. The real difficulties of the implementation arose from three different limitations of the software: the EAD crosswalk related to location data (as explained above), the XML import functionality, and the indexing process. 12  http://ica-atom.org/doc/Release_1.1 [accessed 21 Dec. 2012].

The costs of a free, open-source software implementation

AtoM works on a virtual machine in a VMware ESX 3 server based on an HP ProLiant DL380 G4. While the underlying hardware appears fairly powerful, the virtual machine only uses the CPU (one Xeon at 3.8 GHz). The rest of the system is virtualised and ‘mediated’ by the ESX system. The virtual machine requires 1Gb RAM and 40Gb hard drive and utilises Linux Centos 5.5 – one of the ‘free’ Red Hat Enterprise distributions available – as its operating system. ICCROM found installation of the required PHP version (5.3 or later) challenging, but believes that to be more of a Linux distribution issue than something specific to AtoM. The other components of the system, Apache and MySQL, were installed without any difficulty. XML import: As described above, ICCROM chose to build its fonds by importing XML files using AtoM’s import functionality. At best this functionality can be described as ‘temperamental,’ as no other feature of the system posed more of a challenge to use. While designed simply, it required significant time and effort to make it work. On average, files took anywhere from one to six minutes to load, if they were able to be successfully imported. It appeared that the functionality required a careful balance of three factors: XML file size, maximum execution time and memory limits in the PHP settings. Although AtoM’s programmers assert that ‘There is no absolute maximum size of file that can be imported, it depends on your system limits more than anything,’ the maximum size that the team could get to load was 2MB.13 The team was unable to get files of 2.7 and 4.8MB to import. So the team’s solution was to break these files into component XML files, the total of which required 56 individual imports. In addition to the time required to import each of these individual files, the team then needed additional time to assemble the fonds within the system. Oftentimes, however, import efforts resulted in either a blank screen or a ‘500 Internal Server Error’ error message (indicating that server log should be consulted for more information about why an import failed). The blank screen was a result of the import script timing out. ICCROM resolved this by manipulating the PHP settings to increase both the server’s maximum execution time and memory limits. The system has a default setting of 32MB, which the team increased to 128MB when they first tested the import feature. This appeared to work until they tried to load files greater than 1MB, at which point they decided to increase the limit again to 512MB. This limit enabled the team to import files but was increased to its final setting of 1024MB after their initial indexing effort failed because of too little memory (as described below). In addition to adjusting the PHP settings, the import was facilitated by applying a patch which disabled the search indexing that occurs automatically upon uploading a new file and so significantly increases the resources required of the server. Indexing: Having disabled indexing in order to import files, the search index then had to be rebuilt manually. In doing this, the team encountered three different errors, each of which posed even greater challenges to the project’s timeline. The first, a segmentation fault, caused the system to repeatedly crash. This was fairly quickly resolved using a patch provided by the developers. The second error message, received after the indexing process had been going for 48 hours, indicated that the allocated memory limits of 512MB had been exceeded and, 13  http://groups.google.com/group/ica-atom-users/browse_thread/thread/4a79b291fff8d880/4dd1c106da5b 9d91?lnk=gst&q=import+2+MB#4dd1c106da5b9d91 [accessed 21 Dec. 2012].

83

84

María Mata Caravaca and Anne-Marie Viola

as described above, the programmer was able to resolve this problem by raising the limit to more than 1GB (1024MB), and restarting the process from the beginning. The third error was received after another four days of indexing and was caused by a limit to the number of open files in the operating system, which ICCROM’s programmer increased to 50,000 in accordance with the suggestions made in the forum. After reporting on how long it was taking to index, the forum suggested optimising the indexing process, which enabled ICCROM to complete the entire indexing in four hours instead of four days! Programmers also indicated how to restart the process from the last record indexed by using another Qubit patch, another timesaver.

Descriptive opportunities With the creation of the ICCROM fonds within the AtoM system [Figure 10], the organisation was presented with its first-ever opportunity to describe the fonds as a whole as well as each of its subfonds, something previously only understood after collective review of each subfond’s component data. This meant listing date ranges and calculating the volume of component record groups, as well as summarily describing these record groups using additional descriptive fields like ‘Scope and Content’ and ‘System of Arrangement.’

Figure 10:  In addition to search, AtoM enables browsing by top-level records, which afforded ICCROM the opportunity to describe each of its subfonds

The costs of a free, open-source software implementation

ATOM also affords the opportunity to provide greater context about the organisation’s records by describing their creators in separate authority records. Similar to archival descriptions, authority records are divided into separate descriptive areas. Specific elements are based on the ICA’s International Standard Archival Authority Record for Corporate Bodies, Persons and Families (ISAAR). Creator information was included in ICCROM’s XML files using the field. In addition to naming the creator, the authority record type was indicated: corporate entity (), family (), individual (, or just ). Upon import, AtoM automatically generates separate but linked authority records which include this information along with space for additional description. Elements afford space to provide additional details regarding relevant dates, places, functions and relationships specific to the creator. Yet these corollary records do more than describe. In the same way that location records list all files in the same container, authority records virtually link archival descriptions produced by the same creator by including a list of links to all records within which the creator is named, thereby also providing enhanced browseability. However, the format of ICCROM’s original data – with all creators listed in a single cell – meant that all creators were encoded as a single creator in the conversion process. This resulted in separate records for each instance or combination of creators (e.g. one record for ‘Mike Smith’ and a separate record for ‘Mike Smith, Kelly Jones’), rather than identifying and linking to each individual creator. The same problem occurred when multiple locations were encoded for a single archival record. Separating out individual creators and locations would improve the browseability that these features afford.

Future plans Even if the bulk of the project, consisting of data conversion and transfer, was successfully completed, we cannot say that this project has been concluded. Many improvements still need to be carried out to be able to provide a comprehensive and user-friendly tool for access. After transferring the records inventory, the team undertook a comprehensive review of the data in the system. Small corrections still need to be carried out to increase records’ readability and accessibility: file titles in the same series can appear indistinguishable because ICA-AtoM’s browseable tree menu is too narrow; sometimes child levels of a series are not displayed in the correct order due to errors made during the arrangement of the Excel files; spelling mistakes and so on. As this article was being authored, the inventory in ICA-AtoM still contained only draft records, which means that users were not yet able to access anything. The next step for the team was to publish the records. Yet because physical location data remains visible to any authenticated person, ICCROM was unable to differentiate access to internal and external users. This means that internal users continue to need to refer to the archivist for the organisation’s most recent records. The promise of a future AtoM upgrade that would allow administrators to hide certain fields to users, such as location data in the Physical Storage field, is important, as only then will ICCROM provide appropriate functionality to its internal user group.

85

86

María Mata Caravaca and Anne-Marie Viola

However, before making the site available to anyone, a usability study of the system is needed to test the site’s effectiveness, efficiency and satisfaction by a select group of representative users. These users will help by identifying problem areas and providing useful feedback. The study has been planned and designed, but not yet implemented. Future resources will be allocated to this project to properly maintain and complete the archival description data, and to make the Archives accessible to users.

Conclusion In summary, AtoM affords valuable opportunities to expand access to archival inventories. Users will be pleased with the variety of ways to search, while archivists will appreciate the simplicity of the point-and-click interface that allows them to describe new accessions and have greater intellectual and physical control. AtoM’s free download is certainly budgetfriendly, but archivists will do well to consider the organisational costs outlined herein, to budget extra time for possible contingencies, and to advocate strongly and early on in the project planning for the time and resources necessary to succeed.