Overview

Current Practice

What we need

Conclusion

Towards a Data Interchange Format for XAS and Related Spectroscopies Bruce Ravel Synchrotron Methods Group, Ceramics Division Materials Science and Engineering Laboratory National Institute of Standards and Technology & Local Contact, Beamline X23A2 National Synchrotron Light Source Workshop on Improving the Data Quality and Quantity for XAFS experiments 8 December 2011

Towards a Data Interchange Format for XAS and Related Spectroscopies N

1 / 12

Overview

Current Practice

What we need

Conclusion

Acknowledgements The people participating on the XASFORMAT mailing list: Matt Newville (CARS & APS)

Armando Sol´e (ESRF) (whose picture cannot be found on the web!) James Hester (ANSTO)

Gerd Wellenreuther (DESY)

Darren Dale (CHESS)

Towards a Data Interchange Format for XAS and Related Spectroscopies N

2 / 12

Overview

Current Practice

What we need

Conclusion

A single-spectrum XAS data standard Is this talk really going to be about such a boring topic?

Towards a Data Interchange Format for XAS and Related Spectroscopies N

3 / 12

Overview

Current Practice

What we need

Conclusion

A single-spectrum XAS data standard Is this talk really going to be about such a boring topic?

Why, yes! Yes, it is! A single XAS spectrum is a very useful unit of currency for a number of reasons: We send data to and receive data from our collaborators We publish individual spectra in journal articles We extract XAS spectra from large, complex, multi-spectral data sets We write web-based and desktop applications (for instance, a standards database) that traffic in single spectra We write data analysis software (ATHENA, for example) that needs to reliably import data that comes from many sources. Towards a Data Interchange Format for XAS and Related Spectroscopies N

3 / 12

Overview

Current Practice

What we need

Conclusion

Conventional XAS 1

In a conventional XAS experiment, we measure a sample somewhere between 2 and 10,000 scans, possibly requiring dead-time or other corrections

2

Some data processing happens to correct, calibrate, and/or align the data

3

Those scans are then merged into a single spectrum that becomes our unit of currency

A data interchange standard is about how we express the merged spectrum (i.e. the blue one).

Towards a Data Interchange Format for XAS and Related Spectroscopies N

4 / 12

Overview

Current Practice

What we need

Conclusion

We do a lot of fancy things these days

We do anomalous scattering experiments that yield energy-dependent scattering intensities.

We do imaging experiments that sample the heterogeneity of our samples.

We do non-resonant inelastic scattering experiments.

Tappero et al, New Phytologist 175:4, 641-654, (2007) doi:10.1111/j.1469-8137.2007.02134.x Ravel et al. PRB 60, 778-785 (1999) doi:10.1103/PhysRevB.60.778 Bergmann, et al. Chem. Phys. Lett. 369 184 (2003) doi:10.1016/S0009-2614(02)02003-1 Towards a Data Interchange Format for XAS and Related Spectroscopies N

5 / 12

Overview

Current Practice

What we need

Conclusion

We do a lot of fancy things these days

We do imaging experiments that sample the heterogeneity of our samples.

We do anomalous scattering experiments that yield energy-dependent scattering intensities.

...in this case, a spectrum from a particular spot on the sample.

...in this case, the XAS spectrum from a particular crystallographic site.

We do non-resonant inelastic scattering experiments. ...in this case, a low energy K edge measured by energy loss.

Tappero et al, New Phytologist 175:4, 641-654, (2007) doi:10.1111/j.1469-8137.2007.02134.x Ravel et al. PRB 60, 778-785 (1999) doi:10.1103/PhysRevB.60.778 Bergmann, et al. Chem. Phys. Lett. 369 184 (2003) doi:10.1016/S0009-2614(02)02003-1 Towards a Data Interchange Format for XAS and Related Spectroscopies N

5 / 12

Overview

Current Practice

What we need

Conclusion

Beamline data formats Every beamline has it’s own way of recording data Most use ASCII files, some use more complex data formats Each beamline has good reasons for doing things their own way

Towards a Data Interchange Format for XAS and Related Spectroscopies N

6 / 12

Overview

Current Practice

What we need

Conclusion

Beamline data formats Every beamline has it’s own way of recording data Most use ASCII files, some use more complex data formats Each beamline has good reasons for doing things their own way NSLS XDAC

Photon Factory and SPring-8

XDAC V1.4 Datafile V1 "au.b04" created on 3/15/09 at 1:28:27 PM on X-23A2 Diffraction element= Si (311). Ring energy= 2.80 GeV E0= 11919.00 NUM_REGIONS= 4 SRB= -200 -20 30 60 20k SRSS= 10 0.25 0.05k 0.05k SPP= 1 1 1 0.25k Settling time= 0.30

9809 KEK-PF BL12C G:hgcys-11.001 07.05.12 23:28 - 07.05.12 23:55 Hg:H2Cys 1:2 pH = 12.86, 100 mM, prep. at PF, 5 Ring : 2.5 GeV 348.8 mA - 342.8 mA Mono : SI(111) D= 3.13551 A Initial BL12C Transmission( 2) Repetition= 6 Param file : A:hgk16 energy axis(2)

mm Teflon, stirred 4 angle= 9.25969 deg Points= 818 Block = 6

Block Init-Eng final-Eng Step/eV Time/s 1 12049.00 12150.00 6.00 1.00 2 12150.00 12320.00 .35 1.00 3 12320.00 12400.00 1.00 2.00 4 12400.00 12600.00 2.50 3.00 5 12600.00 13040.00 4.00 3.00 6 13040.00 13260.00 5.00 4.00 Ortec(-1) NDCH = 3 ----------------------------------------------------------Angle(c) Angle(o) time/s 2 3 Energy I0 It IntTime Mode 0 0 1 2 11719.00294 18352.0000 15872.2222 1.0000 Offset 0 0 826.150 652.975 11728.99732 18380.0000 15934.2222 1.0000 9.44433 9.44420 1.00 252916 592687 11739.00126 18381.0000 15980.2222 1.0000 9.43958 9.43960 1.00 256349 604260 ... 9.43483 9.43480 1.00 256429 607846 Towards a Data Interchange Format for XAS and Related Spectroscopies Offsets= 122.00 85.78 0.00 Gains= 8.00 8.00 1.00 Au foil, NSLS X23A2, 20% Ar in Io and It with harmonic rejection mirror

N

Num 17 486 80 80 110 45

6 / 12

Overview

Current Practice

What we need

Conclusion

Problems with beamline formats They require additional processing in order to display µ(E ), including Conversion to energy Dead-time or other corrections

Ambiguous metadata, for instance How is the beamline identified? What consitutes a user comment? What describes the condition of the source or the beamline?

XAS data analysis software and other plotting software may have difficulty importing and interpreting the data This data is probably not appropriate for submission to a journal as supplemental material

Data interchange A standard for the interchange of µ(E ) data would address most of these concerns. Towards a Data Interchange Format for XAS and Related Spectroscopies N

7 / 12

Overview

Current Practice

What we need

Conclusion

Goals of a data interchange format

1

Establish a common language for transferring data between XAS experimenters, data analysis packages, web applications, journals and anything else that needs to process XAS data.

2

Increase the relevance and longevity of experimental data by reducing the amount of data archeology future interpretations of that data will require.

3

Enhance the user experience by promoting inter-operability among data acquisition systems, data analysis packages, and other applications.

4

Provide a mechanism for extracting and preserving a single XAS or XAS-like data set from a related experiment or from a complex data structure.

Towards a Data Interchange Format for XAS and Related Spectroscopies N

8 / 12

Overview

Current Practice

What we need

Conclusion

Goals of a data interchange format

1

Establish a common language for transferring data between XAS experimenters, data analysis packages, web applications, journals and anything else that needs to process XAS data.

2

Increase the relevance and longevity of experimental data by reducing the amount of data archeology future interpretations of that data will require.

3

Enhance the user experience by promoting inter-operability among data acquisition systems, data analysis packages, and other applications.

4

Provide a mechanism for extracting and preserving a single XAS or XAS-like data set from a related experiment or from a complex data structure.

Towards a Data Interchange Format for XAS and Related Spectroscopies N

8 / 12

Overview

Current Practice

What we need

Conclusion

Goals of a data interchange format

1

Establish a common language for transferring data between XAS experimenters, data analysis packages, web applications, journals and anything else that needs to process XAS data.

2

Increase the relevance and longevity of experimental data by reducing the amount of data archeology future interpretations of that data will require.

3

Enhance the user experience by promoting inter-operability among data acquisition systems, data analysis packages, and other applications.

4

Provide a mechanism for extracting and preserving a single XAS or XAS-like data set from a related experiment or from a complex data structure.

Towards a Data Interchange Format for XAS and Related Spectroscopies N

8 / 12

Overview

Current Practice

What we need

Conclusion

Goals of a data interchange format

1

Establish a common language for transferring data between XAS experimenters, data analysis packages, web applications, journals and anything else that needs to process XAS data.

2

Increase the relevance and longevity of experimental data by reducing the amount of data archeology future interpretations of that data will require.

3

Enhance the user experience by promoting inter-operability among data acquisition systems, data analysis packages, and other applications.

4

Provide a mechanism for extracting and preserving a single XAS or XAS-like data set from a related experiment or from a complex data structure.

Towards a Data Interchange Format for XAS and Related Spectroscopies N

8 / 12

Overview

Current Practice

What we need

Conclusion

XDI: XAS Data Interchange

XDI is an ad hoc format loosely based on the format of e-mail and structured in a way that looks like a familiar column-data file. The metadata is grouped into arbitrary (but useful) families with a dot (.) as a “namespace” separator. Application-specific information uses it’s own “namespace”.

# XDI/1.0 MX/2.0 # Beamline.name: APS 10ID # Beamline.edge-energy: 7112.00 # Beamline.d-spacing: 3.1356 # Ring.energy: 7.00 # Source.type: undulator a # Source.undulator-harmonic: 1 # Time.start: 2005-03-08T20:08:57 # Optics.crystal: Si 111 # Optics.harmonic-rejection: flat Rh-coated mirror # Column.1: energy eV # Column.2: mu # Column.3: i0 # MX.Num-regions: 1 # MX.SRB: 6900 # MX.SRSS: 0.5 # MX.SPP: 0.1 # MX.Settling-time: 0 # MX.Offsets: 11408.00 11328.00 13200.00 10774.00 # MX.Gains: 8.00 7.00 7.00 9.00 #/// # Fe K-edge, Lepidocrocite powder on kapton tape, RT # 4 layers of tape # exafs, 20 invang #--# energy mcs3 mcs4 mcs6 mcs5 6899.9609 -1.3070486 149013.70 6900.1421 -1.3006104 144864.70 6900.5449 -1.3033816 132978.70 6900.9678 -1.3059724 125444.70 6901.3806 -1.3107085 121324.70 (....etc....)

Towards a Data Interchange Format for XAS and Related Spectroscopies N

9 / 12

Overview

Current Practice

What we need

Conclusion

xasCIF

xasCIF would use the syntax of CIF, the Crystallographic Information File, and a dictionary of the scientific-field-specific ontology (i.e. the list of column names and their definitions) to encode the metadata.

data_copper_rt01 _xas_source synchrotron _xas_source_type "APS beamline 13ID, APS Undulator A" _xas_source_harmonic_rejection "rhodium-coated mirror" _xas_source_power 7.0 _xas_radiation_collimation None _xas_radiation_monochromator "Si 111" _xas_radiation_focusing yes _xas_exptl_start_time 2001-06-26T22:27:31 _xas_edge_element Cu _xas_edge_line K _xas_edge_energy 8980.0 _pd_spec_description "Cu metal foil" #<- from the Powder CIF dictionary _xas_exptl_ambient_temperature 293 _gse_sample_crystal_structure Cu _xas_detector_I0 "10cm / N2" _xas_detector_I1 "10cm / N2" loop_ _xas_processed_energy _xas_processed_mu _xas_processed_monitor 8779.0000 -1.3070486 149013.70 8789.0000 -1.3006104 144864.70 8799.0000 -1.3033816 132978.70 8809.0000 -1.3059724 125444.70 8819.0000 -1.3107085 121324.70 8829.0000 -1.3138152 119447.70 8839.0000 -1.3072055 119100.70 ...

Towards a Data Interchange Format for XAS and Related Spectroscopies N

10 / 12

Overview

Current Practice

What we need

Conclusion

Pros and cons

XDI

xasCIF

Pros

familiar format easy to import into many common XAS and plotting programs

Pros

separates syntax from ontology easily extensible, can handle more than one spectrum per file uses the successful CIF format

Cons

made up by a few guys with no particular authority no separation of syntax and ontology

Cons

structurally unfamiliar to XAS folk may not be imported as-is by many common XAS or plotting programs

Towards a Data Interchange Format for XAS and Related Spectroscopies N

11 / 12

Overview

Current Practice

What we need

Conclusion

Getting involved

While it is clear that a well-defined, broadly-agreed-upon data interchange format is needed, it is not yet clear what that format should be. This is an ongoing discussion

The XASFORMAT group is open to any and all interested parties. Please join us! http://millenia.cars.aps.anl.gov/mailman/listinfo/xasformat

Towards a Data Interchange Format for XAS and Related Spectroscopies N

12 / 12

Towards a Data Interchange Format for XAS and Related ... - GitHub

Mar 15, 2009 - We write web-based and desktop applications (for instance, a standards database) that traffic in single spectra. We write data analysis software ...

726KB Sizes 9 Downloads 415 Views

Recommend Documents

Keycode Format 1.0 Abstract 1 Format - GitHub
1.7 The free tag bit. In the specification thus far, the most significant bit ... plays a role in meeting constraint (C). Though the specialized encoding defined in X.

ZMQ Message Format - GitHub
Camera Images on an android smartphone are oriented in landscape mode. We assume however that the smartphone is used in portrait mode. So in order to ...

Towards an SMT Proof Format
where the side conditions are described using a simple functional programming language. This paper introduces the ... the input using program commands (adding support for mutual recursion would not be difficult). The recursive ..... 4 Preliminary Emp

Method and device for pictorial representation of space-related data
Feb 21, 2013 - all there arises a central storage requirement of 39.2 MBytes,. Which lies in the same order of magnitude as the storage requirement for ...

Method and device for pictorial representation of space-related data
Feb 21, 2013 - J.D. Foley et al., “Computer Graphics Principles and Practice”, ... Geographic Information System”, pub. Oct. 29, 1995 ...... master section. With a ...

Cigarette smoke and related risk factors in neurological ... - GitHub
Although nicotine was first recommended as a medical drug for the ... active and passive smokers revealed that a typical smoker takes 10 puffs on a ... occur with dementia were found to disrupt work, social activities, and relationships and .... quit

pdf-85\interchange-level-3-workbook-a-interchange-fourth-edition ...
fm.author_biographical_note3. Page 3 of 6. pdf-85\interchange-level-3-workbook-a-interchange-fourth-edition-by-jack-c-richards.pdf.

A PROPOSAL FOR COMMUNITY DRIVEN AND ... - GitHub
Dec 4, 2012 - 1. INTRODUCTION. Astronomical catalogues and databases are almost as old as ... In recent years, the internet allowed the open source com- munity to make ..... and tools that the computer science and open source commu- nity uses. .... d

Towards a Strategy and Results Framework for the CGIAR - CGSpace
Jun 3, 2009 - new crop variety, management system, or policy concept. ... population distribution in the future (map 1 and Annex A), ...... Developing a global commons of molecular tools and techniques to harness advanced science for.

Bradshaw and Bayes: Towards a Timetable for the ...
archaeological and scientific detail are inescapable if reliable chronologies are to be built. The dates ...... Of course, there may well be a limited number of ...

Runtime Array Fusion for Data Parallelism - GitHub
School of Computer Science and Engineering ... collective operations are implemented efficiently and provide a high degree of parallelism, the result of each ...

Bio data Format Bharat Electronics Limited Recruitment 2017 for ...
Bio data Format Bharat Electronics Limited Recruitment 2017 for Deputy Company Secretary.pdf. Bio data Format Bharat Electronics Limited Recruitment 2017 ...

Towards a Strategy and Results Framework for the CGIAR - CGSpace
Jun 3, 2009 - The Team is in regular communication by email and teleconferences. It held its first face- to-face meeting on May 3 and 4, 2009, in Washington, ...

RTCM SC 104 Data Format for Radio ... -
Oct 28, 2013 - INTERNET SOURCES. 7 %. PUBLICATIONS. 0 %. STUDENT PAPERS. 1. 2. 3. 4. 5. 6. 7. EXCLUDE QUOTES. OFF. EXCLUDE BIBLIOGRAPHY ON. EXCLUDE MATCHES. < 5 WORDS. RTCM SC 104 Data Format for Radio Communication of GPS. Data. ORIGINALITY REPORT.

Towards a Secure, Resilient, and Distributed Infrastructure for ... - EWSN
Runs on. Inspired by IEC 61131. Offers Values as Datapoints. Hardware. View. Software. View. Cyclic. Task. Component. Component. Composition. Component.

towards a caribbean open institute: data ... -
Open Data, Communications and Monitoring & Evaluation: Initial Guidelines . ..... Latin American Faculty of Social Sciences - Dominican Republic.

Towards hardware-driven design of low-energy algorithms for data ...
bined view towards data analysis algorithm design and ... for data analysis, and not only human resources, .... icon as algorithms and software are not ready.

Multi-modal emotion-related data collection within a ...
Next to this camera a laptop exists that displays the VR environment the pupil sees ... In detail, 14 pupils (5 boys and 9 girls) of age between 9 and 17 years were ...

Towards a global participatory platform - democratising open data ...
Towards a global participatory platform - democratising ... data, complexity science and collective intelligence.pdf. Towards a global participatory platform ...

The “Related Round Robin Rounds”-format: A new ... -
about inconsistent and unreplicable empirical results, multiple resort to ad-hoc excuses, and then finally people just sort of lose interest in the thing and pursue ...

Data Science and Machine Learning Essentials - GitHub
computer. Enter the following details as shown in the image below, and then click the ✓icon. • This is a ... Python in data science experiments in later modules.