Effective Metadata Management in Federated Sensor Networks

Hoyoung Jeung† Sofiane Sarni† Ioannis Paparrizos† Saket Sathe† Karl Aberer† Nicholas Dawes‡ Thanasis G. Papaioannou† Michael Lehning‡ † Ecole Polytechnique F´ed´erale de Lausanne (EPFL) ‡ WSL Institute for Snow and Avalanche Research (SLF)

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Federated Sensor Networks (FSN)

a local sensor network

internet

interconnected multiple sensor networks

share huge volumes of sensor & meta data over networks

FSN Example – Swiss Experiment
Collaborative environmental research project

FSN Example – Microsoft SensorMap
Web-based visualization of real-time sensor data – open for any sensor data publisher

FSN Example – Planetary Skin
NASA-Cisco climate change monitoring platform – $39 billion – online collaborative platform to process data from satellite, airborne and sea- and land-based sensors around the globe

Open Questions in FSN
Effective system design?
How to enable join processing of (sensor + meta) data?
How to capture dynamic metadata? – manual creation is infeasible in many application scenarios


How to search metadata? – volumes of metadata can also become very large – keyword search is not sufficient, visualization is important

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Global Sensor Networks (GSN)
Open source project by LSIR-EPFL – Infrastructure for managing data streams on the Web – Integrates data from various sources (local sensor networks)

Sensor Metadata Repository (SMR)
Portal, metadata database interface
Built on (semantic) mediaWiki – using RDF format to store semantics

System Overview

processed

queried

captured

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Processing Overview

Handling Sensor Redeployments
Step 1 - Users compose metadata in SMR

SMR offers a bulk-loading functionality of metadata

Handling Sensor Redeployments
Step 2 - SPARQL wrapper configuration in GSN – Server to query, fields to read, rate, query

Handling Sensor Redeployments
Step 3 - SPARQL query composition by GNS

an example of query results

Handling Sensor Redeployments
Step 4 - Caching SPARQL query results in GSN
Step 5 - Join processing – SPARQL query results + real time streams

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Why/How ?
Monitoring large numbers of sensor data streams – Manual approaches are infeasible


Processing overview

Dynamic Reporting of Dirty Data
Step 1 - Users define/implement an operation in GSN – e.g., model-based data quality measure

original data stream ↓ approximation using user-selected models ↓ detecting anomalies ↓ user confirmation: anomaly is an actual error?

Dynamic Reporting of Dirty Data
Step 2 – Configurations in GSN – linking to the operation, what metadata to be created


Step 3 – Dynamic metadata composition by GSN – Only when the user-defined operation is triggered – SensorML-based metadata template EPFL_Deployement multidata temperature 2010-05-17T14:00:41.826+02:00 2010-05-17T14:00:41.826+02:00 1.0


Step 4 – Metadata page creation in SMR – SMR creates a new page, containing the metadata received from GSN

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Motivations
The volume of metadata in FSN may become large
(Semantic) mediaWiki supports only keyword search – “report all sensors deployed in 2009 at region A”


Search results may contain too much relevant information – e.g., too many relevant metadata pages – which one is more relevant?


Visualization of sensor metadata is often necessary – “display all our sensors on a map”

Architecture of Query Processor

PageRank

Query Interface
Provides a rich set of search conditions – – – –

Multiple properties can be joined Geographical search Boolean operations (e.g., < >) can be combined Dynamic presentation of existing data values

Visualization of Query Results
Supports various types of visualization – e.g., bar, pie, graph, and map – aggregation of query results (e.g., sum, min, max and count) – search results export -> XML/CSV

Introduction Framework Overview Distributed Metadata Join Automated Metadata Generation Advanced Search Conclusions

Conclusions
Introduced applications & challenges in FNS
Proposed a framework for managing sensor & metadata – GSN + SMR architecture in a distributed setting


Described key components of the framework – distributed metadata join – automated metadata creation – advanced metadata search

Thank you