Proposal to discontinue TREC Federated Web Search in 2015 Track coordinators: • Djoerd Hiemstra (University of Twente, The Netherlands)* • Thomas Demeester (Ghent University - iMinds, Belgium) • Dolf Trieschnigg (University of Twente, The Netherlands) • Dong Nguyen (University of Twente, The Netherlands) • Ke Zhou (University of Edinburgh, UK)

Introduction Federated search is the approach of querying multiple search engines simultaneously, and combining their results into one coherent search engine result page. Building large-scale search engines increasingly depends on combining search results from multiple sources. A web search engine might combine results from numerous verticals: videos, books, images, scientific papers, shopping, blogs, news, recipes, music, maps, advertisements, Q&A, jobs, social networks, etc. Typically, the search results provided by each source differ significantly in the provided snippets, the provided additional (structured) information, and the ranking approach used. For on-line shopping, for instance, the results are highly structured, and price, bids, ratings and clickthrough rate are important ranking criteria, whereas for scientific paper search the number of citations is an important ranking criterion. Federated search also allows the inclusion of results from otherwise hidden web collections that are not easily crawlable. The FedWeb track provides specific research challenges in federated web search, together with a test collection that can be applied in many areas related to federated search, including aggregated search, distributed search, peer-to-peer search and meta-search engines [Shokouhi & Li 2011]. An appropriate dataset or TREC task reflecting an actual web environment was absent before the initiation of the FedWeb track. Before FedWeb, collections were created artificially from web collections [Hawking & Thomas 2005] or proprietary collections were used to evaluate distributed and aggregated search [Arguello et al. 2009]. In the 2013 edition of this track, the focus was on classical federated search challenges: (1) selecting which from multiple resources are most relevant for a specific topic, and (2) merging their results into a single result list. We gained experience in crawling and scraping a large variety of search engines and set up a framework for judging snippets and documents. In the 2014 edition of the track, we introduced vertical selection, requiring systems to think of diversity in terms of resource categories for the composition of the final result lists, hence targeting full aggregation of heterogeneous result sets.

Track goals and plans The track evaluates federated and aggregated search in a large heterogeneous setting using real search results of existing search engines. In its first year, the track evaluated two classical

distributed search challenges [Callan 2000]: resource selection, and result merging in this new setting. Two sample and a topic collections are available for training to the participants, called FedWeb12 (composed well before this track’s first edition), FedWeb13 and FedWeb14 (designed and crawled specifically for use in this track). Apart from studying resource selection and result merging in a web context, there are some new research challenges that readily appear, e.g.: ● How can we model a user’s vertical intent (or orientation towards specific verticals)? How should this be translated into a fully aggregated result page? ● How does the snippet quality influence results merging strategies? How well can relevance of results be estimated based on snippets only? Recently, [Zhou et al., WWW 2013] found that when assessing vertical relevance, a search task's vertical orientation is more important than the topical relevance of the retrieved results. In another contribution, [Zhou et al., 2012] proposed a general evaluation framework to explicitly model user vertical intent in evaluating aggregated search. For some of the emerging research questions for evaluating heterogeneous federated search, see also [Zhou et al., MUBE 2013]. In case other research want to continue the track, we believe re-using the FedWeb13 and FedWeb14 collection will be possible. This might bring an extra incentive for participants to take part again, and will provide potential new participants with better preparation possibilities. We already anticipated on re-using the collection, when designing the test topics. We in fact crawled the results for just over 500 test topics, from which 350 are now released to the participants, and from which 110 are used for evaluation. When choosing the original 500 topics, we anticipated on possible future research issues. For example, there is a balance between topics which hint at the targeted vertical (e.g., ‘buy bausack’, for shopping), and those that don’t (e.g., ‘tuning fork’, also for shopping).

Available test collections: FedWeb12, FedWeb13, FedWeb14 Currently, three federated web search test collections, including relevance judgements, are available. The collections are the result of joint work of Ghent University and the University of Twente [Nguyen et al. 2012]. The collection provides the top ten results of over a hundred search engines for fifty existing TREC Web track topics. The search results are provided in a standard format, and consist of the usual title, summary, and url, and optionally thumbnail images and attribute value pairs. The collection provides full pages for each result in several formats (web page only, full web page including style sheets and images, and web page screen shots). An extensive analysis of the FedWeb12 collection [Demeester et al. 2012] shows that interassessor consistency is similar to the consistency reported by [Voorhees 2000] for old TREC collections. We also found that there is a considerable chance that users miss relevant pages because of bad snippets. For general web search engines (Google, Yahoo, Bing, Baidu, etc.) 1 out of 5 snippets judged as “Unlikely” (i.e., the user would not click the result) points to a highly relevant page.

The FedWeb13 and FedWeb14 collections are larger than the FedWeb12 collection and consists of a sample and topic collection from about 150 search engines covering 24 categories including games, health, news and general web search engines. The sample collection consists of the results and pages of 2000 to 3000 queries. Each sample consists of up to 10 search results for a query, including the extracted title, description, link and thumbnail for each search result. Additionally, the linked documents were provided to the participants. The first 1000 sample queries are based on a fixed list of queries samples from the vocabulary of ClueWeb. The second 1000 queries are engine dependent and sampled from the vocabulary seen in the title and description of the search results for earlier queries. The sample collection have been released to build resource descriptions. For the FedWeb13 track, 200 topics have also been released, and another 150 for FedWeb14 (respectively 50 and 60 topics are judged). The topic collection consists of the search results (snippets and documents) for these 200 topics. For the result merging task the participants had to merge the results into a single optimal list. The collections include full judgments on the top 10 results from each search engine included in the test collection (each single resource is judged, both its snippets and its pages, independently). The collection does not suffer from scalability and reusability problems with the Cranfield paradigm, as summarized in [Pavlu et al.,2012].

Track history ● ●



2012 (full test collection crawled) 2013 (1st edition) ○ Task 1: Resource Selection (9 participants) ○ Task 2: Results Merging (6 participants) 2014 (2nd edition) ○ Task 1: Resource Selection (10 participants) ○ Task 2: Resource Selection (10 participants) ○ Task 3: Results Merging (6 participants)

Conclusion There are plenty of challenges and problems left for federated and aggregated search. A possible continuation of the Federated Web Search track might focus on full aggregated search, with the explicit composition of result pages (based on the crawled snippets). Unfortunately, the lack of funding and the fact that interest is only marginally increasing for this track, made us decide not to propose a 2015 edition of the Federated Web Search track for TREC.

Bibliography Jaime Arguello, Fernando Diaz, Jamie Callan, Jean-Francois Crespo. Sources of evidence for vertical selection. In: Proceedings of SIGIR 2009. Jamie Callan (2000). Advances in Information Retrieval., chapter 5 Distributed information retrieval, pages 127–150. Kluwer Academic Publishers,.

Thomas Demeester, Dong Nguyen, Dolf Trieschnigg, and Djoerd Hiemstra (2012). What Snippets Say About Pages in Federated Web Search. In Proceedings of the 8th Asia Information Retrieval Society Conference (AIRS), 2012. David Hawking and Paul Thomas. Server Selection Methods in Hybrid Portal Search, In: Proceedings of SIGIR 2005. Dong Nguyen, Thomas Demeester, Dolf Trieschnigg, and Djoerd Hiemstra (2012). Federated Search in the Wild: The Combined Power of over a Hundred Search Engines. In Proceedings of the 21st ACM Conference on Information and Knowledge Management (CIKM). (to appear) Milad Shokouhi and Luo Si (2011). Federated search. Foundations and Trends in Information Retrieval, 5(1):1–102. Ellen Voorhees (2000).: Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36, 697–716 K. Zhou, R. Cummins, M. Lalmas and J. M. Jose. Which Vertical Search Engines are Relevant? Understanding Vertical Relevance Assessments for Web Queries, in International World-Wide Web Conference (WWW 2013), Rio de Janeiro, Brazil, 2013. K. Zhou, T. Sakai, M. Lalmas, Z. Dou and J. M. Jose. Evaluating Heterogeneous Information Access (Position Paper), SIGIR 2013 Workshop on Modeling User Behavior for Information Retrieval Evaluation (MUBE 2013), Dublin, Ireland, 2013. K. Zhou, R. Cummins, M. Lalmas and J. M. Jose. Evaluating Aggregated Search Pages, 35th ACM Conference on Special Interest Group on Research and Development in Information Retrieval (SIGIR 2012), Portland, 2012.

Proposal to discontinue TREC Federated Web Search in 2015

Federated search is the approach of querying multiple search engines ... the provided additional (structured) information, and the ranking approach used. ... task's vertical orientation is more important than the topical relevance of the retrieved ...

88KB Sizes 2 Downloads 251 Views

Recommend Documents

Proposal to discontinue TREC Federated Web Search in 2015
combining their results into one coherent search engine result page. Building ... the provided additional (structured) information, and the ranking approach used.

Overview of the TREC 2014 Federated Web Search Track
collection sampled from a multitude of online search engines. ... course, it is quite likely that reducing the admin- istration cost ... the best medium, topic or genre, for this query? ..... marketing data and analytics to web pages of many enter-.

Overview of the TREC 2014 Federated Web Search Track
evaluation system (see Section 4), and the remaining 50 are the actual test topics (see ... deadline for the different tasks, we opened up an online platform where ...

Enabling Federated Search with Heterogeneous Search Engines
Mar 22, 2006 - tional advantages can be gained by agreeing on a common .... on top of local search engines, which can be custom-built for libraries, built upon ...... FAST software plays an important role in the Vascoda project because major.

Enabling Federated Search with Heterogeneous Search Engines
Mar 22, 2006 - 1.3.1 Distributed Search Engine Architecture . . . . . . . . . . 10 ..... over all covered documents, including document metadata (author, year of pub-.

Enabling Federated Search with Heterogeneous ...
Mar 22, 2006 - This report analyses Federated Search in the VASCODA context, specifically fo- cusing on the existing TIB Hannover and UB Bielefeld search infrastructures. We first describe general requirements for a seamless integration of the two fu

A Plugin Architecture Enabling Federated Search for ...
nary evaluation results show ranking results equal to a centralized setup. ... Unfortunately, simply indexing all documents in a single search engine ... Good surveys on distributed information retrieval problems and solutions can be found.

Entity Recommendations in Web Search - GitHub
These queries name an entity by one of its names and might contain additional .... Our ontology was developed over 2 years by the Ya- ... It consists of 250 classes of entities ..... The trade-off between coverage and CTR is important as these ...

TREC CPN.pdf
Page 1 of 1. TEXAS REAL ESTATE COMMISSION. P.O. BOX 12188. AUSTIN, TEXAS 78711-2188. (512) 936-3000. THE TEXAS REAL ESTATE COMMISSION ...

2015-Roadmap-to-Web-Accessibility-in-Higher-Education.pdf ...
2015-Roadmap-to-Web-Accessibility-in-Higher-Education.pdf. 2015-Roadmap-to-Web-Accessibility-in-Higher-Education.pdf. Open. Extract. Open with. Sign In.

DREAM PROPOSAL FORM for OCTOBER 2015 INTAKE
factors is key to the application of drought risk mitigation in local water .... Justin Sheffield and his Princeton group on droughts as part of her INTENSE ERC ...

DREAM PROPOSAL FORM for OCTOBER 2015 INTAKE
Predicting hydrological drought risk in Europe using big data of natural and socio-economic factors. Key words hydrological drought risk, prediction, combining ...

Delegation Proposal IMO 2015 BAHASA.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Delegation ...

AVEC 2015 Proposal for ACM MM - SSPNet
Imperial College London, UK [email protected]. Important Dates. Challenge ... will be based on relevance to the workshop, novelty, and technical quality.

DREAM PROPOSAL FORM for OCTOBER 2015 INTAKE
2. - “Tools for developing, managing and analysing 'Big Data', to understand risk better, ... www.bafg.de/GRDC/EN/Home/homepage_node.html), International ...

AVEC 2015 Proposal for ACM MM - SSPNet
University of Nottingham, UK ... Roddy Cowie. Queen's University Belfast, UK ... the type of data that both multimedia and human-machine/human-.

query formulation in web information search
(search system, computers, or web) expertise improves performance in search tasks. In query formulation, .... questionnaire and filled in the questionnaire independently. ... task are included in the query, the query is defined as broad. In some ...

Database Selection and Result Merging in P2P Web Search
document ranking measures, to address the specific issues of P2P Web search. ..... Conference on Research and Development in Information Retrieval ACM SIGIR '99, .... ference on Database Systems for Advanced Applications DASFAA '97, ...

Context disambiguation in web search results
government reports of “Mumbai blasts 1993” and. “Mumbai blasts 2003” in the same context or cluster ... IBM's Textract to identify and extract relations in text.

Social annotations in web search - Research at Google
diversity, and personal relevance of social information on- line makes it a .... Beyond web search and social question-answering systems, simply displaying or ...

Challenges in Running a Commercial Web Search Engine - MSU CSE
As of today. – Users give a 2-4 word query. – SE gives a relevance ranked list of web pages. – Most users click only on the first few results. – Few users go below ...

Perception and Understanding of Social Annotations in Web Search
May 17, 2013 - [discussing personal results for a query on beer] “If. [friend] were a beer drinker then maybe [I would click the result]. The others, I don't know if ...

Identifying Users' Topical Tasks in Web Search
cool math for kids. 1. Table 1: An example of tasks in one ...... tic Gradient Descent, L_BFGS (Limited-memory Broyden Fletcher. Goldfarb Shanno) and SMO ...

Using English Information in Non-English Web Search
The leading web search engines have spent a decade building highly specialized ..... This yields the optimization problem from step (3) of Figure 2. Qin et al.