Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

Automated Question Answering From Lecture Videos: NLP vs. Pattern Matching Jinwei Cao University of Arizona [email protected]

Dmitri Roussinov Arizona State University [email protected]

José Antonio Robles-Flores Arizona State University / ESAN [email protected]

Jay F. Nunamaker, Jr. University of Arizona [email protected]

Abstract This paper explores the feasibility of automated question answering from lecture video materials used in conjunction with PowerPoint slides. Two popular approaches to question answering are discussed, each separately tested on the text extracted from videotaped lectures: 1) the approach based on Natural Language Processing (NLP) and 2) a self-learning probabilistic pattern matching approach. The results of the comparison and our qualitative observations are presented. The advantages and shortcomings of each approach are discussed in the context of video applications for e-learning or knowledge management.

1.

Introduction

Learning is an important way of transferring knowledge, and effective learning is highly desired in the current knowledge-based economy. In recent years, advances of information technology have affected learning dramatically. Thousands of online courses, including degree and certificate programs, are now offered by universities world-wide. It was predicted that U.S. online education student enrollment will top one million by 2005 [1]. The fundamental value proposition of online learning – access to quality education or training freed from the boundaries of time and location – is growing with the demand for higher education and professional training in the United States and worldwide. In current online education or training programs, multimedia learning materials such as videotaped lectures and PowerPoint slides are commonly

provided as a way to help learners engage in the learning process. For example, in online courses at Stanford University, a video of an instructor is synchronized with his/her PowerPoint slides (http://scpd.stanford.edu/scpd/students/onlineClass.ht m). Such multimedia lectures are considered to be able to give students a perception of listening to a lecture in real-time, make them pay more attention to the learning task, and help them retain more information through vivid and rich presentations [2]. However, simply watching a mentor talking in a lecture video is still quite different from learning with a real mentor. An important factor of learning – “learning interactions” – is usually missing in multimedia online lectures and often results in higher dropout rates as compare to classroom learning [3]. For example, a student in classroom learning can ask questions and get them answered by the instructor. Unfortunately, in current commercial online learning systems, students can only watch the instructor talking in the video. They cannot ask questions unless the instructor is online at the same time with a chat program or a video conference system. In the meanwhile, recent advances in Natural Language Processing (NLP) and AI in general have approached to the point where people may converse with a machine in natural language to get answers to their questions. Several known “futurists” believe that computers will reach capabilities comparable to human reasoning and understanding of languages by 2020 [4]. Automated Question Answering (QA), the technology that locates, extracts, and represents a specific answer to a user question expressed in natural language, is now being studied very actively by researchers and practitioners. We believe that automated QA technologies may be applied to online

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

1

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

learning systems to make the learning process more interactive. However, the majority of studies have been focusing on QA from a collection of text documents [10]. In order to answer student questions we need QA technology that works on a collection of video files. Nevertheless, research on video-based QA is just initiated, with a focus on news video collection [10]. The algorithms used in these QA systems, such as video genre analysis, cannot be directly applied to lecture videos because the difference in genre. For example, lecture videos usually have very few scene changes (e.g. only a “talking instructor” on the screen); speeches in lecture videos are typically unscripted and spontaneous; and more importantly, speech in lecture videos frequently involves domainspecific terminology (e.g. UTP as “Untwisted Pairs” in the domain of data communication) which cannot be found from a general knowledge source such as WordNet [11]. In our study, we have explored the applicability of two text-based automated approaches, namely natural language processing (NLP) based approach and pattern based approach (PBA), for the purpose of automated question answering from lecture videos. Our motivation was to first see if providing such a feature within a proof of concept prototype is feasible. Second, we wanted to compare two QA techniques which are inherently different on the set of test questions and correct answers to them. Our prototype system converts the sound track into transcripts by using speech recognition software and applies several enhancement methods including phonetic-based transcript error correction and external domain knowledge. Next section reviews the explored QA approaches, followed by the section describing our transcribing process, then by the section on our comparison tests.

2.

Technology involved

Automated Question Answering (QA) is the technology that locates, extracts, and represents a specific answer to a user question expressed in natural language. A QA system would take as input a question such as “What is mad cow disease?” and produces the answer, e.g. “Mad cow disease is a fatal disease of cattle that affects the central nervous system. It causes staggering and agitation.” Modern Question Answering (QA) technologies rely on many components, including document retrieval, semantic analysis, syntactic parsing and explanation generation. QA promises an important new way of information access for many types of users, including novices in technology, a natural step beyond the keyword query and document retrieval characteristic of today’s

information quests, such as those provided by current web search engines. The TREC question answering evaluation [5] is the motivating force behind a recent surge in question answering research. Systems participating in TREC have to identify exact answers to “factual questions” (who, when, where, what, etc.). Most of TREC QA systems are designed based on techniques from natural language processing, information retrieval and computational linguistics. For example, Falcon [6], one of the most successful systems, is based on a prebuilt hierarchy of dozens of semantic types of expected answers, complete syntactic parsing of all potential answer sources, and automated theorem proving to identify the answers. In addition to “deep” linguistic approaches, QA researchers have explored more “shallow” approaches ground on pattern matching successfully used earlier for information extraction [7]. Pattern matching systems performed well in recent TREC QA competitions: the system from InsightSoft [7] won 1st place in 2002 and 2nd place in 2001. Roussinov & Robles [8] combined the redundancy based approach suggested in [9] with the automatically learned patterns suggested earlier (e.g. in [16]) into a single probabilistic completely trainable approach. The system achieved comparable performance with one of [9]. Its strong advantage over other techniques that it does not require any manually developed rules or substantial linguistic resources. We provide a detailed introduction of the two different approaches in the sections 2.1 and 2.2. An attempt of applying NLP-based QA approach to a collection of videos is described in 2.1. In the section 2.3, we present a general prototype architecture that is particularly designed for QA from lecture videos. Either QA approach can be used in this architecture. The major innovation of this prototype is the incorporation of information extracted from PowerPoint slides in the QA process.

2.1.

NLP based approach

As we mentioned earlier, most of modern Question Answering systems are designed based on techniques from natural language processing and information retrieval. They typically consist of the following three steps [12]: 1. Question Understanding includes recognizing the type of a question. E.g. “Who is the CEO of IBM” expects the answer to be a person. 2. Document Filtering uses keywords in the question as query and some query expansion mechanisms (e.g. morphological or lexical, using

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

2

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

WordNet) to perform retrieval. Only the retrieved documents are processed to extract candidate answers. 3. Answer Extraction performs a shallow or deep parsing of the returned documents to detect entities of the same type as the answer. Those entities are later treated as candidate answers and assigned scores based on the relevance between the processed query and sentences in the documents extracted in step 2. (E.g. “Samuel Palmisano”) Zhang and Nunamaker [13] applied a natural language based QA approach to a collection of transcribed videos. In that approach, the transcripts of the manually segmented video clips are treated as text documents, and a similar three-step approach is used to identify the answers. However, unlike some TREC QA systems such as Falcon [6], it uses a templatebased approach for question understanding and answer extraction. Such a template-based approach does not rely on complicated deep semantic analysis such as automated theorem proving in Falcon, and is believed to be more suitable for spontaneous speech text. The approach uses a parser called Conexor iSkim [14]. Major verbs, nouns, noun phrases, named entities in the question are extracted from the iSkim output. Their synonyms are found from the WordNet dictionary. A query is formed using the original words, their synonyms and the named entities. The answer type of the question is also derived and the question is filled into a question template with nine slots, including answer type, question focus, person, organization, governor, objects, number, time, and location (Figure 1) [13].

Figure 1. Question template. From [13]. The basic query is then sent to a Boolean information retrieval engine. The search results are processed to extract answers. Sentences in the retrieved documents are then parsed using iSkim and transformed into sentence templates (ST) in a similar form as the question template (QT). Similarity

between the QT and the ST is calculated based on the combination of the following three factors [13]. Matched_Slots_Score (MSS) compares the slot values of QT with STs. Same_WordSequence_Score (SWS) computes the number of keywords in the question that appear in the same sequence in the current sentence. AnswerType_Found_Score (AFS) checks if either Answer Type or Question Focus of the question is found in the current sentence. Finally, a sliding-window method is used to calculate the total similarity between the question and each five-sentence window in the document. The highest score is taken as the relevancy score of the document and the top relevant documents are returned. This natural language based approach works well in a limited domain such as lecture videos. In this scenario, the interpretations of questions are clear and the parsing and understanding of sentences is feasible in real time since the set of retrieved documents is small.

2.2.

Pattern based approach

A pattern based QA approach used in our study has been developed by Roussinov & Robles [8] while extending prior research [9, 16]. While searching for an answer to a question (e.g. “Who is the CEO of IBM?”) their approach looks for matches to certain patterns. For example “The CEO of IBM is Samuel Palmisano.” matches the pattern “\Q is \A .” where \Q is a question part (“The CEO of IBM”) and \A = “Samuel Palmisano“ is the text that forms a candidate answer. The approach automatically creates and trains up to 200 patterns for each type of a question (examples of types of questions are what is, what was, where is, etc.) based on a training set of given question-answer pairs. Through training, each pattern is assigned the probability that the matching text contains the correct answer. This probability is used in ranking the candidate answers. \A, \Q, \p (punctuation mark), \s (beginning of a sentence) and * (wildcard that matches any words) are the only special symbols currently used in their pattern language. Figure 2 summarizes their approach.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

3

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

What is Calvados?

What is … Who was… Where is… …

Natural Language Questions

Find patterns

PATTERN: \Q is \A Where: \Q = (“Calvados”)

Using the Q&A System

Answers to original questions

QUERY: “Calvados” NEAR “is”

ANSWER: \Q is \A: “calvados” is “a dry apple brandy” Real Questions

Rewrite Query Training mechanism

\A is now: “a dry apple brandy”

Extracting the answer to the original question

Using Commercial Search Engines: Mining for answers

Triangulation: Finding most probable answers ...CALVADOS Often used in cooking chicken , pork and veal dishes , Calvados is a dry apple brandy made in Calvados-located in the Normandy region of northern France ....

… Multiple possible answers … sometimes conflicting

Figure 2. The general Web QA approach. From [8]. Answering the question “Who is the CEO of IBM?” demonstrates the steps of their algorithm: Type Identification. The question itself matches the pattern who is \Q ?, where \Q = “the CEO of IBM” is the question part and “who is” is the type identifier. Query modulation (although present in the original approach for Web QA) is not necessary in our case since our system just scans each sentence in the transcripts. Answer Matching. The sentence “Samuel Palmisano recently became the CEO of IBM.” would result in a match and produce a candidate answer “Samuel Palmisano recently”. Answer Detailing produces more candidate answers by forming sub-phrases from the initial candidate answers. The sub-phrases do not exceed 3 words (not counting “stop words” such as a, the, in, on) and do not cross punctuation marks. In our example, the detailed candidate answers would be Samuel, Palmisano, recently, Samuel Palmisano, Palmisano recently.

When fewer than 5 matches are found, the system resorts to the “fall-back” approach by simply retrieving the segments according to their relevance to the question. For this, the questions and text segments are represented as normalized to unit length vectors in the classical vector space model [15] and the dot product is used as the relevance score.

2.3.

Lecture video specific issues

Of the two QA approaches described above, only the NLP one has been applied to video collections before and it completely relies on the transcribed text of the video. In our study, we tried to apply each of the approaches to video collections. Figure 3 presents the overall prototype architecture of our video QA system. Specifically, we have added phonetic-based transcript error correction and enhanced video transcripts with the text from the PowerPoint slides that accompanied them, thus treating the slides as additional domain knowledge. Because teachers typically use PowerPoint

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

4

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

slides to outline key concepts during their lectures and usually provide students the slides in electronic form, it is reasonable to use the text extracted from these slides as a resource of domain knowledge to improve the answer extraction performance.

3.

Evaluation experiment

We compare our two approaches using the archived videos of a professional training course

Question Question Processing Parsing & Template Matching

Answer Extraction

QT Query

Document Retrieval

Relevant Transcripts

Segment Extractor

Video

Speech Recognition

Video Answers Transcript Correction

Lecture Domain Concepts

Transcript Corpus

Video Preprocessing Figure 3. Overall prototype architecture.

The text transcripts of lecture videos are generated by a speaker-independent speech recognition tool: Virage VideoLogger® (http://www.virage.com). This tool also generates time stamps that synchronize the video stream with the transcribed text at word level. However, speech recognition errors could incur in the video transcripts, and these errors may greatly reduce the accuracy of retrieval. We solve this problem by doing transcript correction based on phonetic matching, a method described in [10]. A list of domain concept words is developed based on content in the domain knowledge base. Particularly, these domain concept words are extracted from the text of the PPT slides that are associated with the video. Words in transcripts are converted to phonetic sounds and are compared to the phonetic sequence of the words in this list. Similar sound words in the transcripts are changed to the one in the word list. Finally, the corrected video transcripts are indexed and stored in a database. Once the text answers are identified using any of the QA approaches, the links to the video segments that contain them are presented to the user in the rank order of decreasing relevance and can be played upon request.

(“Deception Detection”). There are two lectures (videos) in this collection, and they are pre-segmented into 100 short clips (segments). This is a relatively small collection, but human generated transcripts are available for comparison purpose. We are now in the process of developing a larger video lecture corpus for future evaluation. To evaluate the two approaches, a PhD student who is an expert in the domain has created a set of 30 questions and manually identified the best answer segment for each of the questions. We assumed that there is only one correct answer for each question. Although various metrics have been explored, we used mean reciprocal rank of the answer (MRR) according to the following formula [12] to evaluate the two approaches.

MRR =

# questions 1 1 ¦ max(i) answer rank # questions i =1 i

Where answeri rank is the rank of the first correct answer for the question i, and if the answer is found at multiple ranks, the best rank will be used. If no relevant answer is found in the top 5, the score for that particular question is zero. The highest MRR score is 1 and the lowest is 0. For example, MRR score of 0.5

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

5

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

can be roughly interpreted as “in average” the correct answer being the second answer found by the system. MRR is closely related to the average precision measure used extensively in document retrieval. The drawback of this metric is that it is not very sensitive since it only considers the first correct answer, ignoring what follows.

3.1 NLP based approach For the natural language based approach, we have compared the results using three different sets of transcripts, and for each set, we compare the method of using PPT slides in answer extraction to the one without using PPT slides. Our results are listed in Table 1. We conclude that the NL based approach works best for human generated transcripts plus the help from PPT slides. QA based on the transcripts directly generated by the speech recognition software has even higher MRR than QA based on human generated transcripts only. This is encouraging and shows that it is possible to get satisfying retrieval results without the time-consuming human correction by just using PPT slides. However, it is surprising that although the MRR for QA on transcripts with automatic correction is higher than QA on transcripts without error correction, using PPT slides does not increase the MRR as much as on the transcripts without error correction. One possible explanation is that we use the same set of PPT slides in transcript correction so there is not much extra knowledge in answer extraction to boost the performance. Also, the transcripts may be over-corrected and new errors may be introduced. More test questions and larger test collections may be needed in future to arrive at more reliable conclusions. Table 1. Results of our NLP based approach

MRR Transcript without error correction with error correction human generated

Transcript Only 0.423

Transcript + PPT Slides 0.524

0.476

0.517

0.511

0.561

3.2 Pattern based approach Our pattern based QA (PBA) approach produced MRR of 0.622 on “Transcripts Only.” We did not run

PBA on “Transcripts + PPT Slides” but instead concentrated our efforts on trying to explain the difference in performance over NLP based approach and exploring what can further improve either approach. We present here our informal discussion, leaving more formal experiments for future. After analyzing our log files, we observed that only 6 questions were answered by pattern matching and the rest were by our “fall-back” (vector space retrieval) approach. Since the NLP based approach does not resort to “fall back” retrieval this may explain why it suffered some loss in performance. Thus, our study suggest to the practitioners that it is desirable to have a “fall back” approach in a QA system, which should automatically take over in situations that there are no good matches identified by QA technology. Also, every applied QA system should be tested against simple retrieval as a baseline in order to verify that it provides any value added before investing any resources in it. We further explored the possibility of improving answering accuracy. However, on this particular data set that did not seem to be feasible due to the following consideration. When we run the “fall-back” approach only (simple retrieval instead of QA) the answers were not perfect (reciprocal rank not equal to 1) only for 6 questions out of 30. By inspecting those “trouble-making” questions, we concluded that none of them would be handled by the pattern based QA anyway since they were not factual (e.g. Yes/No questions) or relatively “complex” (e.g. What are the differences between linguistic diversity and linguistic complexity?). Our pattern based QA system was trained on TREC [5] questions from which the types presented within our test collection were drastically different. Specifically, there were 3 Yes/No questions (e. g Is human good at detecting deceptions?), 4 definitional questions (e.g. What is truth bias?), 7 “how” questions (e.g. How many sub-categories of indicators are related to emotion?) and 2 “why” questions. The rest of the question would approximately fall into definitional category (“what is”, “what are”, etc.) when at least a sentence long answer is expected. Our next step was to try our pattern based QA system to get the answers to the questions from the Internet first, then “project” the answers on our transcripts similarly as it was done by several TREC participants [5]. We were posed to see if the projection may improve the answering accuracy. Our first run of the questions through the Internet resulted in the answers that were either too general or entirely off-topic. This was not surprising since the questions were asked out of context (e.g. “What do unusual details indicate?”). We modified our QA system to add the user-specified “context string” (“Deception Detection” in our experiment) to all the

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

6

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

search engine queries similarly to as it was done in [17]. We informally observed some improvement due to this but the answers were still not likely to be very helpful in improving retrieval accuracy. For this reason, we decided not to implement and test the answer projection component at the moment. Again, the questions not being factual was obviously the primary reason of the inadequate performance of our pattern based QA system in that particular setting. The secondary reason was that the transcripts topic was very narrow and the answers to some of the questions (e.g. How can deceivers manipulate information in computer-mediated communication?) were hard to find on the Web and summarize even for a human. Nevertheless, for approximately 70% of the questions we were eventually able to find web pages that would provide the answers. It is interesting to note, that the system was still able to come up with some answers from the Internet that were not in the transcripts (e.g. “What is gaze aversion?” – “Gaze aversion, like other commonly held stereotypes about liars, isn’t correlated with lying at all …”), thus the Internet QA system can be used as complementary to the lecture material.

4.

Conclusions, limitations and future research

Although the overall accuracy of answering was encouraging, the size of the test collection and types of the questions were not adequate to test reliably if Question Answering Technology (either Natural Language or Pattern Matching based) would provide any value added over simple retrieval in the setting of our experiment. Thus, more tests again seem to be needed in future. Since both NLP and Pattern-matching, approaches present limitations; we plan to combine the approaches in the future. NLP seems to work very well in the interpretation of questions that are related to the specific domain (a video digital library, a specific database, etc.). On the other hand, patternmatching performs well only when there is a large set of documents with redundancy (such as very large digital libraries, large intranets or the entire WWW) but it has reduced advantages when the domain is smaller. Since students do not always only need to recall what was on the lectures but sometimes need to go beyond the lectures, we believe that using open domain QA on the Web as part of the process can provide a more thorough solution. The application described can also be extended for different business applications: as a business intelligence tool, knowledge management tool or a

knowledge source for integrating multiple database schemas.

5.

References

[1] Gallagher, S. Online Distance Education Market Update: A Nascent Market Begins to Mature, Eduventures. [2] Agius, H.W., and Angelides, M.C. (1999). Developing knowledge-based intelligent multimedia tutoring systems using semantic content-based modelling, Artificial Intelligence Review, 13, pp 5583. [3] O'Connor, C., Sceiford, E., Wang, G., and FoucarSzocki, D. (2003). Departure, Abandonment, and Dropout of E-learning: Dilemma and Solutions, TechLearn 2003 Conference. [4] Lempert, R. J., Popper, S. W., Bankes, S. C. (2003). Shaping the next one hundred years: new methods for quantitative, long-term policy analysis, RAND, Santa Monica, CA. [5] Voorhees, E. M., Tice, D. M. (2000) Building a question answering test collection, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, Athens, Greece. [6] Harabagiu, S., et al. (2000) FALCON: Boosting Knowledge for Answer Engines. In Proceedings of the Text Retrieval Conference (TREC-9). [7] Soubbotin, M., & Soubbotin, S. (2002). Use of patterns for detection of likely answer strings: A systematic approach. In the Proceeding of TREC 2002. [8] Roussinov, D., and Robles, J. (2004b). Web Question Answering: Technology and Business Applications. In the proceedings of 2004 American Conference on Information Systems. August 6 – 8, New York, NY. [9] Dumais, S., Banko, M., Brill, E., Lin, J., and Ng, A. Web Question Answering: Is More Always Better? (2002) Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland. [10] Yang, H, Chaisorn, L., Zhao, Y., Neo, S., and Chua, T. (2003). VideoQA: Question Answering on News Video. In Proceedings of the ACM conference on Multimedia (Multimedia’03), Berkeley, CA, November 2-8. [11] Miller, G. (1990). WordNet: An On-line Lexical Database. In International Journal of Lexicography, 3, 4.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

7

Proceedings of the 38th Hawaii International Conference on System Sciences - 2005

[12] Voorhees, E.M. (1999). The TREC-8 Question Answering Track Report. Eighth Text REtrieval Conference (TREC-8), Gaithersburg, MD.

[15] Salton, G. and McGill, M.J. (1983). Introduction to Modern Information Retrieval. New York. McGraw-Hill.

[13] Zhang, D., and Nunamaker, J. (2004). A Natural Language Approach to Content-Based Video Indexing and Retrieval For Interactive E-Learning, IEEE Transactions on Multimedia, 6, 3.

[16] Ravichandran, D., & Hovy, E. (2002) Learning surface text patterns for a question answering system. In Proceedings of the 40th Annual Meeting of the ACL, pages 41-47.

[14] Voutilainen, A. (2000). Helsinki taggers and parsers for English. In J. M. Kirk (Ed.) Corpora Calore: Analysis and Techniques in Describing English. Rodopi, Amsterdam & Atlanta.

[17] Roussinov, D., and Zhao, L., Automatic Discovery of Similarity Relationships through Web Mining, Decision Support Systems, 35, 2003, pp. 149166.

0-7695-2268-8/05/$20.00 (C) 2005 IEEE

8

Automated Question Answering From Lecture Videos

Proceedings of the 38th Hawaii International Conference on System Sciences ... Question Answering systems are designed based on .... Calvados is a dry apple.

466KB Sizes 0 Downloads 232 Views

Recommend Documents

Building Effective Question Answering Characters
conference (see Appendix C for a photograph of .... set of answers which we can call documents in ac- ..... tions (i.e., when and how to activate the micro-.

Knowledge Representation and Question Answering
For example, an intelligence analyst tracking a particular person's movement would have text ..... by means of a certain kind of transportation, or a certain route”).

Question Answering and Generation
Type of knowledge: Is the knowledge organized as a semantic network, plan, causal structure, ..... Dr. Piwek pioneered research on Text-to-Dialogue (T2D) generation based on ... templates, which offer limited flexibility and content coverage.

Knowledge Representation and Question Answering
Text: John, who always carries his laptop with him, took a flight from Boston to ..... 9 where base, pos, sense are the elements of the triple describing the head ...

Online Question Answering System
Consequently, this additional feature brings dictionary functionality into .... between the question and the available question and answer pair in the database.

Building Effective Question Answering Characters
We are developing virtual characters with similar capabilities. ... Automatic question answering (QA) has been ..... phone and how to adjust the microphone posi-.

Open-domain Factoid Question Answering via Knowledge Graph Search
they proved being beneficial for different tasks in ... other knowledge graph and it increases their domain ... ciated types in large knowledge graphs, translating.

Text-to-text generation for question answering
In this process, we use a graph-based model to generate coherent answers. We then apply sentence fusion to combine partial an- swers from different sources ...

Description of SQUASH, the SFU Question Answering ... - CiteSeerX
... in order to participate in the 2005 Document Understanding Conference ... ing the Rouge score, while the next stage which we call the Editor module, focuses ...

How to evaluate your question answering system every ...
whether or not node labels matter, and how partial credit should be ... Which company created the internet browser ... This range in accuracy is small because the vast ..... back of the card). ... Processing conference in 2000, and in the reading.

SQAD: Simple Question Answering Database - raslan 2014 - NLP ...
Database, SQAD, contains 3301 questions and answers extracted and pro- cessed from the .... 3.3 Mistakes in Morphological Analysis. In case of unknown ...

QuASM: A System for Question Answering Using Semi ...
Question answering (QA) systems augment IR systems by taking over the ..... The second database contained the content files with tables extracted and prose ...

SQAD: Simple Question Answering Database - raslan 2014 - NLP ...
as the IBM's Watson [1,2] that had a big success in popular television compet- itive show ... SQAD contains 3301 records with the following data fields (see Figure 1 .... of Education of CR within the LINDAT-Clarin project LM2010013 and by the.

Paraphrase for Open Question Answering: New ...
velop the latter as an alternative or in combination with the former. One major challenge for question answering from a KB is the many ways in which relations ...

Trajectory Matching from Unsynchronized Videos
casion is matching features of adjacent images in a video se- ... ests on studies of multiple videos. ...... European Conference on Computer Vision, 1:430–443,.

How to evaluate your question answering system every ...
answering system allows the user to pinpoint information in this collection. ..... the Graduate Management Assessment Test's Analytical. Writing Assignment ...

QuASM: A System for Question Answering Using Semi-Structured Data
document contains suitable data for answering questions. Many of the documents ... Permission to make digital or hard copies of all or part of this work for personal or classroom use ...... Andre Gauthier for the disk space. Any opinions, findings ..

man-95\work-from-home-answering-service-operator.pdf ...
man-95\work-from-home-answering-service-operator.pdf. man-95\work-from-home-answering-service-operator.pdf. Open. Extract. Open with. Sign In.

Answering Your Patients.pdf
Some patients may be able to have an excimer laser. treatment or PRK to improve their vision without glasses. after they have healed from the cross-linking ...

APeC: Automated People Counting from Video
Connected component analysis is ... people from other objects present in each frame of the video ... Afterwards, a connected component algorithm clusters the.

Young Gay Boys Videos
Young Gay Boys Videos - https://www.boysc.com/rss.xml