150
Int. J. Mobile Learning and Organisation, Vol. 4, No. 2, 2010
Computational approaches to support image-based language learning within mobile environment Oleg Starostenko*, Vicente Alarcon-Aquino and Humberto Lobato-Morales Department of Computer Science, University UDLAP, Puebla, México E-mail:
[email protected] E-mail:
[email protected] E-mail:
[email protected] *Corresponding author
Oleg Sergiyenko Department of Applied Physics, Engineering Institute, University UABC, Mexicali, BC, Mexico E-mail:
[email protected] Abstract: The exploration of emerging data exchange technologies and design of image-based language learning (IBLL) applications are presented in this paper. For integrating, the mobile devices to learning process the generic interfaces have been created for portable personal spaces (PoPS) providing mobile access to multimedia documents based on XML technologies. The IBLL implies image processing, recognition and retrieval, thereby some algorithms have been proposed for learning assistant applications used by mobile devices. Furthermore, for multimedia data exchange in wireless environment the compression of visual information based on wavelet transforms and several thresholding techniques are supported. The proposed approaches can suggest ways of studying and organising resources which provide long-term guidance on developing skills and support experiential learning. They have been tested for selecting the best ones with the highest processing speed and recognition grade for interpretation of Japanese kanji or Mayan glyphs on mobile devices with limited resources and restricted networking capabilities. Keywords: mobile language learning; computer-based learning assistance; networking technologies; image processing; PoPS; portable personal spaces. Reference to this paper should be made as follows: Starostenko, O., Alarcon-Aquino, V., Lobato-Morales, H. and Sergiyenko, O. (2010) ‘Computational approaches to support image-based language learning within mobile environment’, Int. J. Mobile Learning and Organisation, Vol. 4, No. 2, pp.150–171.
Copyright © 2010 Inderscience Enterprises Ltd.
Computational approaches to support IBLL
151
Biographical notes: Oleg Starostenko is a Full Professor in the Department of Computer Science at the University de las Americas Puebla, Mexico. He received the PhD in Physics and Mathematics from the University BUAP, Puebla, Mexico in 1996. He is the Author of more than 100 research papers. His research interests include development of systems for multimedia information access, retrieval and processing within distributed environments. Vicente Alarcon-Aquino is currently a Full Professor and Coordinator of Postgraduate Studies in the Department of Computing, Electronics and Mechatronics at the University de las Americas Puebla, Mexico. He received the PhD and DIC degrees in Electrical and Electronic Engineering from Imperial College, London UK in 2003. He has also authored more than 80 research papers published in journals and conference proceedings. His research interests include signal and image processing using wavelets and wavelet theory applied to performance monitoring of communication networks. Humberto Lobato-Morales has obtained the BSc and the MSc from UDLAPuebla University, Mexico in 2006 and 2008, respectively. During the Masters degree carrier his interest was the developing digital processing techniques using wavelets. He joined the Emerging Microwave Technologies Group, EMT at Research Institute INAOE, Mexico in 2008, where he is currently a Research Engineer. His project includes characterisation and measurements of materials using microwave techniques. He is the author of seven published papers in international conferences and journals. Oleg Sergiyenko received the BS and MS in Kharkiv National University of Automobiles and Highways, Ukraine, in 1991 and 1993, respectively. He received the PhD in Kharkiv National Polytechnic University on Specialty Tools and Methods of Non-Destructive Control in 1997. He has written 47 papers and holds 1 patent of Ukraine. In December 2004, he was invited by Engineering Institute of Baja California Autonomous University, Mexico for researcher position where he is currently Head in the Department of Applied Physics. His scientific interests are in automated metrology and smart sensors, control systems, robot navigation and 3D coordinate measurement.
1
Introduction
The emergence of ubiquitous computing introduces wireless and portable technologies that democratise access to information and services and thereby opens new research challenges in mobile language learning area. From a computational point of view, the language learning environment may be defined as virtual space which consists of repositories with digitised information and a wide range of user interfaces and services for collaborative work, digital documents searching, management of distributed databases, multimedia data processing and retrieval (Lockyer et al., 2008). The present time is characterised by the evolution and migration of traditional learning out of the school onto the network. According to Horkoff (2008), it means that new mobile learning approaches are now in demand combining learning media (comprehensive, searchable, relevant, timely and learning objects), open community (a social/collaborative learning environment) and personalisation (customisable tools and content options for the individual). That is, the expansion of wireless networks, appearance of novel telecommunication technologies, the explosion of power, functionality and capacity of
152
O. Starostenko et al.
new communicating facilities such as smart phones, PDAs, laptops or tablet PCs provide acceptance and integration of mobile devices into contemporary life as part of our social practice (Ellen, 2005). The common features of these devices (ubiquity, accessibility, flexibility, security, reliability, interactivity, etc.) which now include internet access, voice- and text-messaging, cameras and video-recording enable communicative language practice, access to authentic content and task completion (Chinnery, 2006). Obviously, learners are now continually on the move. Thereby, the theory of learning must be based on contemporary accounts of practices that enable successful learning taking into account the ubiquitous use of personal and shared technology (Sharples et al., 2005). The main advantage of mobile language learning is a possibility to access information and services anytime and anywhere. However, a mobile device has physical features (small screen, reduced processing power, limited memory, secondary storage and bandwidth capabilities) that are very different in traditional computers. Therefore, a simple migration of application from computers to new mobile devices is not possible. The specific architectures and approaches have to be adopted and developed for efficient data management in mobile environments. This is the first objective of this paper: taking into account the convergence occurring between the new conceptions of learning and the new mobile technologies, evaluate and propose appropriate wireless infrastructures for data exchange in distributed environments applying client–server or port-to-port approaches; optimise and adapt user interfaces for mobile devices with limited resources and capabilities of visualisation, particularly, for mobile language learning activities. The online language learning deals with search, access, retrieval and visualisation of digital documents with text, images, audio and video. Particularly, the learning imagebased language (Japanese, Mayan, etc.) implies indexing, recognition and interpretation of visual information. As result, there is a growing need of efficient visual information processing and retrieval approaches to be fast and simple enough for migration to mobile devices. Thus, the second objective of this paper is to design some algorithms for multimedia documents processing, recognition and retrieval appropriate for mobile devices. Obviously, the real language learning goes beyond symbol learning, but the ability to download support resources, information and services and instantly to carry out the actions according to the particular behavioural model empower users to become active learners responsible for their own knowledge acquisition and decision making anywhere, anytime (Katz, 2005). The fact that a mobile device with limited resources will access and manage visual information, the compression of data streams now becomes a significant problem. The last part of this paper shows the proposed techniques of visual information compression to be adopted for image-based language learning (IBLL) assisted by mobile devices.
2
Pedagogical aspects of mobile learning
Traditional pedagogy requires a teacher to assist the learning process and to solve learner problems and misunderstandings, generally, in a two-way conversation. In fact, the communication between teacher and learner is provided through the mediating tools such as reference textual and audiovisual material and lab equipment. Also the environment plays a role of mediator between them using tools and resources such as textbooks, dictionaries, logbooks, etc. supporting in this way reflective learning.
Computational approaches to support IBLL
153
Most theories of pedagogy fail to capture the distinctiveness of mobile learning because they are predicated on the assumption that learning occurs in a classroom environment, mediated by a trained teacher (Sharples et al., 2005). Thus, novel mobile learning theories, particularly, for IBLL, will be tested against the following criteria: 1
learners are on the move and use networked devices which allow them to become part of the dynamic system
2
the vast amount of learning takes place in informal learning situations
3
the dynamic context of learning provides people with the knowledge and skills they need to succeed in a rapidly changing world
4
the ubiquitous nature of learning
5
learning as a constructive and social activity.
Currently, the mobile learning is interwoven with other activities as part of everyday life. Some activities such as conversation, reading, museum visits or reviewing material on mobile devices can be resources and contexts for learning, and others activities are integrated with non-learning tasks such as shopping or entertainment in which a person strives to overcome a learning problem. The emerging wireless communication technologies have some advantages combining the functions of phone, PDA, camera and multimedia services for text and visual data messaging. As a consequence, the convergence is occurring between new conceptions of learning (individualised, learner centred, situated, collaborative, ubiquitous and lifelong) and new mobile technologies (personal, user centred, mobile, networked, ubiquitous and durable) (Sharples et al., 2005). It is important to analyse the new conceptions of mobile learning with respect to the proposed approach for IBLL. We suppose that learner with networked mobile device may acquire the knowledge about new language and participate practically in all learning activities. Analysing recent learning paradigms and conceptions it is possible to generate some specific usage scenarios where the proposed computational approaches for IBLL will be useful and effective. We start with the behaviourist learning paradigm where learning is facilitated through the reinforcement of an association between a particular stimulus and a response. Thus, mobile devices can enhance the behaviourist learning process by presenting teaching materials/content specific questions (stimulus), obtain responses from learners (response) and provide appropriate feedback (reinforcement) (NESTA, 2009). The possible scenario in this case may be described in context of IBLL using mobile phones, when students will send frequent vocabulary messages and revision material via SMS text or MMS messages. Mobile phones with online services in proposed approaches allow students to access multiple choice questions and answers, and practical exercises, permit to review, listen and practice speaking and provide services such as symbol or phrase translation, description and visualisation. Analysing the constructivist mobile learning used in recent pedagogical theories, the proposed approaches permits to learner construct new ideas or concepts based on their current and past knowledge. The usage scenarios cover applications, for example, textual, audio or visual games where learners play an active role in the simulation of a dynamic process or where context-sensitive data and social contacts (interviews with virtual experts) are used to supplement real world interactions (NESTA, 2009).
154
O. Starostenko et al.
In context of the situated learning paradigm, the mobile learning involves as the acquisition of knowledge by individuals as a process of social participation. In different situations, student will use mobile devices to support field studies of image-based language, for example, listen to pre-loaded instructions, taking observational notes, taking photo, querying networked database and comparing data, record students own reflections, etc. For example, the learner finds unknown symbol (kanji), takes a picture of this symbol using camera of mobile device, and submits a query to find its significance. The service application will respond with translation and audiovisual description of symbol and will permit to learner participating within a community of practice. The proposed approaches also provide an interactive audiovisual tour (museum), allowing visitors to view video and still images, listen to expert commentary and reflect on by answering questions or mixing a collection of sound clips to create their own soundtrack for an artwork. According to the sociocultural theory of learning the mobile environment can make a significant contribution to process the communication between agents of collaborative group. By facilitating the rapid access to other users anytime anyplace, sharing content, knowledge, experience and gossip, learners can develop ‘communities of practice’ as well as informal discussion groups. One example of collaborative work supported by proposed approaches for IBLL is the case when a group of students work together recollecting data from a remote digital repositories of images, online dictionaries or tutorials in the internet to record pattern, practice with proposed exercises and sharing their own results. In this context, Sharples et al. (2005) suggests a conversational approach for a mobile learning when the teacher may be substituted by computer-based mentor or intelligent tutoring system providing access of learner to computer-based knowledge organisers and mind tools. However, for successful learning process the computer-based teacher must be able to carry out the kinds of teacher dialogue conversing with the learner about the task, general conceptions and misunderstandings. Recently, the development of tutoring and pedagogical software is a promising direction of the language learning (Dougiamas, 2009). The proposed computational approaches presented in this paper have been developed to support computer-based teaching of the image-based languages. It is important to mention that recently two challenges for mobile language learning stand out. First one is the need to understand how the complex setting functions to influence learning and second how the mobile device itself is understood and adopted by users. The activity theory proposed by Vygotsky (1987) gives the best current accounts of this complex problem and is currently applied to develop an understanding of human practice. According to Vygotsky, the learning has three features involving a subject (the learner), an object (the task or activity) and tool or mediating artefacts. The relationship between the subject and the object of an activity is mediated by a tool and a human behaviour is situated within a social context that influences their actions. Therefore, mobile technologies provide the potential to support such activities. With regard to informal, accidental learning, learning episodes are impossible to predict. The personal and portable nature of mobile technologies make them very strong candidates for recording, reflecting on and sharing this type of informal learning including that the new educational imperative is to empower people to manage their own learning in a variety of contexts throughout their lifetime (NESTA, 2009). Thinking about the development of applications for IBLL, the proposed approaches may be classified as the tutoring/mediating tool or as the learning assistant for situated
Computational approaches to support IBLL
155
and personalised training/learning activities. The designed software for these type of computer-based mentor embedded in portable device may be considered as pedagogical agents for mobile learning acting as a tool to perform a task, to assist learning, to help a learner to generate and organise ideas, to provide guidance on developing skills, etc. Another purpose of the proposed approaches for IBLL is the development of methods for multimedia information retrieval, filtering, processing and visualisation as well as design of tool for learning to learn and for organising knowledge within sharable conceptual frameworks such as dictionaries, concept maps, learning organisers, schedulers, etc. According to analysed learning paradigms and theories for computer-based mentor, the mentioned scenarios of learner interaction with an IBLL environment may be classified following to suggestions of Ayala et al. (2009): just in time knowledge (a learner has access to learn contents anywhere and at anytime); simulating the activity (a learner interacts in a simulated communicative activity through designed potable personal spaces; collaborative problem solving (the interaction of learner with others in a learning task is limited but it is not impossible) and social construction of knowledge (a learner provides to the community documents and reports about their own experiences ). Finally the main challenge in mobile language learning is to develop the infrastructure for search and retrieval of technical, learning and training content from an integrated data environment. According to the report of Katz (2005), the emerging technical standards and specifications for wireless network technologies can permit exchange of reusable Content Objects among common source Data Repositories that store both Technical Data and Training/Learning Content. The delivery of technical data or training content objects to the appropriate impromptu communities or to particular user relies on Data Request and Learning Content Object Data Resolution Web Services through XSLT engine that transforms XML file into wireless markup language (WML) or XHTML (see Figure 1). The proposed infrastructure with automated content data configuration guarantees that users will have access to the most recent content data whether it is in the form of technical manuals or associated training and educational resources. This infrastructure is well suited for IBLL and may be used as prototype for development of proposed computational approaches. Figure 1
Data flow and resolution to deliver learning and technical objects to mobile technology end users (see online version for colours)
Source: Katz (2005).
156
3
O. Starostenko et al.
Infrastructures for mobile applications
The modern mobile devices have web browser and provide interactive content via wireless access protocol (WAP) or hyper text transfer protocol (HTTP). Therefore, the simplest way for the development of language learning systems is based on server–client architecture adapting user interfaces to resources of particular mobile device. In the field of digital document searching, retrieval and visualisation, the main challenge is to design a framework to exploit the advantages of mobility. To provide users with web pages which contain multimedia information and that are well suited to limited screen and capabilities, three major solutions are known: methods to summarise web pages, tool to convert HTML into WML and languages to create web interfaces which can be accessed from any device (Smith, 2002). The methods used to summarise and browse web pages on small screen of mobile devices are based on results of user tests, which show that a combination of keyword extraction and text resume gives the best performance for discovered tasks on web pages. Hence, one method extracts significant keywords from the text units and another one uses information retrieval techniques to find the most significant sentence of each text unit. This technique is difficult to use if retrieval of visual information is needed. In the context of WAP, a conversion tool (transcoder) extracts text from HTML pages and then reformats it into WML, which is used by WAP phones. There are two approaches for the conversion: fully automated and configurable. They are unfortunately more oriented to textual than to visual documents. Fully automated methods extract all possible contents from the page (such as title, welcome message, links, etc.). Configurable methods extract specific parts of the page, for example, headlines or stock prices (Arehart, 2006). The third approach exploits XML-based markup languages to create generic interfaces, which in our work have been used for the design of portable personal spaces (PoPS). This concept promotes the separation of the interface description from the application as well as a scalable interface design. The most representative language of this approach is user interface markup language (UIML) that describes user interfaces in a device-independent manner; these descriptions are automatically mapped to the language used by a target device (Java, HTML, VoiceXML and WML) (Castellanos et al., 2004). This approach is just one of many possible implementation of the wider service-oriented architecture (SOA) model. However, the SOA model has some disadvantages. According to Warren and Bishop (2005) in the server side of Java, in certain situations, JRMP is much more useful and effective than simple systems based on the HTTP. In portable devices sometimes the WAP protocol can be limited and inappropriate. While accessing the internet via WAP could be very effective, this technology can be limited when it is compared with the array of abilities available through Java-enable device. Therefore, the network technology based on J2ME platform characterised by execution of applications on mobile terminal is the alternative way for design of port-to-port architecture PPA. Either SOA or PPA models are valid and appropriate for the development of language learning applications. The selection of the particular infrastructure depends on specific requirements to mobile equipment or available software for learning applications.
Computational approaches to support IBLL
4
157
PoPS on SOA for language learning applications
For successful learning activities, the proposed interaction of a learner with computerbased mentor must be defined by one of some well-known learner models. These models are representations of some characteristics and attitudes of the learners which are useful for achieving the adequate and individualised interaction established between computational environments and students (Dougiamas, 2009; Paiva et al., 1995). In this way, the adapting the learning environment itself to the individual learner may be achieved. Thereby, the development of PoPS proposed in this paper is considered as the important contribution to solve the problem of interaction of a learner with computerbased mentor. The implementation technique selected for exploiting mobility and access to information and services in applications for language learning was the model view controller (MVC) on SOA. It has some advantages for the development of web-based systems including the simple definition of data structures and methods for multimedia data processing (application of Java Beans in case of Java-based systems); efficient management of distributed data (using servlets in web applications) and easy development of high performance dynamic interactive interfaces (using Java server pages). The system architecture as infrastructure for web-based applications has been implemented using J2EE BluePrints web application framework (WAF) that provides all basic functions for processing requests, methods invocations, screen selection and assembling. This infrastructure provides Java servlets by front controller for updating the screens, permits extensions of JSPs (custom tags libraries) created for manipulation of the parameters and attributes used for development of interfaces, integrates abstract classes for all multilayers actions executed on web layer, supports the information in the appropriate format and language. For user assistance in language learning applications, the concept of personal spaces is used referring to virtual areas in a learning system through which users can manage resources and services, select contents relevant to their interests and organise them according to their individual needs and preferences. Designed PoPS provides mobile access to resources and services using XML via user interfaces which form personal space learning environments (Castellanos et al., 2004). Applying PoPS to IBLL, the following properties of learner model proposed by Paiva et al. (1995) have been taken into account: 1
understandable (the content of the model should be clear and motivated by cognitive-based assumptions that describe human mental states, so that it can be understandable by other agents)
2
transferable (the system that allows externalisation must be able to communicate knowledge with other agents)
3
usable (the externalised models must have a degree of abstraction that makes them usable by the other agent).
PoPS consists of three components: Generic Personal Space (GPS), Converter and Interface Generator (see Figure 2).
158 Figure 2
O. Starostenko et al. Context diagram of the designed portable personal space (see online version for colours)
The generic PoPS contains the high-level descriptions of user interfaces. These descriptions are written in device-independent manner using an XML-syntax. Each interface is kept in one single XML file. This approach provides highly scalable descriptions: if a new tool or service is added to the personal space, only the new specifications must be added to the original. The Converter transforms the interface descriptions into code, which is written in the language used by target device. Therefore, it transforms the interfaces of the generic PoPS into XHTML (for PDAs) or WML (for WAP phones). It has two components: XSL stylesheets and an XSLT engine. The XSLT engine transforms the XML file into WML or XHTML according to the instructions provided by the XSL stylesheet; each of the target languages requires its own XSL stylesheet. Therefore, the converter has two XSL stylesheets: first for XHTML and second for WML-based devices. The scalability of PoPS relies on the following feature: to convert the GPS document to a new language, only the appropriate XSL stylesheet of the new language must be designed. The Interface Generator IG contains JSP files to build the interfaces. The JSP files use JavaBeans to perform search and they update personal space configurations. To build interfaces for personal spaces IG receives user parameters and device specifications and then generates database queries to retrieve the user’s personal space configuration, for example, the latest chapter or exercises in language learning support application. According to the user configuration the corresponding interface is built and returned to the user. Whenever user performs a search, IG receives the search parameters and forwards them to the respective information retrieval service. When the service sends back the search results, IG composes the appropriate interface to present them to the user. This interface composition is made by wrapping the search results into code of the device language. Finally, the interfaces containing the obtained results are sent to the user. If the users modify their individual personal space configuration, IG updates these changes onto the database. The mobile users can access PoPS from designed WAP emulator or a PDA using corresponding generic interfaces. PDAs connect to PoPS via internet through a wireless network and the emulator via WAP gateway emulator implemented on Openware and Siemens 5.0.2 tools.
Computational approaches to support IBLL
5
159
Image-based symbols recognition
In case of IBLL, the image processing has to be applied for recognition of symbols and whole text. To propose some image recognition algorithms appropriate for mobile applications, in this paper the scripts with kanji and Mayan glyphs have been used. It is important to note that image processing methods and algorithms usually may not be directly applied to mobile applications. There are some considerations for development of novel approaches for image processing on mobile devices such as: 1
the image pre-processing, indexing and classification must be simple enough to be migrated to mobile device
2
for the real-time applications the complex and time consumption image processing algorithms are not appropriate
3
pre-processing and image compression must be implemented for reduction of the amount of data during exchange of visual information
4
mobile device must be able to run image pre-processing and compression applications either for PPA or SOA.
Kanji is the Chinese characters that are used in the modern Japanese logographic writing system along with hiragana, katakana and the Arabic numerals. There are a lot of commercial systems for kanji translation and interpretation (Japanese Translators, 2008) but most of them cannot be used in the real-time applications. The Mayan glyphs formed by significant number of segments are more complicated for recognition due to possible variations of their representation or writing (FAMSI, 2008). Currently, there are a lot of systems for automatic pattern recognition which operate according to some stages: image acquisition, feature extraction and classification based on the analysis of the low-level image characteristics, such as colour, texture or shape (Flores-Pulido et al., 2008; Gonzalez and Woods, 2007). However, this type of systems does not provide semantics associated with the content of an image. For example, the complex script-based language learning application which deals now not only with textual but also with multimedia information, may be treated as content-based image retrieval CBIR system. The novelty of this approach is that the content-based querying and searching procedures are made using the image characteristics instead of textual descriptions used in well-known translators. The novel methods for global image description, pattern indexing and recognition such as elasticity correspondence, Curvature scale space approach, B-splines and chain case codes, Fourier and wavelets spectral descriptors, neural networks and fuzzy classifiers, statistical and predictive algorithms, etc. are sometimes too complex for fast processing, and they are usually sensitive to spatial variations of symbols in image (Starostenko, 2005). Therefore, a simple and efficient algorithm must be proposed for applications assisted by mobile devices taking into account their restricted capabilities.
5.1 Complex symbol recognition by SNM The basic service for IBLL applications is the image recognition and interpretation. This service provides processing and translation of acquired images from multimedia documents or from video input of mobile device. The proposed method for pattern
160
O. Starostenko et al.
recognition by segments and neighbours matching (SNM) is based on fuzzy logic for handwritten character recognition. In a fuzzy recogniser, the analysed pattern x belongs to a certain class A with some membership degree d = M[A(x)]. The method determines which class seems to be the most probable according to their membership degree. Our method does not manipulate directly with binary representation of pattern, the image skeleton must be obtained from the pattern using Pavlidis’ algorithm which works as the simplest edge detector (Gonzalez and Woods, 2007). The proposed SNM method is implemented as follows: 1
The automatic normalisation of an image and the extraction of symbol skeleton are carried out. Then the fuzzy pattern is obtained by classification of segments in the skeleton according to some predefined orientations.
2
The classification means that a set of segments obtained from the fuzzy image are compared with all classes of existing previously classified patterns stored in memory. The segments are mapped in the simplest case according to one of four possible orientations as it is shown below:
μH = 1 – (Angle[|180q T|, |0q ± 360q T|]/45q or 1) horizontal segment μV = 1 (Angle[|90q T|, |270q T|]/45q or 1) vertical segment μP = 1 (Angle[|45q T|, |225q T|]/45q or 1) segment with angle T about 45q μN = 1 (Angle[|135q T|, |315q T|]/45q or 1) segment with angle T about 135q ‘Angle’ is the function which converts segment inclination T to number between 0 and 1 defining in this way the orientation of the segment. If a symbol must be represented with more details the additional angles of segment orientation may be added. The algorithm for similarity detection between elements of input image and four types of predefined orientations is based on computing the variable Pij. For a given two segments i, j we have: μij = 1 if i = j; μij = 0.5 if i = H and j V H; μij = 0.5 if i = V and j H V; μij = 0.5 if i = P and j N; μij = 0.5 if i = N and j P; μij = 0.1 other cases. Here, we introduce the threshold Ui which defines a similarity degree. For example, let U2 = 30q and segment angle is T = 80q, thus, the segment will be considered as vertical, because 90q U2 80q 90q + U2 (segment angle is between 60q and 120q). For automatic definition of segment type, the simple algorithm can be applied: 1
select the first active point (pixel of image) P1, initialise segment type variable st = ´´.
2
select the active point P2 at distance s from point P1.
3
while P2 exists, calculate the derivative, among the points P1 and P2
4
if result is between 0q U1 and 0q + U1, segment type variable at point stp = h (horizontal) if (90q U2) T (90q + U2), stp = v(vertical); if (45q U3) T (45q + U3), stp = n(45q) if (135q U4) T (135q + U4), stp = p(135q)
Computational approaches to support IBLL 5
if stp = st, the segment stp is part of segment st
6
otherwise, add st to list of SEG, with all points that form it and assign stp to st
7
calculate PN as a distance from point P1 to next active point
8
return the list of segments SEG.
161
The classification algorithm is based on calculating the membership degree between patterns obtained according to algorithm 1 and stored classification patterns as follows: 1
given analysed pattern n
2
for each stored class m
3
calculate membership Ps = f(n, m)
4
select the class c for which Ps is maximum, and assign to class a value P
5
if there are some other classes v different to c and whose membership Ps is similar to P select also them as possible classes to which n may belong
6
if the biggest membership P does not exceed a predetermined value U, n is the possible candidate to be considered as new class.
The membership of the pattern n towards a class m is calculated comparing Pand 1/U 1
for each segment i of m
2
find k more similar segments in n, named j
3
calculate the similarity between i and j, and with their neighbours according predefined U value
4
choose the segment j with the highest similarity to i, including neighbours mi
5
segment j is now removed from the list of possible segments
6
similarity P = 6Pi/(number of segments in m).
Table 1 presents the membership degree (the number between 0 and 1,000) which defines how the input pattern is similar to the selected set of reference symbols. The noise level can be evaluated as a number of removed points and additional segments founded during the skeletonisation and classification processes. According to these algorithms, the whole complex symbol can be recognised and interpreted with predefined error introduced by U value.
162 Table 1
O. Starostenko et al. Membership grade for noised Mayan glyph of class #130 and for Japanese kanji
Glyph, class #130 Pattern
Kanji Membership grade
Pattern
Membership grade
944
932
880
908
5.2 Image recognition based on VIR approaches It is possible to improve the well-known methods for complex symbols recognition by applying new approaches used in visual information retrieval VIR systems. These approaches use multiple feature vectors which define the low-level image characteristics as well as the image content. The disadvantage of the CBIR approaches is their complexity which may be solved by using simple algorithms for feature vector generation, content indexing and computing similarity to reference images. The well-known systems usually provide retrieval of images, graphics and video data from online collections using low-level image features such as colour, texture or shape (Starostenko et al., 2008). Although, the contributions of these systems to field of VIR systems design were important, they do not provide mechanisms to represent the meaning of the image. Applying the approaches that estimate machine-understandable semantics may improve search, access and retrieval processes in CBIR systems (Iqbal, 2007). The proposed method may be introduced as a combination of specific descriptors based on shape pre-processing used for feature extraction invariant to scale, rotation and illumination, including the design of descriptive name space database for definition of retrieved images content. Traditionally, a shape is more informational feature described by the set of segments which may be extracted by any well-known method for border detection. However, the representation of a symbol by a set of segments is not a convenient form for calculation how similar is that symbol to another. Any changes in scale or rotation of the same shape makes quite difficult for the comparison of the obtained feature vector with vectors described the pre-processed shapes in digital collection. Thereby, the computing similarity between feature vectors using so-called two-segment turning function (2STF) is proposed (Chávez-Aragon et al., 2007). A set of segments obtained for example, by Pavlidis’ algorithm is represented by a step function. The step on x-axis represents the normalised length of segment and the y axis represents the turning angle between two selected segments as it is shown in Figure 3. The invariance to rotation is achieved by computing the relative angle between two consecutive segments instead of absolute segment angle with respect to x-axis. The advantage of this approach is the invariance to translation, scaling, reflection and rotation. 2STF is built taking into account the relative position between segments (turning angle) and the relative length of each segment (relative to accumulative length of all segments in a shape). This allows getting the same representation for a set of symbols even though they are placed in different positions or has been reflected or rotated. The
Computational approaches to support IBLL
163
similarity between two shapes is evaluated by computing the differences between 2STFs as it is shown in Figure 4 where the shaded area represents how similar two shapes are. The disadvantage of 2STF approach is significant time that it takes to find the best correspondence between two shapes. Experimentally, we determined that this time is some seconds for more than 50-segment symbols. The processing time is more critical for Mayan glyphs than for kanji. It may be reduced by applying another matching strategy, for example, using the Star Field shape matching (Chavez-Aragon et al., 2007). Now we can conclude that the proposed approaches: 1
can recognise patterns with high level of variations in symbol representation
2
generate a set of possible classes of similar patterns with corresponding matching grades
3
work with image noise without reduction of efficiency
4
generate new classes for unknown input patterns
5
reduce error by variation of membership degree
6
2STF is more exact than SNM but it is not so fast
7
may be used for applications for mobile devices.
Figure 3
Mayan glyph and Japanese kanji with their 2STFs (see online version for colours)
Figure 4
Computing similarity between two shapes
164
6
O. Starostenko et al.
Implementation of CBIR system for mobile language learning
After analysing and testing some networking architectures for mobile access to distributed information used in language learning systems the SOA-based approach has been selected due to its wide acceptance by developers of mobile applications. With the same success, the PPA may also be used. To support the usage scenarios of language learning described in Section 2some services has been designed for the multimedia information access, processing, indexing, retrieval and interpretation. The block diagram of the designed prototype for IBLL assisted by mobile devices is shown in Figure 5. This proposal is based on infrastructure described in Section 2 (see Figure 1). Applying these services to IBLL the input query on the mobile device may be an image with symbol or a set of symbols to translate or textual description of symbol. The connection of client with web server may be carried out using ‘wrappers’ when data structures are wrapped and sent via server to the client by TCP/IP. However, the simplest way has been applied in this prototype. It does not require a specific support for wrappers and it is based on implementation of queries by binary strings received by the server. The server according to the received string calls the services for different types of queries (visual or textual), recovers the results of searching and returns to client the XML document with retrieved information as it has been shown in Section 4 (see Figure 2). The visual query is processed by feature extraction module that defines the shape of a symbol by set of segments. The indexing and feature vector generation are provided by using proposed either SNM or 2STF approaches. Computing similarity between feature vectors of analysed symbols and corresponding reference symbols involves pre-processed image database DB. If some symbols with highest grade of similarity (the lowest grade of similarity about 80% has been defined empirically) have been found in Image DB, the corresponding to these images descriptions are selected from Descriptive Name Space (vocabulary) database. Figure 5
Block diagram of the CBIR system (see online version for colours)
Computational approaches to support IBLL
165
The textual descriptions in the Name Space DB consist of keywords linking to the previously pre-processed images to the symbol semantics. The separation of the textual and visual data has some advantages for data access, management, registration and organisation. This permits to speed up the matching process and reduces the number of iterations in searching process. Finally, the retrieved images with their interpretation are sent to Response Generator which returns the XML document to mobile as response to the user query. If a query is a textual description, the visual presentation of symbol is retrieved from pre-processed Image DB corresponding to that description. The system requirements for the designed prototype are: PC computer as a host server and PDA Ipaq h5555 with Operating System Pocket PC (Windows CE or Handheld PC), with installed Compact Media Framework, which support wireless protocol IEEE 802.11b 1
router Hub Netgear Wireless Router MR814V2
2
programming language C# for .NET platform used for the development of native Windows applications for any type of PDA or Smartphone
3
the input images are in JPG, JPEG or BMP format with resolution 620 u 480 pixels.
In general case of CBIR the Image and Name Spaces DBs have specific organisation for fast image and its content searching and retrieval. The description of semantics is established by direct relationship between a set of segments defining the symbol shape and meaning of that symbol. This approach facilitates the implementation of useroriented vocabulary of terminology in Name Space DB and corresponding images in preprocessed Image DB. The proposed system can translate sequences of glyphs and phrases. Figure 6 shows the simplest case of concatenated glyph recognition using Mayan grammatical rules. The meaning of each glyph can be grouped together to obtain a word or phrase. The proposed prototype has been tested using online image collection with Mayan glyphs descriptors and Japanese kanji translators. Figure 6
Concatenated glyphs recognition (see online version for colours)
Note: Result: CUTZ, glyph means TURKEY.
166
O. Starostenko et al.
The performance of the system has been evaluated by using the precision and recall metrics traditionally used for CBIR systems:
Precision
A , Recall B
A C
where A is a set of relevant images retrieved by system, B is a set of relevant and irrelevant images retrieved by system for particular query and C is a set of all relevant images in collection for particular query. The recall defines the proportion of relevant images in the entire database which are retrieved as response to a query. The precision is proportion of the retrieved images that are relevant to the query from a set of the retrieved images. Some experiments have been done for computing the average recall and precision according to previous equation. When the proposed approach is applied over a small set of images, recall and precision values are low. When the number of images increases recall/precision also increases because there are more possibilities for applying the similarity metrics over a major number of shapes with the similar segment orientation. The average recall/precision values for these experiments lie in the intervals about (0.65 / 0.85)/(0.38 / 0.52), respectively. The experiments show that proposed approaches are rigorous enough and do not accept wide range of images as candidates to be retrieved.
7
An approach for image compression
The growth of multimedia information in language learning applications has conducted to the development of new techniques for image compression and image de-noising. The standard image compression formats as JPEG uses sub-images decomposition, bi-dimensional discrete cosine transform (DCT) and the Huffman coding algorithm (Gonzalez and Woods, 2007). In this regard, several spectral transforms have also been reported (including Fourier, Walsh, Hadamard, Hartley transforms, etc.). The best results are obtained with wavelet transforms used in the last JPEG version (JPEG2000) where the discrete wavelet transform (DWT) instead of DCT provides higher peak signal-to-noise ratio (PSNR) and compression ratios (Rc) (Talukder and Harada, 2007). Due to special characteristics of the human visual system, the images may be distorted without significant degradation. Thus, some of the high frequency coefficients can be suppressed achieving good compression performance. The proposed solution for image compression based on the DWT consists of the following steps. In the first step, the image decomposition in 8 u 8 sub-images is applied. Each one is then converted to a 64 point-vector following a scan with the Hilbert fractal curve (Gonzalez and Woods, 2007). The resultant vectors are located in a new matrix called TE with size p u q, where p = m u n/64 and q = 64. The values m and n compose the original image dimensions. The idea of multiresolution analysis is to write a signal x(t) as a limit of successive approximations. The differences between two successive smooth approximations at resolution 2j-1 and 2 give the signal at resolution 2j. In other words, after choosing the initial resolution J, any signal x (t ) L2 () with its wavelet {dj,n} and scaling {cj,n} coefficients can be expressed as:
Computational approaches to support IBLL
167
f
x (t )
¦
c J , n M J , n (t )
n =
j , n\ j , n (t )
j J n=
f
d j ,n
¦¦ d
2 j /2
³
f
x(t )\ j ,n 2 j t n dt , c j ,n
f
2 j /2
³ x(t )M 2 j ,n
j
t n dt
f
The last equations express that a signal x(t) is decomposed in details {dj,n} and approximations {cj,n} are found. The one-dimensional DWT is realised to each row of the TE matrix (TEi = cij 1,m ), and the resulting scaling and wavelet coefficients ( cij ,n and d ij ,n , respectively) form the correspondent ith row-vector TFi of new transformed matrix TF: cij , n
i l j 1,2 n l ,
¦g c
d ij , n
lZ
i l j 1,2 n l , i
¦h c
1, 2, , m u
lZ
n ; 64
j 1, 2, ..., J
where J corresponds to the sub-band coding level of the DWT, cij 1,2n l is the ith row of TE matrix, and gl and hl are low- and high-pass filters, respectively used for DWT. The second step consists in suppression of high frequency coefficients using some thresholding criteria. For high frequency coefficients suppression, a threshold level must be obtained in order to eliminate only the lower energy (defines contents of the data details) and consequently achieving low image distortion. Thresholding schemes can be classified into hard- and soft-threshold. The first one eliminates the coefficients under a given threshold Ȝ, and maintain the other at the original values. The soft threshold, on the other hand, is an extension of the former, but has one difference, the coefficients below the threshold are eliminated and the others are setting to a new value. The hard- and soft-thresholds analysed signal x(t) are described by x(t )
x(t ) | x |! O , x(t ) ® ¯ 0 | x |d O
sgn( x)(| x O |) | x |! O ® 0 | x |d O ¯
To obtain the threshold value Ȝ, four algorithms can be used: minimax, fixed form, rigorous sure and heuristic sure; all of these are based mainly on the number of samples S of the transformed signal. The final step uses an entropy Huffman coding algorithm for lossless data compression. It is a symbol probability-based method that forms new code-words in order to remove data redundancy. To reconstruct the image, the inverse process of decoding data from the Huffman algorithm forms again the TF matrix. The inverse discrete wavelet transform (IDWT) is applied to each row TFi of the transformed matrix. The IDWT process is obtained in the same way as the DWT using the scaling and wavelet coefficients, the same filters and the same mother wavelet function utilised for the direct transformation:
TEis
cij 1,n
i l j ,2 n l
¦g c lZ
¦h d l
i j ,2 n l ,
i 1, 2, , m u n / 64, j 1, 2, , J
lZ
Then the 64-point rows of TEis reconstruct the correspondent sub-images following the same Hilbert fractal curve generating the new image.
168
O. Starostenko et al.
Implementing the proposed approach, the images with Mayan glyph or kanji are decomposed into 8 × 8 sub-images and each one is translated to a TEi vector following the Hilbert fractal curve. In this case, i = 1, 2, …, 1,024, so a 1,024 × 64 TE matrix is then generated. A five-level DWT is applied to each of the TE rows, forming the TF matrix, which for some cases is the same size as TE. High frequency coefficients are suppressed based on one of the thresholding criteria: high frequency mean, high frequency standard deviation, minimax, fixed form, rigorous sure and heuristic sure; and results are shown for comparison. The Huffman coding algorithm is then applied to the TE matrix in order to achieve data compression. The PSNR is used to assess the performance of the proposed approach through the mean square error (MSE) which are calculated as § 2552 · PSNR (dB) 10 log10 ¨ ¸ , MSE © MSE ¹
m 1
n 1
x 0
y
¦ ¦
2
ª fˆ ( x, y ) f ( x, y ) º ¼ 0¬ , mun
f ( x, y ) and fˆ ( x, y ) correspond to the original image and resulting image values, respectively, x and y are the spatial coordinates and m and n are the total image dimensions. The compression ratio RC is simply obtained by RC = m u n u 8/(coded bits). Figure 7 shows the PSNR(dB) for each of six thresholding algorithms. A Daubechies1 mother wavelet (db1) and 5-level of decomposition are used. The figure also compares the hard- and soft-threshold schemes. Figure 8 depicts the resulting images for Japanese kanji using heuristic sure thresholding algorithm based on hard-threshold scheme, five-level decomposition and Daubechies wavelet db1. The mean, standard deviation and rigorous sure thresholding algorithms generate similar reconstructed images and PSNR in which the differences between them are slightly noticeable by the HVS; as a result, only the resulting image from the heuristic sure algorithm is shown here. By using this thresholding technique only the lowest details are suppressed, and because of that, a sense of border detection can be appreciated in its respective error image. When mean, standard deviation, rigorous sure and heuristic sure algorithms are applied to the real images (not only black on white symbols) with Mayan glyphs or Japanese kanji compression and de-noised image are better as it is observed in Figure 9 for image with Mayan symbol. PSNR (dB) for hard- and soft-threshold applied to images with kanji (see online version for colours) 35 30 25 PSNR(dB)
Figure 7
20 hard
15
soft
10 5 0 mean
sd
rigorous sure
heuristic fixed form minimax sure
Computational approaches to support IBLL Figure 8
Original, reconstructed and error images using for Japanese kanji
Figure 9
Original, reconstructed and error images for image with Mayan glyph
169
The similar results were also obtained using other types of wavelets: Daubechies 2, Symmlet 3 and Coiflet 1. The minimal variation of Rc and PSNR was due to the fact that the mother wavelet function generates different transformed vector lengths. It was also observed that for simpler wavelet, the performance is better. Using higher dimension images, the performance tends to increase. The compression ratio tends to increase for all the thresholding algorithms when bigger image sizes are used. The PSNR also increases noticeably for all the thresholding algorithms using Daubechies wavelets (db1) with five levels of decomposition.
8
Conclusions
Considering that portable devices have limited resources, small screens, reduced amount of memory and restricted processing capabilities and the optimisation of interfaces for data management have been developed using the PoPS as highly scalable architecture that provides mobile access to digital collections using XML technologies. The client–server and port-to-port networking technologies were a base for selection of appropriate infrastructure for integration of mobile devices to the distributed environments. Both the SOA- and PPA-based infrastructures provide efficient data exchange within distributed communities in wireless or non-wireless applications. For IBLL assisted by mobile devices, the visual information processing services have been designed using the proposed SNM and two segments turning function approaches. They permit simple and fast image processing, recognition and retrieval based on symbol segments/shape indexing. The disadvantages of the proposed SNM and 2STF are the presence of errors during spatial sampling and generation of the image feature vectors as well as the required amount of system memory. It is important to mention that the presence of well-defined
170
O. Starostenko et al.
segments and contours is the main restrictions for input visual queries. Additionally, significant occlusions between symbols, week borders or complex background in image are not recommended in these applications. The analysis of factors like tolerance to occlusion, incompleteness and deformation, robustness against noise and feasibility of indexing are considered as possible extension of the proposed approach. After performing an analysis of the proposed image compression approach based on DWT and evaluation of several thresholding techniques some of them have been selected for mobile applications. The results reported in this paper also show that higher compression ratios generate higher level of distortions in the reconstructed images. Note also that a better performance is obtained when higher levels of decomposition are used (five-level wavelet transforms). The computational approaches implemented and tested in proposed system can suggest ways of studying and organising resources remembering ideas and events within PoPS. Thereby, designed application can provide such long-term guidance on developing skills as supporting experiential learning, particularly when the user could have direct access to the technology needed for performing the skill. The system can act as a learning assistant in performing tasks or solving problems by suggesting solutions and recommendations. Finally, we can conclude that the presented results could be considered as alternative way for the development of visual information retrieval facilities in wireless environments particularly used for IBLL assisted by mobile devices.
References Arehart, C. (2006) Professional Wap. Birmingham, UK: Wrox Press. Ayala, G., Paredes, R.G. and Castillo, S. (2009) Computational Models for Mobile and Ubiquitous Second Language Learning. Available at: http://www.inderscience.com/browse/index.php? journalID=179&year=2010&vol=4&issue=2. Castellanos, N., et al. (2004) ‘An exploration of network technologies for mobile data access in digital libraries’, Journal of WSEAS Transactions on Communications, Vol. 3, No. 1, pp.104–109. Chávez-Aragon, A., Starostenko, O. and Flores Pulido, L. (2007) ‘Star fields: improvements in shape-based image retrieval’, Journal of Research on Computing Science, Vol. 27, pp.79–90. Chinnery, G.M. (2006) ‘Emerging technologies, going to the mall: mobile assisted language learning’, Language Learning and Technology, Vol. 10, No. 1, pp.9–16. Dougiamas, M. (2009) M. Moodle: Learning Management System and Virtual Learning Environment. Available at: http://www.moodle.org. Ellen, D. (2005) ‘Enabling mobile learning’, EDUCAUSE Review, Vol. 40, No. 3, pp.40–53. FAMSI Mayan Translation (2008) Foundation for the Advancement of Mesoamerican Studies, Inc. Available at: http://research.famsi.org/montgomery_dictionary/mt_search.php. Flores-Pulido, L., et al. (2008) ‘Wavelets vs shape-based approaches for image indexing and retrieval’, Novel Algorithms and Techniques in Telecommunication Automation and Industrial, Electronics, Springer. Gonzalez, R.C. and Woods, R.E. (2008) Digital Image Processing. New Jersey: Pearson Prentice Hall. Horkoff, H. (2008) Praxis Language and Mobile Learning. Available at: http:// thenetworksense.com/2008/07/18/praxis-language-mobile-learning/. Iqbal, Q. (2007) ‘Content based image retrieval system’, PhD, Comp and Vision Research Center, University of Texas at Austin, Available at: http://amazon.ece.utexas.edu/~qasim/ research.htm.
Computational approaches to support IBLL
171
Japanese Electronic Translators (2008) Talking Dictionary. Available at: http:// www.languageteacher.com/39_japanese_electronic_talking_hand_held_translators_digital_sp eaking_pocket_dictionaries.html. Katz, H.A. (2005) ‘Streaming mLearning objects via data resolution and web services to mobile devices: design guidelines and system architecture model’, MLEARN, South Africa, October 2005, Available at: http://www.mlearn.org.za/CD/papers/Katz & Worsham.pdf. Lockyer, L., Bennett, S., Agostinho, S. and Harper, B. (2008) Handbook of Research on Learning Design and Learning Objects: Issues, Applications and Technologies. Publ. In. Sc. Reference. NESTA (2009) ‘Futurelab series report 11: literature review in mobile technologies and learning’, Available at: http://www.futurelab.org.uk/. Paiva, A., Self, J. and Hartley, R. (1995) ‘Externalizing learner models’, Proceedings of AIED-95, AACE Publication. Available at: http://w5.cs.uni-sb.de/~dominik/um/papers/ 1-softwarewerkzeuge/paiva1995-TAGUS.pdf. Sharples, M., Taylor, J. and Vavoula, G. (2005) ‘Towards a theory of mobile learning, mLearn’, Available at: http://www.mlearn.org.za/CD/papers/Sharples-Theory of Mobile.pdf. Smith, P. (2002) Net Works Guide to Wap and WML. Delhi: Jaico Book House. Starostenko, O. (2005) A Hybrid Approach for Image Retrieval with Ontological Content-Based Indexing, Lecture Notes, Progress in Pattern Recognition, Germany: Springer-Verlag. Starostenko, O., et al. (2008) ‘Shape indexing and retrieval: a hybrid approach using ontological descriptions’, In K. Elleithy (Ed.), Innovations and Advanced Techniques in Systems, Computing Sciences and Software Engineering. USA: Springer Science+Business Media B.V. Talukder, K.H. and Harada, K. (2007) ‘Development and performance analysis of an adaptive and scalable image compression scheme with wavelets’, Proceeding of International Conference on Information and Communication Technology ICICT, Dhaka, Bangladesh. Vygotsky, L.S. (1987) The Collected Works of L.S. Vygotsky. New York: Plenum Press. Warren, N. and Bishop, P. (2005) Taking Service-Oriented Architectures Mobile, Part 1: Thinking Mobile. Available at: http://today.java.net/pub/a/today/2005/06/21/mobile1.html.
