IMCL 2007, Amman/Jordan

A new automated method of e-learner's satisfaction measurement Armands Strazds, Atis Kapenieks Riga Technical University Distance Education Studies Centre / Latvia

Key words: Tele-learning, m-learning, non-linear learning, multitasking evaluation, ambient usability, quality assessment, humancomputer interaction Abstract: This paper presents a new method of measuring learner’s satisfaction while using electronic learning materials (e-courses, edutainment games, etc.) in virtual non-linear environments. Method is based on a relation of Discovering and Learning probability distribution curves obtained by collecting and evaluating the human-computer interaction data. While being near real-time, this measurement is considered highly unobtrusive and cost-effective because of its automated approach. The first working prototype EDUSA 1.0 was developed and successfully tested by the Distance Education Studies Centre of Riga Technical University.

1 Definition of learner’s satisfaction ISO 9241-11 (1998) Guidance of Usability standard defines satisfaction as one of the core components of the product usability assessment. Nielsen and Shneiderman [1, 2] describe subjective satisfaction as a part of “usefulness” in a framework of system acceptability. According to Keller [7] satisfaction relates to perceptions of being able to achieve success and feelings about the achieved outcomes. From this perspective, several studies have explored student satisfaction with online learning materials [8, 9, 10, 11].

2 Learner’s satisfaction assessment methods 2.1

Non-automated satisfaction measurement

Johnson et al indicate that studies of learner satisfaction are typically limited to onedimensional post-training perceptions of learners. Learner’s satisfaction is too often measured with “happy sheets” that ask learners to rate how satisfied they were with their overall learning experience [6].

1(8)

IMCL 2007, Amman/Jordan

Harrison, Seeman et al. [12] identified four major components of effectiveness in distance education programs: instruction, management, telecommuting, and support. Within each of these broad categories are two to five subcomponents. Jegede et al. described another example of a validated approach to assessing a deeper degree of satisfaction identifying eight components of effective learning environments: interactivity, institutional support, task orientation, teacher support, negotiation, flexibility, technological support, and ergonomics. By building on these valid and reliable measures of effective learning environments, a more significant assessment of learner satisfaction and outcomes can be obtained [13]. 2.2

Automated satisfaction measurement

According to the literature there have been very few attempts or in very strictly defined environments (e.g. MS Word) started until now to develop methods of truly automated system-wide evaluation of learner’s satisfaction. In order to perform a satisfaction measurement of today’s technology-savvy non-linear [5] learner, a holistic automated measurement approach is required. 2.3

Browsing, Discovering and Learning probability distributions

2005 Distance Education Studies Centre at Riga Technical University started a research project based on earlier defined concepts of E-Gestures and Good Content Indicators [3, 4] and developed a first working prototype (called EDUSA 1.0) with the functionality of automated measurement of learner’s satisfaction. 2006 the research area was extended by adding EDUSA tests for m-learning within the scope of ‘PUMPURS’ project (VPD1/ERAF/CFLA/05/APK/2.5.1./000078/038). First experimental data gathered from 11 man/days and 60 test participants revealed the presence of two characteristic probability distributions that were called Discovering and Learning curves. The third component - Browsing curve - was later added to complete the model.

Figure 1 Discovering and Learning curves according to the EDUSA-Model

2(8)

IMCL 2007, Amman/Jordan

2.4

Discovering/Learning Behaviour Assessment

EDUSA-Test emphasizes the organic and functional relation between all parts (tasks) and the whole system (in broader sense – human, computer and surrounding ambience). It acts task independently at the very core of the operational system. It connects to the human-computer interface to scan all the communication between user and system. The resulting information is searched for programmatically recognizable patterns of human behaviour (e-gestures) and used to identify learner’s subjective satisfaction with the learning material. Optionally EDUSA can build and export learner’s profile that can be later used with other multi-tasking evaluation sessions. During the testing session EDUSA writes every task-related action to a XML log file. This way it can handle both continued and discontinued learning sessions while analyzing the recorded data subsequently. EDUSA has the ability to reconstruct discontinued learning tasks and analyze them by putting in different evaluation contexts (task scopes, e-gesture sets, etc.). EDUSA is aware of all learner-computer interactions provided by the interface. This allows evaluation of both linear and nonlinear learning sessions. To examine the user behaviour/satisfaction patterns, EDUSA-Tests with two different e-learning product categories were made: an eLearning course represented by the eCourse SQL Fundamentals and online game represented by Marketplace game. Results available after the automated data analysis included: (1) a reference user activity index, (2) a per-cent deviation between user data and calculated curve, (3) a per-cent relation between Browsing, Discovering and Learning (BDL) integral values, (4) time points of BDL curve maximum occurrences and (5) width values for the BDL curves.

Browsing Discovering Learning

S1

S2

S3

Logged off

S4

t

Figure 2 Browsing/Discovering/Learning (BDL) time slots

Figure 2 shows the time slots according to the EDUSA BDL model. In the reality these time slots are almost never strictly separated, but rather constitute an overlapping 3-curve (Browsing, Discovering and Learning curves) system that can be effectively separated and analysed by the system.

3(8)

IMCL 2007, Amman/Jordan

2.4.1 EDUSA-Test for eCourse

Figure 3 Game-like Discovering/Learning behaviour

Subject Session Profile Duration Activity Deviation

e-Course: SQL Fundamentals 20051125 Liepaja Group 1 3 hours 9602 4,8%

EDUSA-Test parameters (SQL/Liepaja/1)

Weight Max at Max Width

Browsing 15,05% 1s 823,9 1

Discovering 59,27% 3s 898,8 0,06

Learning 25,68% 15s 74,9 0

EDUSA-Test results (SQL/ Liepaja /1)

GAME

BOOK

Game-Book Index (GBI) calculated using EDUSA-Test for SQL/Liepaja/1

EDUSA-Test shows that this particular target group (one or more users) has treated the subject learning material more game-like than book-like. If the intention of the particular eContent creator was to develop a game-like eLearning product, then the target group can be considered as satisfied with the product at 59,27%.

4(8)

IMCL 2007, Amman/Jordan

2.4.2 EDUSA-Test for eGame (1)

Figure 4 Neutral Discovering/Learning behaviour

Subject Session Profile Duration Activity Deviation

e-Game: Marketplace 20060626 Dikli Group A 3 hours 11594 2,02%

EDUSA-Test parameters (SQL/Dikli/A)

Weight Max at Max Width

Browsing 9,76% 1s 645,3 1

Discovering 46,32% 2s 1935,9 0,4

Learning 43,93% 5s 430,2 0,02

EDUSA-Test results (SQL/Dikli/A)

GAME

BOOK

Game-Book Index (GBI) calculated using EDUSA-Test for SQL/Dikli/A

EDUSA-Test shows that this particular target group (one or more users) has treated the subject learning material partially game-like and partially book-like (neutral Discovering/Learning behaviour). If the intention of the particular eContent creator was to develop an eLearning product with both the game and the book user behaviour components, then the target group can be considered as satisfied with the product at 90,25%.

5(8)

IMCL 2007, Amman/Jordan

2.4.3 EDUSA-Test for eGame (2)

Figure 5 Book-like Discovering/Learning behaviour

Subject Session Profile Duration Activity Deviation

e-Game: Marketplace 20060626 Dikli Group F 3 hours 9794 1,61%

EDUSA-Test parameters (SQL/Dikli/F)

Weight Max at Max Width

Browsing 40,66% 2s 1517,4 0,45

Discovering 8,34% 5s 252,9 0,3

Learning 51% 10s 168,6 0

EDUSA-Test results (SQL/Dikli/F)

GAME

BOOK

Game-Book Index (GBI) calculated using EDUSA-Test for SQL/Dikli/F

EDUSA-Test shows that this particular target group (one or more users) has treated the subject learning material more book-like than game-like. If the intention of the particular eContent creator was to develop a book-like eLearning product, then the target group can be considered as satisfied with the product at 51,00%.

6(8)

IMCL 2007, Amman/Jordan

3 Application scenarios of learner’s satisfaction measurement method Learner’s satisfaction measurement can be applied to various types of electronic learning content packages including learning objects, e-courses and edutainment games. 3.1

E-content target group determination

A consolidated target group value can be calculated for a specified eLearning product in order to determine a probability of satisfaction distribution within specific target groups of learners, e.g.: students with specific age, gender, skills, etc. 3.2

E-content product categorization

Electronic learning materials can be categorized using EDUSA model in Game-like and Book-like products. The stronger the component of Discovering within the EDUSA measurement, the greater is the probability that product is Game-like. The same is true also for the Learning component and Book-like products.

4 Conclusion The new method of e-learner's satisfaction measurement with its high degree of unobtrusiveness and cost-effectiveness can support industry of developers and producers of electronic learning materials (e-courses, edutainment games, etc.) in efficient, early and automated usability assessment offering new possibilities to better adjusting learning products to the needs of specific target groups and learning context requirements. According to the EDUSA-Test results target groups can be considered as satisfied or unsatisfied with the certain component category of the electronic learning material if the intention of the content producer corresponds to the treatment pattern of the user. The automatically measured Game-Like vs. Book-Like material treatment by the user can be substantially valuable for the electronic content producers to better determine target group markets for the new and/or existing eLearning products.

7(8)

IMCL 2007, Amman/Jordan

References: [1] Nielsen, J.: Usability Engineering, Morgan Kaufmann Publishers, ISBN 0-12-518406-9, 1994 [2] Shneiderman, B.: Software Psychology, 1980 [3] Strazds, A., Kapenieks, A. et al.: Piloting of EDUSA in non-linear multimedia learning environments, Conference on Interactive computer aided learning (ICL 2006), Villach, Austria, September 26 - 28, 2006, CD-ROM, ISBN 3-89958-195-4, Kassel University Press, 2006 [4] Strazds, A.: m-Learning evaluation - a multi-tasking approach, Conference on Interactive Mobile and Computer Aided Leaning (IMCL 2006), Amman, Jordan, April 19 - 21, 2006, CD-ROM, ISBN 3-899958-177-6, Kassel University Press, 2006 [5] Veen, W.: Teaching the media generation: Coping with Homo Zappiens. Gotenborg, 2006 [6] Johnson, S.D., Aragon, S.R. et al: Comparative Analysis of Learner Satisfaction and Learning Outcomes in Online and Face-to-Face Learning Environments, Journal of Interactive Learning Research 11(1), 29-49, 2000 [7] Keller, J.: Motivational design of instruction. In C. Reigeluth (Ed.), Instructional design theories and models: An overview of their current status (pp. 386-434). Hillsdale, NJ: Erlbaum, 1983 [8] Debourgh, G.A.: Learner and instructional predictors of student satisfaction in a graduate nursing program taught via interactive video conferencing and world wide web/internet. Unpublished doctoral dissertation, University of San Francisco, 1989 [9] Enockson, J.: An assessment of an emerging technological delivery for distance education. Unpublished doctoral dissertation, Northern Arizona University, 1997 [10] Johanson, T.L.: The virtual community of an online classroom: Participant’s interactions in a community college writing class by computer mediated communication. Unpublished doctoral dissertation, Oregon State University, 1996 [11] McCabe, M.: Online classrooms: Case studies of computer conferencing in higher education. Unpublished doctoral dissertation, Columbia University Teachers College, 1997 [12] Harrison, P.J., Seeman, B.J., et al.: Development of a distance education assessment instrument. Educational Technology Research & Development, 39(4), 65-77, 1991 [13] Jegede, O.J., Fraser, B., Curtin, D.F.: The development and validation of a distance and open learning environment scale. Educational Technology Research & Development, 43(1), 90-94, 1995

Authors: Armands Strazds, M. A. Riga Technical University, Distance Education Study Centre Azenes 12, Riga [email protected] Atis Kapenieks, Dr. Phys. Riga Technical University, Distance Education Study Centre Azenes 12, Riga [email protected]

8(8)

A new automated method of e-learner's satisfaction ...

This paper presents a new method of measuring learner's satisfaction while using electronic learning materials (e-courses, edutainment games, etc.) in.

59KB Sizes 1 Downloads 221 Views

Recommend Documents

A new automated method of e-learner's satisfaction ...
satisfaction measurement of today's technology-savvy non-linear [5] learner, a ... discontinued learning tasks and analyze them by putting in different evaluation .... Educational Technology Research & Development, 39(4), 65-77, 1991.

Automated Satisfaction Measurement for e- Learning ...
is based on a method developed by the Distance Education Study. Centre at Riga Technical ... technology-savvy non-linear [6] learner, a holistic automated measurement .... suggestions about the contents of this article. REFERENCES.

A fully automated method for quantifying and localizing ...
aDepartment of Electrical and Computer Engineering, University of Pittsburgh, .... dencies on a training set. ... In the current study, we present an alternative auto-.

Automated satisfaction measurement for e-learning ...
studies have explored student satisfaction with online learning materials [9, 10, 11,. 12]. ... degree of satisfaction identifying eight components of effective learning .... nursing program taught via interactive video conferencing and world wide ..

A Discriminative Method For Semi-Automated Tumorous ... - CiteSeerX
A Discriminative Method For Semi-Automated. Tumorous Tissues Segmentation of MR Brain Images. Yangqiu Song, Changshui Zhang, Jianguo Lee and Fei ...

Automated Satisfaction Measurement for e- Learning ...
Centre at Riga Technical University describing the relation between Discovering ... 000078/038). First experimental data gathered from 11 man/days and 60 test.

A Discriminative Method For Semi-Automated ...
In the feature, we plan to do some fast sparse algorithm and a C++ implementation, which could make our algo- rithm used in an interactive way. References. [1] M. Belkin, P. Niyogi and V. Sindhwani: On Manifold Regu- larization. AI & Statistics, pp.

A Discriminative Method For Semi-Automated ...
ti. ). For the algorithms that do not need to predict for the unobserved data, there are mainly two ways to model this probability: a) : P(XN. ,tN. ) = P(XN. |tN. )P(tN. ).

A fully automated method for quantifying and localizing ...
machine learning algorithms including artificial neural networks (Pachai et al., 1998) .... attenuated inversion recovery (fast FLAIR) (TR/TE= 9002/56 ms Ef; TI=2200 ms, ... imaging data to predefined CHS visual standards and representative of ...

A Discriminative Method For Semi-Automated Tumorous ... - CiteSeerX
Unsupervised methods are difficult to produce good result fully au- .... Tumorous Tissues in MR Images via Support Vector Machine and Fuzzy Clustering. IFSA.

Modeling of a New Method for Metal Filaments Texturing
Key words: Metallic Filament, Yarn, Texturizing, Modeling, Magnetic Field. Introduction ... The Opera 8.7 software is used for simulating the force of rotating ...

Development of a new method for sampling and ...
excel software was obtained. The calibration curves were linear over six .... cyclophosphamide than the analytical detection limit. The same time in a study by.

A New Method of Estimating the Pollen Dispersal Curve ... - Genetics
perform the estimations for a single simulation repli- cate. For this reason, we performed a limited ...... should cover as many pairwise-distance classes as possi-.

A New Method of Estimating the Pollen Dispersal Curve ... - Genetics
perform the estimations for a single simulation repli- cate. For this reason, we performed a limited ...... should cover as many pairwise-distance classes as possi-.

An Automated Method for Circular-Arc Metro Maps
Email: [email protected]. †. TU Eindhoven, The Netherlands. Email: 1a.i.v.goethem,w.meulemans,[email protected]. ‡. Universität Osnabrück, Germany. Email: [email protected]. I. INTRODUCTION ... places two arcs

Validation of a constraint satisfaction neural network for ...
In addition, the effect of missing data was evaluated in more detail. Medical Imaging .... Receiver Operating Characteristics (ROC) analysis. We used the ROCKIT ...

Richard A. Murray. Job Satisfaction of Professional and ...
standard instrument used to measure job satisfaction of employees in non-profit and human services ... in partial fulfillment of the requirements ... attempted any sort of meaningful comparison of the two groups. The current study ... early stages, b

New exact algorithms for the 2-constraint satisfaction ...
bound on the size of a vertex separator for graphs in terms of the average degree of the graph. We then design a simple algorithm solving MAX-2-CSP in time O∗(2cdn), cd = 1 − 2α ln d d for some α < 1 and d = o(n). Keywords: exact exponential ti

Keyword Spices: A New Method for Building Domain ...
domain-specific search engine for computer science research papers. ... We call this the filtering model for building .... simplify keyword spices in the way that results in high value ..... national World Wide Web Conference(WWW6), pages 189–.

A New Shot Change Detection Method Using ...
Department of Electronic Engineering, Shanghai Jiao Tong University ..... Consumer Electronics, vol. 49, pp. ... Circuits and Systems for Video Technology, vol.

A new characterisation method for rubber (PDF Download Available)
heterogeneous mechanical test, measuring the displacement/strain field using suitable ..... ments, load, specimen geometry and unknown parameters.