COMPARING THE RESULTS OF CLASSROOM AND ONLINE LEARNERS READING COMPREHENSION USING EYE TRACKER (FRENCH LANGUAGE, A1)

ABSTRACT

In this article, we used the eye-tracker to compare the ability to read and understand texts in A1-level French students who were trained in the classroom and online. To conduct the study, we selected 12 students aged 18 to 22 who had yet to study French. The selected students were distributed by classroom and online class, while for an even distribution, we took into account their age, gender, language skills, and average score. To teach French in both the classroom and online, we used the French textbook ‘ECHO A1,’ compiled by the requirements of the Common European Framework of Reference for Languages. In both these classes, the lessons were conducted by the same teacher, according to the same notes, and according to the same methodology. The entire training period was 13 weeks, and there was a total of 52 sessions of 104 hours each in the classroom and online, with a frequency of 4 sessions per week, each lasting 90 minutes. In order to accurately examine the results of the above two types of learning, knowledge acquired and progress, we used linguistic research equipment to test students every 14 days regularly. Six formative experiments and tests were carried out; the results were compared. We carried out experimental studies for three months, using the instruments at the disposal of the Center for Experimental Linguistics at the NUM. This study is unique and essential in that we empirically established differences in reading and comprehension skills between students taught in face-to-face and online classes using modern equipment.

АННОТАЦИЯ

В данной статье мы, используя айтрекинг, попытались сравнить способность читать и понимание текстов у студентов французского языка уровня А проходивших обучение в классе и онлайн. Для проведения исследования мы выбрали 12 студентов в возрасте от 18 до 22 лет, ранее не изучавших французский язык. Выбранные студенты были распределены по-аудиторному и онлайн-классу,  при этом для равномерного распределения  мы учитывали их возраст, пол, языковые навыки и средний балл. Для преподавания французксого языка в обоих классах мы использовали учебник французского языка “ECHO А1”, составленный  согласно требованиям Общеевропейской компетенции владения иностранными языками и в обоих классах уроки проводил один и тот же учитель, по одному и тому же конспекту, и по одной и той же методике. Весь период обучения составил 13 недель и всего было проведено 52 занятия по 104 часа каждое в классе и онлайн, с частотой 4 занятия в неделю, каждое занятие продолжительностью 90 минут. Чтобы точно изучить результаты двух вышеупомянутых типов обучения, приобретенных знаний и прогресса, мы использовали оборудование для языкового тестирования, чтобы регулярно проверять студентов каждые 14 дней. Всего было проведено шесть промежуточных экспериментов и тестов, результаты которых сравнивались. Мы проводили экспериментальные исследования в течение 3 месяцев, используя приборы, находящиеся в распоряжении Центра экспериментальной лингвистики при МонГУ. Данное исследование уникально и важно тем, что мы опытным путем установили различия в умении читать и понимать у студентов проходивших обучение в очных классах и в онлайн классах, используя современное оборудование.

Keywords: eye tracking, level A1, reading and understanding.

Ключевые слова : айтрекинг , уровен  А1, чтение и понимание.

Introduction

In this era, as relations between world countries are becoming closer and more globalized, the demand for learning foreign languages is increasing significantly. From this situation, issues on the methodology of learning and teaching foreign languages and contrastive linguistics are attracting researchers' attention. In the information age in which we live, the significant changes resulting from technological development are affecting not only people's daily lives but also scientific fields to a certain extent. The more computers, information systems, and artificial intelligence are developing at a high level, the more the demand for studying the nature and phenomenon of language is rising. Due to this situation, experimental research based on modern advanced techniques and technologies is being conducted in many fields, such as linguistics and foreign language teaching.

As a result of technique and technological development, new learning methods such as online, e-learning, and distance learning have been created and developed intensively. Since 2000, world countries have introduced e-learning in their education sectors [1]. In particular, due to the impact of the COVID-19 pandemic, secondary and high schools, universities, colleges, and training centers worldwide were forced to switch to a new form of teaching completely. However, it is not sure that the knowledge acquired through traditional classroom courses is at the same level as the knowledge obtained through online courses, a possible form of education that meets the requirement of society and a particular time. Foreign language universities and institutes in Mongolia started online courses in 2020, and research on studying the result of this kind of teaching has yet to be conducted. Therefore, studying the results of the above two teaching forms is vital. In this study, we aimed to compare the reading comprehension skill results of elementary, i.e., A1 level French classroom and online learners, by using the eye tracker. The use of the device has significance for revealing the actual difficulties and problems of the learners during the reading comprehension process.

A total number of twelve students consisting of ten female and two male students aged 18-22 with a GPA of 2.9-3.9, participated in this experiment. When the reading comprehension process of elementary French language learners was studied, the students were asked to read a French text related to the lesson content taught in the fortnight and answer test questions based on the text while the eyetracker assessed these processes. Using the eyetracker, we obtained the text reading time, video records of eye saccades, and data from heat maps and gaze plots. Precisely, we analyzed the heat map fixation and attention, gaze plot fixation, i.e., quantitative data on the total number of eye movements, regressive saccades, progressive saccades, and gaze plot attention and compared it with the performance of tests as well as the results of six formative tests. The result of the summative test was averaged and comparatively studied.

1. A Theoretical Basis for Eyetracking Research

According to research conducted by French scientist Émile Javal between 1878-1879, he defined for the first time that eye movements during the reading process do not go along the text line like drawing a straight line. However, eye saccades jerk from one point to another point and stop at the point for a short period [2]. However, in 1937, American educational psychologist Guy Thomas Buswell recorded eye movements on the film and determined a technique difference between reading aloud and silently. After that research, in 1980, American psycholinguist Keith Rayner proposed an eye movement technique. Rayner mainly studied how information is processed during reading and defined that the one left-hand and two right-hand words from the target word, i.e., short pause point, are simultaneously processed during the text reading. Therefore, the readers can process the meaning of the context without necessarily reading the word or the end of the sentence [3].

The eyetracker uses an exact method for recording how the eyes move in real-time so it can be used for detecting difficulties and problems encountered while reading. In other words, with the eye-tracking method, it is possible to reveal the comprehension difficulty of the reader by determining what, when, and where the problems are [4]. In 1991, Portuguese researcher Costa A. M.  conducted a study on the technique of reading aloud and noted that when the reader encountered an unknown word or phrase and a problem, the reading speed slowed down. The reader tended to stutter, slow down, become silent, and look again. However, the reading act is a reasonably constant process, and the eyes do not always move forward from left to right. A certain percentage of eye movements are backward movements. The eyes look back when the information is not processed, understood, or known. This process is called a regressive saccade. The eyes move to the next point when the word is fully identified, and the information is processed. This process is called a progressive saccade [5].

While reading, those brief pauses at a certain point last for an average of 100 to 800 milliseconds for an adult. An average pause is 250 milliseconds for the students who participated in the experiment. In other words, since a second equals 1000 milliseconds, the average eye movement is four times per second. On the other hand, eye movements take 10-20 milliseconds to move from one word to another and 60-80 milliseconds to jerk from the previous line to the next [6]. For a good reader, the average pause time is about 200 to 250 milliseconds, and the average transition distance of jerking from one point to another is 7 to 9 letter skips. There is only one stop at a word, and short or many repeated words are skipped. It was also observed that short pauses were made on predictable words [7].

Eye movements and cognitive acts are directly related, i.e., the eyes reflect cognition [8]. Johnston P. J. considered that cognitive acts, including eye movements, can be monitored during the reading process using special devices. However, ‘comprehension’ cannot be observed, so we should ask the reader to do something to check comprehension. Text-related questions are an indirect measurement of reading comprehension.

2. The Result of the Experimental Study

An average result of 6 formative tests and a summative test taken by the 12 classroom and online students is shown in Graph 1 and Table 1.

Graph 1. Average Reading Test Scores of A1 French Language Learners in Classroom and Online Classes

Table 1.

Result of Mean Reading Test Scores for A1 French Language Learners in Classroom and Online Classes

The results of the other tests are similar, except for a significant difference of 37% in the second test with a ratio of 77:40. The results of tests 1-3 of the classroom students have a steady increase or improvement. According to the table and graph above, the starting results of the classroom and online class students are at the same point in terms of reading comprehension. On the other hand, the results of the second test of the online class students decreased, and a significant difference was observed. Three of the six students did not attend class in three of the eight sessions in 14 days of the course. Therefore, we consider that the attendance caused lesson backwardness, and that situation influenced the result of the study. Moreover, the results of the fourth and fifth tests of both the classroom and online types of students went down consecutively due to the difficulty of the course content. By then, we had studied the most difficult elementary French grammar: the complex past tense. It takes an average of a month to fully acquire knowledge of the complex past tense. However, the result of the sixth test increased again. That rise may indicate that the students mastered the previous knowledge and learned to apply it.

In order to check whether the students who participated in the experimental study have fully mastered the A1 level of the French language, we chose a group of professional class students who have completed the A1 level, studying in ‘the French Language and Area Studies’ program at NUM as a benchmark and set a goal of reaching the standard of that student. Thus, four professional class students were given a summative test to assess their A1-level knowledge and skills. The same test was given to the experimental class students at the end of the study. The test result of the professional class students was 94%, time spent 67 seconds, gaze plot fixation was 224, of which 3%, i.e., 7 were regressive saccades, the remaining 97% or 217 were progressive saccades, and gaze plot attention was 93. The performance of the classroom students was quite close to the result. For example, the test result was 92%, gaze plot fixation, gaze plot attention, and the number of regressive saccades was very close to the results of the professional class students.

Compared to the students of the professional class, the experimental class students studied relatively fewer hours. However, their results were only 2% different from the standard assessment, and the other numerical indicators were also very close to the benchmark. Therefore, the experimental classroom students achieved the target results regarding reading comprehension. However, the online experimental class students fell 13% short of reaching the benchmark, but they came pretty close to the benchmark.

All the results of the six formative tests and the last integrated test of the A1 level French students were averaged and compared (Table 2.)

Table 2.

Average Results of All Tests for Classroom and Online Students

As Table 2 shows, the test results of the students who studied in the classroom are 13% higher, the time spent reading the original text of 112 words is 26 seconds less, and the number of gaze plot fixation, regressive saccades, and gaze plot attention is lower than the others. According to this result, as the reading ability improves, total gaze plot fixation, regressive saccades, gaze plot attention, and spending time can be less.

Conclusion  

In this study, we investigated A1-level French students' reading comprehension using the eyetracker. The students who understood the meaning of the text well and scored high on the test had less spent time and a number of gaze plot fixation, regressive saccades, gaze plot attention. However, it was observed that the students who did not fully understand the meaning of the text or who scored low on the test had more spent time and a number of gaze plot fixation, regressive saccades, gaze plots. However, it was noted that the students scrolled quickly and skipped short words and word endings when they looked at well-known words, combinations of letters, and words taught frequently during the class. When the students encounter unfamiliar words, combinations of letters, poorly understood or confusing words, conjunctions, and words taught less frequently during class, their pupils dilate, concentrating more and making regressive saccades.

References:

1. Suvdmaa, T., (2005). E-learning, UB.
2. Rayner, K. (1998). Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin, 3(124), p.372-422. Available at: http://wexler.free.fr/library/files/rayner%20%281998%29%20eye%20movements%20in%20reading%20and%20information%20processing.%2020%20years%20of%20research.pdf
3. Frazier, L., & Rayner, K. (1982). Making and Correcting Errors during Sentence Comprehension: Eye Movements in the Analysis of Structurally Ambiguous Sentences. Cognitive Psychology, 14, 178-210. Available at: https://www.coli.uni-saarland.de/~masta/WS15/FrazierRayner1982.pdf
4. Frenck-Mestre, C. (2005). Eye-movement recording as a tool for studying syntactic processing in a second language: A review of methodologies and experimental findings. Second Language Research, 21, p.175-198. Available at:  https://www.researchgate.net/publication/32222855_Eye-movement_recording_as_a_tool_for_studying_syntactic_processing_in_a_second_language_A_review_of_methodologies_and_experimental_findings
5. Luegi, A Costa & I Faria. (2011). Using eye-tracking to detect reading difficulties. Journal of Eyetracking, Visual Cognition and Emotion. Volume 1, Number 1, p.41-49. Available at: https://repositorio.ul.pt/bitstream/10451/29996/1/2049-1-7022-1-10-20110601.pdf
6. Ivana Roncevic, (2021). Eye-tracking in Second Language Reading. International Journal of Innovation, Creativity and Change, Volume 15, Issue 5, p.416-432. Available at: https://www.academia.edu/69484083/Eye_tracking_in_Second_Language_Reading
7. Leah Roberts and Anna Siyanova-Chanturia, (2013). Using eye-tracking to investigate topics in l2 acquisition and l2 processing. Available at: https://www.researchgate.net/publication/259433937_Using_eye-tracking_to_investigate_topics_in_L2_acquisition_and_L2_processing
8. Liversedge, S. P., & Findlay J. M. (2000). Eye movements reflect cognitive processes. Trends in Cognitive Sciences, 4, 6-14. Available at: https://philpapers.org/rec/LIVEMR