ENGAGING LEARNERS THROUGH REAL-TIME SUBTITLE TRANSLATION: DESIGN AND PILOT EVALUATION

ABSTRACT

This study presents the design and pilot evaluation of an interactive subtitle translation system for second language acquisition. Unlike passive subtitle viewing, the proposed system embeds real-time translation exercises — gap-fill tasks, multiple-choice questions, and short translation prompts — directly into the video player. A controlled experiment with ten intermediate-level learners compared the interactive system against passive dual subtitles. The experimental group achieved a vocabulary gain of 19.6% and a listening comprehension improvement of 27.6 percentage points, versus 7.2% and 7.4 points in the control group. These findings confirm that active translation practice integrated into video-based learning significantly enhances language acquisition outcomes.

АННОТАЦИЯ

В данном исследовании представлены разработка и пилотная оценка интерактивной системы перевода субтитров для изучения второго языка. В отличие от пассивного просмотра субтитров, предлагаемая система встраивает упражнения по переводу в реальном времени — задания на заполнение пропусков, вопросы с несколькими вариантами ответа и короткие подсказки для перевода — непосредственно в видеоплеер. В контролируемом эксперименте с участием десяти учащихся среднего уровня сравнивали интерактивную систему с пассивным просмотром двойных субтитров. Экспериментальная группа достигла прироста словарного запаса на 19,6% и улучшения понимания на слух на 27,6 процентных пунктов, по сравнению с 7,2% и 7,4 пунктами в контрольной группе. Эти результаты подтверждают, что активная практика перевода, интегрированная в видеообучение, значительно улучшает результаты изучения языка.

Keywords: subtitle-based learning, interactive subtitles, second language acquisition, vocabulary gain, listening comprehension, dual subtitles, multimedia learning.

Ключевые слова: обучение с использованием субтитров, интерактивные субтитры, освоение второго языка, расширение словарного запаса, понимание на слух, двойные субтитры, мультимедийное обучение.

1. Introduction

In recent years, the proliferation of on-demand video streaming platforms and the vast availability of multimedia content have revolutionized foreign language learning. Learners now routinely turn to movies, series, and online lectures to supplement traditional classroom instruction. Subtitles have become a cornerstone of this trend, providing a dual channel for input: auditory (spoken dialogue) and visual (written text). According to cognitive theory of multimedia learning, such dual- channel processing can enhance memory and comprehension, provided cognitive load is managed effectively [1]. Indeed, subtitles in educational videos have been shown to bridge comprehension gaps, particularly for learners with limited proficiency, by reinforcing the mapping between sound and meaning [2], [3].

Even though these benefits, however, most subtitle-based learning hitherto have been passive. The students passively view static subtitles in their native language (native-language subtitles, NLS) or in the original language (same-language subtitles, SLS), but no further interaction. Even though NLS as well as SLS can support comprehension as much as incidental vocabulary acquisition [4], passive viewing alone does not necessarily involve higher-order mental processing needed for deep encoding as much as production skills. Second-language acquisition research points out that active use, as much as production practice, such as translation, yield more stable learning gains than viewing mere input [5], [6].

Different strands of literature portray current conditions of subtitle use among language learning. Controlled classroom experiments have found subtitling class-watched video reduces extraneous cognitive load as well as improved retention of technical information [2]. Systematic reviews of interlingual as well as intralingual subtitle effectiveness support standard comprehension gain with subtitles used wisely [3]. Out-of-class use, meanwhile, allows availability of dual-subtitle software to provide for student as well as target-language text display simultaneously. Dual-subtitles research indicates improved vocabulary acquisition along with listening comprehension over monolingual captions [7], with greatest vocabulary learning effect size found in intralingual captioning, but with added comprehension provided by dual subtitles [8]. Pilot studies also find heightened learner confidence and uptake in use of interactive subtitle add-ons to online video streamed services [9].

Theoretical accounts in terms of dual coding theory, as well as cognitive load theory, account for such effects. Subtitles support integration of visual, as well as auditory, channels [10], as well as high-quality multimedia minimizes redundancy, as well as split-attention, effects [11], [12]. Nevertheless, most applications of subtitles are passive. For a transition to active learning, to conduct experiments, authors used interactive subtitle activities. Incorporation of translation exercises, as well as gap-fill exercises, to subtitle streams provides better retention, as well as diminished test anxiety [13]. Student-generated production of subtitles further provides autonomy, as well as acquisition [14]. Incorporation of gap-fill exercises to subtitles strongly improves vocabulary recall over rote memorization [15].

Enthusiasm, or a readiness to engage with language, is a central indicator of language learning success. Multimedia practice integrating audio, visual, and text inputs supports motivation, as long as learners actively participate in production [14]. Listening practice with subtitles was found to feel more enjoyable, as well as more stimulating culturally, than homework [16], and provides contextual information about circumstances required to bridge book language with actual usage [17].

Despite this extensive body of work, a notable gap remains: no system currently integrates subtitle display with real-time translation exercises in a seamless video-viewing environment. Most interactive approaches are limited to offline tasks (e.g., worksheet fill-ins), requiring context switches that disrupt the viewing flow. To fill this gap, we propose an Interactive Subtitle Translation System that:

• displays dual subtitles (original and translated) in sync with video.
• pauses playback at designated intervals to prompt gap- filling, multiple-choice, and short translation tasks directly within the player.
• provides immediate feedback and logs user responses for performance analysis.

It gives feedback instantly and stores user responses for performance tracking. This interactive pairing aims to take advantage of cognitive gains in dual coding as well as motivational gains in interactivity by adding translation practice to natural video comprehension. We detail in the subsequent section the two-phased process—system development and experiment, including pseudocode for task generation as well as a measure to test learning gains.

2. Materials and methods

The project’s research process comprises two consecutive phases, namely, system design of the interactive subtitle translation system and experiments of the system. The two- phase process ensures that all the functions of importance in the system are fully developed before finally testing them in a controlled learning environment.

1. System Development

Four interactive task types were embedded in the player that:

• Gap-fill exercises present a subtitle sentence in which a single key word is blanked out. Learners must type or select the correct word based on context.
• Multiple Choice Questions (MCQs) ask learners to choose the correct translation of a highlighted word or phrase from four options.
• Short Translation Tasks require learners to produce a written translation of a full subtitle sentence.
• Review prompts occasionally invite learners to compare their own translation with the correct answer of the system and to reflect on discrepancies.

These tasks were spaced evenly in time, approximately every 15–20 seconds of video, in order to make the interaction evenly paced over the length of video. The responses, timestamps, and correctness of the task of the learner are captured into a secure back-end store for a future analysis.

Pseudocode for Task Generation:

Figure 1. Subtitle task generation pseudocode

In this pseudocode, select target word randomly selects a high-usage word, drawing upon frequency lists as well as student level. random interval spaces out activities, rather than grouping them, to keep a natural viewing order.

2. Experimental Evaluation

Once a stable system prototype was established, we shifted to a pre-test, intervention, post-test experiment. Target language intermediate adult learners enrolled in language courses at a university participated. Participants all received the same vocabulary and listening comprehension pre-tests, which served as a baseline measurement.

The participants were then randomly assigned to either of two conditions. The Control Group saw the video materials in standard passive subtitle mode (double L2+L1 subtitles) with no inbuilt task. The Experimental Group saw under the interactive subtitle system described above. The same video extracts were seen by both groups over two sessions, a period of two hours.

At the conclusion of the viewing sessions, all participants completed a vocabulary test and a listening comprehension test as post-tests. Experimental Group participants also completed a short survey assessing perceived cognitive load and user satisfaction.

Formula for Learning Gain:

An analogous calculation applies to listening comprehension scores.

Figure 1. Two-Phase Methodology: System Development and Experimental Evaluation. The diagram shows the flow from system features through pre-test, random assignment, group interventions, post-test, and analysis

Backend log information allowed us to examine not only overall test gain but task level performance as well—gap-fill accuracy rates, MCQ accuracy rates, response time, and patterns of error. The satisfaction survey contained a 5-point Likert scale to agree or disagree with, for example, “The interactive exercises taught me new words” and “The system did not spoil my enjoyment of the video.”

This approach integrates system design, task scheduling, experiment control, quantitative gain measurement, and qualita- tive user feedback, providing a complete framework to evaluate added value of active translation activities in subtitle language learning.

3. Results and discussions

For the evaluation of the effectiveness of our proposed inter- active subtitle translation system, a mini-pilot was conducted among ten participants. The learners in each group were inter- mediate level in the target language. Participants were assigned randomly to a control group (n=5) who watched passive dual subtitles with no interactivity, or to an experimental group (n=5), who watched the interactive system with translation activities.

A pre-test and a post-test of vocabulary, as well as listening comprehension, was administered to both groups. The aim was to evaluate learning gains, which can be contributed to by the system. The vocabulary test probed for 20 target words from the video, and listening comprehension tests used comprehension questions from the conversation. Individual scores, along with calculated gains in both of these skills, can be seen in the table provided.

Table 1.

Pre-test and post-test scores with gains for ten participants

The results show a clear difference in performance between the two groups. The control group improved their vocabulary scores from an average of 46.2% to 53.4%, resulting in a gain of 7.2%. Their listening comprehension scores increased from 49.2% to 56.6%, an improvement of 7.4 percentage points.

By contrast, the experimental group achieved significantly higher gains. Their average vocabulary score increased from 49.0% to 68.6%, representing a gain of 19.6%, nearly three times higher than the control group. Similarly, listening comprehension rose from 51.6% to 79.2%, a gain of 27.6 percentage points. These results are consistent with previous studies showing the benefits of dual subtitles combined with interactive exercises for language acquisition [7], [8], [15].

The vocabulary gain difference can also be identified from bar chart provided below, which presents both conditions’ performance side by side for better comparison.

Briefly, mini-pilot substantiates that interactive translation ex- ercises integrated into video watching lead to more vocabulary retention as well as enhanced listening competence than passive viewing of subtitles. With a limited sample, encouraging hints of effectiveness came from findings, which laid groundwork for an evaluation at a bigger scale to follow.

4. Conclusion

We aimed at examining design and early evaluation of an interactive subtitle translation system to afford passive viewing of video passive depth by stimulating language acquisition. Drawing ourselves upon a wide literature testifying to benefits from interactive activities as well as from double subtitles [2], [8], [14], we designed a working prototype combining gap-fills, multiple choice, as well as short translation activities directly into subtitle flow. Mini-piloting with a class of ten intermediate learners demonstrated that a similar method can make a big contribution to vocabulary gain as to listening comprehension: the experiment group obtained a vocabulary gain of 19.6% as well as a listening comprehension gain of 27.6 percentage points, compared to 7.2% as well as to 7.4 points in a control group with passive subtitles.

These, admittedly limited, findings suggest that real-time translation activity elicits more vigorous cognitive processing, as well as greater engagement, but not to a point of unmanage- able cognitive load. Participant feedback evidenced quantitative gains, which students identified as making interactive activities engaging as well as in keeping with viewing.

This pilot, though, does have a few limitations. The sample was narrow, intervention duration was limited, and video content was specialized. Future research needs to get over these limitations by conducting large-scale studies with greater diversity of learners, as well as more varied types of content. There needs to be yet more research to optimize task frequency, adjust task difficulty through adaptive algorithms, as well as to incorporate more sophisticated language technologies like automatic speech recognition, as well as neural machine translation, in order to enable automated task generation, as well as providing feedback.

Overall, we believe that interactive translation practice in subtitled video represents a choice way of maximizing second language acquisition. With authentic multimedia materials included by way of interactive practice, it should make out-of- classroom learning more productive, as well as more enjoyable. More research and testing must occur to introduce interactive subtitles to their potential as a standard tool for language learners.

References:

1. R. E. Mayer and R. Moreno, “Mayer and moreno ways to reduce cognitive load nine ways to reduce cognitive load in multimedia learning what is multimedia learning and instruction?” Tech. Rep., 2003.
2. M. Pannatier and M. Be´ntrancourt, “Learning from academic video with subtitles: When foreign language proficiency matters,” Learning and Instruction, vol. 90, p. 101863, 4 2024.
3.  R. Matielo, R. C. S. F. D’Ely, and L. Baretta, “The effects of interlingual and intralingual subtitles on second language learning/acquisition: a state-of-the-art review,” Trabalhos em Lingu´ıstica Aplicada, vol. 54, pp. 161–182, 6 2015.
4. R. Vanderplank, “Subtitle translation and second language acquisition,” Language Learning Journal, vol. 44, no. 3, pp. 259–274, 2016.
5.  M. Swain, “The output hypothesis: Theory and research,” Handbook of Research in Second Language Teaching and Learning, vol. 1, pp. 471–483, 2005.
6. R. Ellis, “The study of second language acquisition,” Oxford Applied Linguistics, vol. 2, 2008.
7. G. Dizon and B. Thanyawatpokin, “Language learning with netflix: Exploring the effects of dual subtitles on vocabulary learning and listening comprehension,” Tech. Rep., 2021.
8. B. L. Reynolds, Y. Cui, C. W. Kao, and N. Thomas, “Vocabulary acquisition through viewing captioned and subtitled video: A scoping review and meta-analysis,” 10 2022.
9. A. Alm, “Language learning with netflix: Extending out-of-class l2 viewing,” in Proceedings - IEEE 21st International Conference on Advanced Learning Technologies, ICALT 2021. Institute of Electrical and Electronics Engineers Inc., 7 2021, pp. 260–262.
10. A. Paivio, Dual Coding Theory and Education. New York, NY: Weinberg Foundation Press, 2006.
11. J. Sweller, “Cognitive load theory, learning difficulty, and instructional design,” Learning and Instruction, vol. 4, pp. 295–312, 1 1994. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/0959475294900035.
12.  R. Moreno and R. E. Mayer, “Verbal redundancy in multimedia learning: When reading helps listening,” Journal of Educational Psychology, vol. 94, no. 1, pp. 156–163, 2002.
13. M. Lunin and L. Minaeva, “Translated subtitles language learning method: A new practical approach to teaching english,” Procedia - Social and Behavioral Sciences, vol. 199, pp. 268–275, 8 2015.
14. N. T. Zano´n, “Using subtitles to enhance foreign language learning,”Porta Linguarum, vol. 6, pp. 41–52, 2006.
15. J. Birule´s-Muntane´ and S. Soto-Faraco, “Watching subtitled films helps vocabulary learning,” in Proceedings of the 38th Annual Conference of the Cognitive Science Society, 2016, pp. 1234–1239.
16. B. Aksu-Atac¸ and S. Gu¨nay-Ko¨pru¨lu¨, “The role of subtitles in foreign language teaching,” International Online Journal of Education and Teaching, vol. 5, no.3, pp.525–533, 2018. [Online]. Available: http://iojet.org/index.php/IOJET/article/view/350/251.
17. S. Lei, “The role of videos’ subtitles in second language acquisition,” Tech. Rep., 2023.