Shaping Multilingual Access through Respeaking Technology, Project Data, 2021

Davitti, Elena and Wallinheimo, Anna-Stiina (2024). Shaping Multilingual Access through Respeaking Technology, Project Data, 2021. [Data Collection]. Colchester, Essex: UK Data Service. 10.5255/UKDA-SN-856687

Our day and age is characterised by a proliferation of live multimedia and multilingual content, such as news and TV programmes. Such content is not accessible to everyone. In many countries, live subtitles in the same language are required by law to enable deaf and hard-of-hearing audiences to enjoy access to information, culture and entertainment. In the UK and other countries (e.g., Canada, Spain, Switzerland), intralingual respeaking is the most well-established technique to produce these subtitles: it relies on human-machine interaction whereby a respeaker listens to the original sound of a live programme or event and simultaneously dictates it to a speech recognition software that turns each sentence into subtitles displayed on the screen.

Respeaking so far has been used to produce intralingual subtitles, i.e., in the same language. Given the current multilingual content boom, the SMART project aims to investigate whether respeaking can be used to produce interlingual subtitles, i.e. in a different language. This entails adding translation to an already challenging task which includes listening, speaking, adding oral punctuation, controlling prosody to minimise speech recognition errors, and accounting for space constraints to produce meaningful subtitles. Automatic translation alone is unable to provide live subtitles in a different language of sufficient quality. Interlingual respeaking represents therefore an innovative technique that builds on its monolingual predecessor to achieve this goal. This technique, however promising, has not been systematically tested yet. Therefore there is a need for empirical data to assess its viability. To this end, SMART sets out to achieve three main goals:

1. Investigate the feasibility of interlingual respeaking by analysing its process and challenges. Participants' performance during interlingua respeaking tasks will be studied in detail to inform optimal strategies to overcome these challenges. Two pilot projects on students have yielded promising results in this respect. We will need to explore these further in a sample of professionals;

2. Measure the quality of the live subtitles thus produced using an existing, recently-developed measurement tool (NTR model) which could represent a first step towards establishing a quality benchmark for interlingual respeaking;

3. Identify the key competencies needed by professionals already working in the language industry (e.g. interpreting, subtitling) to support timely and efficient acquisition of interlingual respeaking skills.

Interlingual respeaking has the potential to provide interlingual subtitles of virtually any live events, such as lectures, conferences, theatre shows, business meetings, live TV interviews or programmes involving speakers of different languages. This technique could also enhance traditional subtitling methods for pre-recorded programmes, improving productivity. By doing so, interlingual respeaking would make such events and programmes accessible to everyone, irrespective of any sensory or language barriers.

SMART research findings will inform the design of bespoke training courses to equip language professionals with optimal skills to ensure high-quality live interlingual subtitling. We have tested these materials in a Summer School during the project (September 2022). We are currently continuing to refining upskilling through an ESRC Impact Accelerator Account follow-up project, which aims to offer courses that can provide language professionals with a new career path and the possibility to diversify the services they offer. An informative video was also produced to raise awareness of the complex interlingual respeaking technique among the wider public and any other stakeholders interested in providing or benefiting from this service and other emerging workflows involving language professionals and AI-driven technologies like speech recognition and machine translation. The video is accessible here: https://www.youtube.com/watch?v=rxIRKLR2_7o

The UK pioneered the development of intralingual respeaking (today used to subtitle 85% of live TV programmes). SMART has provided the data needed for the UK to become a world leader in interlingual respeaking. This technique is a radical evolution in respeaking methodology and a smart move towards meeting current communication needs, whilst contributing to 'barrier-free' access to different domains of public and private life for all members of society.

Data description (abstract)

The recent global surge in audiovisual content has emphasised the importance of accessibility for wider audiences. The Shaping Multilingual Access through Respeaking Technology (SMART) project addressed this by exploring interlingual respeaking, a novel practice combining speech recognition technology with human interpreting and subtitling skills to produce real-time speech-to-text services across languages. This method evolved from intralingual respeaking, which is widely used in broadcasting to create live subtitles for the deaf and hard-of-hearing. Interlingual respeaking, which involves translating live content into another language and subtitling it, could revolutionise subtitle production for foreign-language content, overcoming sensory and language barriers. Interlingual respeaking involves two shifts: interlingual (from one language to another) and intermodal (from spoken to written). This practice combines the challenges of simultaneous interpreting with the requirements of subtitling. Respeakers must accurately convey messages in another language to a speech recognition system, adding punctuation and making real-time edits for clarity and readability. This method leverages speech recognition technology and human translation skills to ensure efficient and high-quality translated subtitles. Interlingual respeaking offers immense potential for making multilingual content accessible to international and hearing-impaired audiences. It is particularly relevant for television, conferences, and live events. However, research into this practice is still in its early stages. The SMART project's main goals were to study interlingual respeaking's complexity, focusing on the acquisition and implementation of relevant skills (not only procedural, but also cognitive and interpersonal ones), and the accuracy of the final subtitles. The research involved fifty-one language professionals with backgrounds in relevant language-related practices (namely interpreting, translation, subtitling, and intralingual respeaking). The research programme examined three areas: process, product, and upskilling. It sought to understand the variables contributing to language professionals' performance, challenges faced during performance, and how performance can be sustained. Regarding the product, it aimed to identify factors affecting the accuracy of interlingual respeaking and the impact of various individual and content characteristics on accuracy. For upskilling, the focus was on the challenges and strengths of the course delivered as part of the experiment. Key findings included the importance of working memory in predicting high performance and the enhancement of certain cognitive abilities through training. Interpersonal traits like conscientiousness and integrated regulation were also examined. In terms of product accuracy, the average achieved across 153 performances three per participant) after 25h of upskilling was 95.37%, with omissions being the strongest negative predictor of accuracy. High performers outperformed low performers across all scenarios. The upskilling course was innovative, focusing on modular training and combining intralingual and interlingual practices. It addressed real-world challenges and was tailored to different professional backgrounds. The approach proved effective, with 82% of participants finding the course met their expectations and 86% acknowledging its challenging nature. The study confirmed the benefits of a modular and personalised training approach, highlighting the need for flexibility and adaptability to different skill levels and backgrounds.

Downloads

Altmetric